Get Twitter Feed as JSON without authentication

If you're still looking for unauthenticated tweets in JSON, this should work: https://github.com/cosmocatalano/tweet-2-json


You can access and scrape Twitter via advanced search without being logged in:

  • https://twitter.com/search-advanced

GET request

When performing a basic search request you get:

https://twitter.com/search?q=Babylon%205&src=typd
  • q (our query encoded)
  • src (assumed to be the source of the query, i.e. typed)

by default, Twitter returns top 25 results, but if you click on all you can get the realtime tweets:

https://twitter.com/search?f=realtime&q=Babylon%205&src=typd

JSON contents

More Tweets are loaded on the page via AJAX:

https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd&include_available_features=1&include_entities=1&last_note_ts=85&max_position=TWEET-553069642609344512-553159310448918528-BD1UO2FFu9QAAAAAAAAETAAAAAcAAAASAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Use max_position to request the next tweets

The following json array returns all you need to scrape the contents:

https://twitter.com/i/search/timeline?f=realtime&q=Babylon%205&src=typd
  • has_more_items (bool)
  • items_html (html)
  • max_position (key)
  • refresh_cursor (key)

DOM elements

Here comes a list of DOM elements you can use to extract

The authors twitter handle

div.original-tweet[data-tweet-id]

The name of the author

div.original-tweet[data-name]

The user ID of the author

div.original-tweet[data-user-id]

Timestamp of the post

span._timestamp[data-time]

Timestamp of the post in ms

span._timestamp[data-time-ms]

Text of Tweet

p.tweet-text

Number of Retweets

span.ProfileTweet-action–retweet > span.ProfileTweet-actionCount[data-tweet-stat-count]

Number of Favo

span.ProfileTweet-action–favorite > span.ProfileTweet-actionCount[data-tweet-stat-count]

Resources

  • https://code.recuweb.com/2015/scraping-tweets-directly-from-twitter-without-authentication/

Previously the Search API was the only Twitter API that didn't require some form of OAuth. Now it does require auth.

Twitter's Search API is acquired from a third party acquisition - they rarely support it and are seemingly unenthused that it even exists. On top of that, there are many limitations to the payload, including but not limited to a severely reduced set of key:value pairs in the JSON or XML file you get back.

When I heard this, I was shocked. I spent a LONG time figuring out how to use the least amount of code to do a simple GET request (like displaying a timeline).

I decided to go the OAuth route to be able to ensure a relevant payload. You need a server-side language to do this. JavaScript is visible to end users, and thus it's a bad idea to include the necessary keys and secrets in a .js file.

I didn't want to use a big library so the answer for me was PHP and help from @Rivers' answer here. The answer below it by @lackovic10 describes how to include queries in your authentication.

I hope this helps others save time thinking about how to go about using Twitter's API with the new OAuth requirement.

Tags:

Javascript

Twitter

Recent Posts