Avoid Twitter API limitation with Tweepy
The problem is that your try: except:
block is in the wrong place. Inserting data into the database will never raise a TweepError
- it's iterating over Cursor.items()
that will. I would suggest refactoring your code to call the next
method of Cursor.items()
in an infinite loop. That call should be placed in the try: except:
block, as it can raise an error.
Here's (roughly) what the code should look like:
# above omitted for brevity
c = tweepy.Cursor(api.search,
q=search,
include_entities=True).items()
while True:
try:
tweet = c.next()
# Insert into db
except tweepy.TweepError:
time.sleep(60 * 15)
continue
except StopIteration:
break
This works because when Tweepy raises a TweepError
, it hasn't updated any of the cursor data. The next time it makes the request, it will use the same parameters as the request which triggered the rate limit, effectively repeating it until it goes though.
Just replace
api = tweepy.API(auth)
with
api = tweepy.API(auth, wait_on_rate_limit=True)
For anyone who stumbles upon this on Google, tweepy 3.2+ has additional parameters for the tweepy.api class, in particular:
wait_on_rate_limit
– Whether or not to automatically wait for rate limits to replenishwait_on_rate_limit_notify
– Whether or not to print a notification when Tweepy is waiting for rate limits to replenish
Setting these flags to True
will delegate the waiting to the API instance, which is good enough for most simple use cases.