Dealing with exception handling and re-queueing in RQ on Heroku
Here’s my solution
queues = []
def retry_handler(job, exc_type, exc_value, traceback):
# Returning True moves the job to the failed queue (or continue to
# the next handler)
job.meta.setdefault('failures', 1)
job.meta['failures'] += 1
if job.meta['failures'] > 3 or isinstance(exc_type, (LookupError, CorruptImageError)):
job.save()
return True
job.status = Status.QUEUED
for queue_ in queues:
if queue_.name == job.origin:
queue_.enqueue_job(job, timeout=job.timeout)
break
else:
return True # Queue has disappeared, fail job
return False # Job is handled. Stop the handler chain.
queues.append(Queue(exc_handler=retry_handler))
I decided to retry all errors three times unless a certain known exception type was encountered. This allows me to respect failures that are understood, like if a user was deleted after the job was created but before the job was executed, or in the case of an image resize job the image provided is no longer found (HTTP 404) or not in a readable format (basically whenever I know the code will never handle the job).
To answer your question: exc_type is the class, exc_value is the exception instance. traceback
is useful for logging. If you care about this, check out Sentry. Workers are automatically configured with a Sentry error handler if run with SENTRY_DSN
in the context. Much cleaner than polluting your own db with error logs.