Dealing with exception handling and re-queueing in RQ on Heroku

Here’s my solution

queues = []

def retry_handler(job, exc_type, exc_value, traceback):
    # Returning True moves the job to the failed queue (or continue to
    # the next handler)

    job.meta.setdefault('failures', 1)
    job.meta['failures'] += 1
    if job.meta['failures'] > 3 or isinstance(exc_type, (LookupError, CorruptImageError)):
        job.save()
        return True

    job.status = Status.QUEUED
    for queue_ in queues:
        if queue_.name == job.origin:
            queue_.enqueue_job(job, timeout=job.timeout)
            break
    else:
        return True  # Queue has disappeared, fail job

    return False  # Job is handled. Stop the handler chain.

queues.append(Queue(exc_handler=retry_handler))

I decided to retry all errors three times unless a certain known exception type was encountered. This allows me to respect failures that are understood, like if a user was deleted after the job was created but before the job was executed, or in the case of an image resize job the image provided is no longer found (HTTP 404) or not in a readable format (basically whenever I know the code will never handle the job).

To answer your question: exc_type is the class, exc_value is the exception instance. traceback is useful for logging. If you care about this, check out Sentry. Workers are automatically configured with a Sentry error handler if run with SENTRY_DSN in the context. Much cleaner than polluting your own db with error logs.