Celery (Django) Rate limiting

After much research I found out that Celery does not explicitly provide a way to limit the number of concurrent instances like this and furthermore, doing so would generally be considered bad practice.

The better solution would be to download concurrently within a single task, and use Redis or Memcached to store and distribute for other tasks to process.

Celery features rate limiting, and contains a generic token bucket implementation.

Set rate limits for tasks: http://docs.celeryproject.org/en/latest/userguide/tasks.html#Task.rate_limit

Or at runtime:

http://docs.celeryproject.org/en/latest/userguide/workers.html#rate-limits

The token bucket implementation is in Kombu