Celery: number of workers vs concurrency
Intersting question.
Things that I can think of (I'm sure there are a lot more):
- For having high availability:
- You want more than one machine (if one goes down) - so you must use worker per machine.
- Even for one machine - I think it is safer to have 2 workers which run in a two different processes instead of one worker with high concurrency (correct me if I wrong, but I think it is implemented with threads).
- In docs I see that they the recommendation is to use concurrency per CPUs.
- If you want to separate different tasks to different workers..
Of course, you have price for that: more processes that takes more resources (CPU/Memory etc).
I found this question which is quite similar.
I assume that you are running both workers in the same machine. In that case I would recommend you to maintain one worker for a queue.
- Two workers for the same queue does not benefit you by any means. It would just increase the memory wastage.
- Two or more workers when you have multiple queues, to maintain priority or to allocate different number of cores to each worker.
- Two or more workers for a single queue is useful if you run the workers in different machines. The workers in different machines consumes tasks from same queue, you could allocate concurrency based on the cores available in each machine.
I do realise I responded 2+ years later. But I just thought I'll put it here for anyone who still has similar doubts.