Using both gevent (or eventlet) and prefork workers with Celery
So currently the only way to specify per worker Pool implementations is by running independent celery worker
commands:
$ celery -A proj worker start -P gevent -Q:queue1 -c 500
$ celery -A proj worker start -P prefork -Q:queue2 -c 4
celery multi
does not support -P:worker1 gevent, -P:worker2 prefork
. This makes things difficult when using the init.d script provided within the Celery docs to daemonize celeryd. So you either have to modify the init.d script or use something like Supervisor.
There is a way to start different implementations using celery multi
. Just like Amir R. said about -Q
option.
celery multi start 2 -A default_prefork_queue -Q:2 gevent_queue_name -P:2 gevent -c:2 1000
By default it's prefork
option selected.
In example above celery creates one worker with gevent
pool with 1000 limit and one prefork
worker with default limit of processes.
More examples here