Using both gevent (or eventlet) and prefork workers with Celery

So currently the only way to specify per worker Pool implementations is by running independent celery worker commands:

$ celery -A proj worker start -P gevent -Q:queue1 -c 500
$ celery -A proj worker start -P prefork -Q:queue2 -c 4

celery multi does not support -P:worker1 gevent, -P:worker2 prefork. This makes things difficult when using the init.d script provided within the Celery docs to daemonize celeryd. So you either have to modify the init.d script or use something like Supervisor.


There is a way to start different implementations using celery multi. Just like Amir R. said about -Q option.

celery multi start 2 -A default_prefork_queue -Q:2 gevent_queue_name -P:2 gevent -c:2 1000

By default it's prefork option selected.

In example above celery creates one worker with gevent pool with 1000 limit and one prefork worker with default limit of processes.

More examples here