Handling long running tasks in pika / RabbitMQ

I encounter the same problem you had.
My solution is:

1. ture off the heartbeat on the server side
2. evaluate the maximum time the task can possible take
3. set the client heartbeat timeout to the time got from step2

Why this?

As i test with the following cases:

case one

1. server heartbeat turn on, 1800s
2. client unset

I still get error when task running for a very long time -- >1800

case two

1. turn off server heartbeat
2. turn off client heartbeat

There is no error on client side, except one problem--when the client crashes(my os restart on some faults), the tcp connection still can be seen at the Rabbitmq Management plugin. And it is confusing.

case three

1. turn off server heartbeat
2. turn on client heartbeat, set it to the foresee maximum run time

In this case, i can dynamic change every heatbeat on indivitual client. In fact, i set heartbeat on the machines crashed frequently.Moreover, i can see offline machine through the Rabbitmq Manangement plugin.

Environment

OS: centos x86_64
pika: 0.9.13
rabbitmq: 3.3.1

Please don't disable heartbeats!

As of Pika 0.12.0, please use the technique described in this example code to run your long-running task on a separate thread and then acknowledge the message from that thread.

---

_{NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.}

For now, your best bet is to turn off heartbeats, this will keep RabbitMQ from closing the connection if you're blocking for too long. I am experimenting with pika's core connection management and IO loop running in a background thread but it's not stable enough to release.

In pika v1.1.0 this is ConnectionParameters(heartbeat=0)

1. You can periodic call connection.process_data_events() in your long_running_task(connection), this function will send heartbeat to server when it is been called, and keep the pika client away from close.
2. Set the heartbeat value greater than call connection.process_data_events() period in your pika BlockingConnection.