Best practice for Rails App to run a long task in the background?

I have a very large volume site that generates lots of large CSV files. These sometimes take several minutes to complete. I do the following:

  • I have a jobs table with details of the requested file. When the user requests a file, the request goes in that table and the user is taken to a "jobs status" page that lists all of their jobs.
  • I have a rake task that runs all outstanding jobs (a class method on the Job model).
  • I have a separate install of rails on another box that handles these jobs. This box just does jobs, and is not accessible to the outside world.
  • On this separate box, a cron job runs all outstanding jobs every 60 seconds, unless jobs are still running from the last invocation.
  • The user's job status page auto-refreshes to show the status of the job (which is updated by the jobs box as the job is started, running, then finished). Once the job is done, a link appears to the results file.

It may be too heavy-duty if you just plan to have one or two running at a time, but if you want to scale... :)


The Workling plugin allow you to schedule background tasks in a queue (they would perform the lengthy task). As of version 0.3 you can ask a worker for its status, this would allow you to display some nifty progress bars.

Another cool feature with Workling is that the asynchronous backend can be switched: you can used DelayedJobs, Spawn (classic fork), Starling...