How to prevent a Hangfire recurring job from restarting after 30 minutes of continuous execution

Did you look at InvisibilityTimeout setting from the Hangfire docs?

Default SQL Server job storage implementation uses a regular table as a job queue. To be sure that a job will not be lost in case of unexpected process termination, it is deleted only from a queue only upon a successful completion.

To make it invisible from other workers, the UPDATE statement with OUTPUT clause is used to fetch a queued job and update the FetchedAt value (that signals for other workers that it was fetched) in an atomic way. Other workers see the fetched timestamp and ignore a job. But to handle the process termination, they will ignore a job only during a specified amount of time (defaults to 30 minutes).

Although this mechanism ensures that every job will be processed, sometimes it may cause either long retry latency or lead to multiple job execution. Consider the following scenario:

1. Worker A fetched a job (runs for a hour) and started it at 12:00.
2. Worker B fetched the same job at 12:30, because the default invisibility timeout was expired.
3. Worker C (did not fetch) the same job at 13:00, because (it will be deleted after successful performance.)

If you are using cancellation tokens, it will be set for Worker A at 12:30, and at 13:00 for Worker B. This may lead to the fact that your long-running job will never be executed. If you aren’t using cancellation tokens, it will be concurrently executed by WorkerA and Worker B (since 12:30), but Worker C will not fetch it, because it will be deleted after successful performance.

So, if you have long-running jobs, it is better to configure the invisibility timeout interval:

var options = new SqlServerStorageOptions
{
    InvisibilityTimeout = TimeSpan.FromMinutes(30) // default value
};

GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection string>", options);

As of Hangfire 1.5 this option is now Obsolete. Jobs that are being worked on are invisible to other workers.

Say goodbye to confusing invisibility timeout with unexpected background job retries after 30 minutes (by default) when using SQL Server. New Hangfire.SqlServer implementation uses plain old transactions to fetch background jobs and hide them from other workers.

Even after ungraceful shutdown, the job will be available for other workers instantly, without any delays.

I was having trouble finding documentation on how to do this properly for a Postgresql database, every example I was see is using sqlserver, I found how the invisibility timeout was a property inside the PostgreSqlStorageOptions object, I found this here : https://github.com/frankhommers/Hangfire.PostgreSql/blob/master/src/Hangfire.PostgreSql/PostgreSqlStorageOptions.cs#L36. Luckily through trial and error I was able to figure out that the UsePostgreSqlStorage has an overload to accept this object. For .Net Core 2.0 when you are setting up the hangfire postgresql DB in the ConfigureServices method in the startup class add this(the default timeout is set to 30 mins):

    services.AddHangfire(config =>
            config.UsePostgreSqlStorage(Configuration.GetConnectionString("Hangfire1ConnectionString"), new PostgreSqlStorageOptions {
                InvisibilityTimeout = TimeSpan.FromMinutes(720)

            }));

I had this problem when using Hangfire.MemoryStorage as the storage provider. With memory storage you need to set the FetchNextJobTimeout in the MemoryStorageOptions, otherwise by default jobs will timeout after 30 minutes and a new job will be executed.

var options = new MemoryStorageOptions
{
    FetchNextJobTimeout = TimeSpan.FromDays(1)
};
GlobalConfiguration.Configuration.UseMemoryStorage(options);