Django Celery: Execute only one instance of a long-running process

You can use periodic tasks paired with a special lock which ensures the tasks are executed one at a time. Here is a sample implementation from Celery documentation:

http://ask.github.com/celery/cookbook/tasks.html#ensuring-a-task-is-only-executed-one-at-a-time

Your described method with scheduling task from the previous execution can stop the execution of tasks if there will be failure in one of them.

I personally solve this issue by caching a flag by a key like task.name + args

def perevent_run_duplicate(func):
    """
        this decorator set a flag to cache for a task with specifig args
        and wait to completion, if during this task received another call
        with same cache key will ignore to avoid of conflicts.
        and then after task finished will delete the key from cache

        - cache keys with a day of timeout

    """

    @wraps(func)
    def outer(self, *args, **kwargs):
        if cache.get(f"running_task_{self.name}_{args}", False):
            return
        else:
            cache.set(f"running_task_{self.name}_{args}", True, 24 * 60 * 60)
        try:
            func(self, *args, **kwargs)
        finally:
            cache.delete(f"running_task_{self.name}_{args}")

    return outer 

this decorator will manage task calls to prevent duplicate calls for a task by same args.