How to limit Airflow to run only one instance of a DAG run at a time?
You've put the 'max_active_runs': 1
into the default_args
parameter and not into the correct spot.
max_active_runs
is a constructor argument for a DAG and should not be put into the default_args
dictionary.
Here is an example DAG that shows where you need to move it to:
dag_args = {
'owner': 'Owner',
# 'max_active_runs': 1, # <--- Here is where you had it.
'depends_on_past': False,
'start_date': datetime(2018, 01, 1, 12, 00),
'email_on_failure': False
}
sched = timedelta(hours=1)
dag = DAG(
job_id,
default_args=dag_args,
schedule_interval=sched,
max_active_runs=1 # <---- Here is where it is supposed to be
)
If the tasks that your dag is running are actually sub-dags then you may need to pass max_active_runs
into the subdags too but not 100% sure on this.
I changed to put max_active_runs
as an argument of DAG()
instead of in default_arguments, and it worked.
Thanks SimonD for giving me the idea, though not directly pointing to it in your answer.