Airflow : dag run with execution_date = trigger_date = fixed_schedule
Take a quick look at my answer with start times and execution_date examples.
You want to run every Monday at 8am.
So this part is going to stay the same:
schedule_interval: '0 8 * * MON',
You want it to run it's first run on 2018-03-19, since the first run occurs at the end of the first full schedule period after the start date, you should change your start date to:
start_date: datetime(2018,03,12),
You will have to live with the fact that Airflow will name your DagRuns with the start of each period and pass in macros based on the execution_date
set to the start of the interval period. Adjust your logic accordingly.
Your first run will start after 2018-03-19T08:00:00.0Z
and the execution_date
, every other macro that depends on it, and name of the DagRun will be 2018-03-12T08:00:00.0Z
So long as you understand what to expect from the execution_date
and you don't try to base your time off of datetime.now()
your DAGs will be able to be idempotent in operation. Feel free to make a new variable like my_execution_date = execution_date + datetime.timedelta(7)
within any PythonOperator
or custom operator (you get execution_date from the context of the task), use template statements like {{ (execution_date + macros.timedelta(7)).strftime('%Y%m%d') }}
or {{ macros.ds_add(ds, 7) }}
, or use the next_execution_date
.
You can even add a dag level user_defined_macros
like {'dt':lambda d: d+datetime.timedelta(days=7)}
to enable {{ dt(execution_date) }}
. And recently user_defined_filters
were added like {'dt':lambda d: d+datetime.timedelta(days=7)}
enabling {{ execution_date | dt }}
. The next_ds
and next_execution_date
would be easier for your purposes.
While thinking about templating, you may as well read up on the built-in stuff out there: http://jinja.pocoo.org/docs/2.10/templates/#builtin-filters