Running `airflow scheduler` launches 33 scheduler processes
Yes this is normal. These are scheduler processes. You can control this using below parameter in airflow.cfg
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32
These are spawned from scheduler whose pid can be found in airflow-scheduler.pid file
so 32+1=33 processes that you are seeing.
Hope this clears out your doubt.
Cheers!
As of v1.10.3, this is what I found. My settings are:
parallelism = 32
max_threads = 4
There are a total of
- 1 (main) +
- 32 (executors) +
- 1 (dag_processor_manager) +
- 4 (dag processors)
= 38 processes!