Running `airflow scheduler` launches 33 scheduler processes

Yes this is normal. These are scheduler processes. You can control this using below parameter in airflow.cfg

# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that should run simultaneously
# on this airflow installation
parallelism = 32

These are spawned from scheduler whose pid can be found in airflow-scheduler.pid file

so 32+1=33 processes that you are seeing.

Hope this clears out your doubt.

Cheers!


As of v1.10.3, this is what I found. My settings are:

parallelism = 32
max_threads = 4

There are a total of

  • 1 (main) +
  • 32 (executors) +
  • 1 (dag_processor_manager) +
  • 4 (dag processors)

= 38 processes!

Tags:

Python

Airflow