Broken DAG: (...) No module named docker

I got it resolved by installing docker-py==1.10.6 in the PyPI section of composer.

However, to get DockerOperator to work properly requires a bit more effort as the composer workers do not have access to the Docker daemon. Head to the GCP console and perform the following steps; after getting cluster credentials).

  1. Export current deployment config to file

    kubectl get deployment airflow-worker -o yaml --export > airflow-worker-config.yaml

  2. Edit airflow-worker-config.yaml (example link) to mount docker.sock and docker, grant privileged access to airflow-worker to run docker commands

  3. Apply deployment settings

    kubectl apply -f airflow-worker-config.yaml


As explained in other answers, the Docker Python client is not preinstalled in Cloud Composer environments. To install it, add it as a PyPI dependency in your environment's configuration.

Caveat: by default, DockerOperator will try to talk to the Docker API at /var/run/docker.sock to manage containers. This socket is not mounted inside Composer Airflow worker pods, and manually configuring it to do so is not recommended. Use of DockerOperator is only recommended in Composer if configured to talk to Docker daemons running outside of your environments.

To avoid more brittle configuration or surprises from bypassing Kubernetes (since it is responsible for managing containers across the entire cluster), you should use the KubernetesPodOperator. If you are launching containers into a GKE cluster (or the Composer environment's cluster), then you can use GKEPodOperator, which has more specific GCP-related parameters.


This means: whereever your Airflow instance is installed, the Python package named docker is missing.

If I configure my personal machine, I can install missing packages with

pip install docker

EDIT

Within the source code of the docker component https://airflow.incubator.apache.org/_modules/airflow/operators/docker_operator.html

there is an import statement:

from docker import Client, tls

So the new error cannot import name Client seems to me to be connected to a broken install or a wrong version of the docker package.