Broken DAG: (...) No module named docker
I got it resolved by installing docker-py==1.10.6 in the PyPI section of composer.
However, to get DockerOperator to work properly requires a bit more effort as the composer workers do not have access to the Docker daemon. Head to the GCP console and perform the following steps; after getting cluster credentials).
Export current deployment config to file
kubectl get deployment airflow-worker -o yaml --export > airflow-worker-config.yaml
Edit airflow-worker-config.yaml (example link) to mount docker.sock and docker, grant privileged access to airflow-worker to run docker commands
Apply deployment settings
kubectl apply -f airflow-worker-config.yaml
As explained in other answers, the Docker Python client is not preinstalled in Cloud Composer environments. To install it, add it as a PyPI dependency in your environment's configuration.
Caveat: by default, DockerOperator
will try to talk to the Docker API at /var/run/docker.sock
to manage containers. This socket is not mounted inside Composer Airflow worker pods, and manually configuring it to do so is not recommended. Use of DockerOperator
is only recommended in Composer if configured to talk to Docker daemons running outside of your environments.
To avoid more brittle configuration or surprises from bypassing Kubernetes (since it is responsible for managing containers across the entire cluster), you should use the KubernetesPodOperator
. If you are launching containers into a GKE cluster (or the Composer environment's cluster), then you can use GKEPodOperator
, which has more specific GCP-related parameters.
This means: whereever your Airflow instance is installed, the Python package named docker
is missing.
If I configure my personal machine, I can install missing packages with
pip install docker
EDIT
Within the source code of the docker component https://airflow.incubator.apache.org/_modules/airflow/operators/docker_operator.html
there is an import statement:
from docker import Client, tls
So the new error cannot import name Client
seems to me to be connected to a broken install or a wrong version of the docker
package.