Google Cloud Composer(Airflow) - dataflow job inside a DAG executes successfully, but the DAG fails

This is a confirmed bug in the sdk version 2.20.0

https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/dataflow.py#L47

Please use 2.19.0 version and it should work correctly.

 <dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
  <version>2.19.0</version>
  <scope>runtime</scope>
</dependency>