Adding logs to Airflow Logs
I think you can work around this by using the logging module and trusting the configuration to Airflow.
Something like:
import ...
dag = ...
def print_params_fn(**kwargs):
import logging
logging.info(kwargs)
return None
print_params = PythonOperator(task_id="print_params",
python_callable=print_params_fn,
provide_context=True,
dag=dag)
If you look at the PythonOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/python_operator.py#L80-L81, looks like there is no way to log STDOUT/STDERR from the python callable into the airflow logs.
However, if you look at the BashOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/bash_operator.py#L79-L94, the STDOUT/STDERR from there is logged along with the airflow logs. So, if logs are important to you, I suggest adding the python code in a separate file and calling it using the BashOperator.
Inside python callable for PythonOperator you can use:
import logging
LOGGER = logging.getLogger("airflow.task")
LOGGER.info("airflow.task >>> 2 - INFO logger test")
This will produce correct output like:
[2019-12-26 09:42:55,813] {operations.py:86} INFO - airflow.task >>> 2 - INFO logger test
For case with your custom logger:
LOGGER = logging.getLogger(__name__)
LOGGER.info("__name__ >>> 2 - INFO logger test")
You'll get duplication of formatting:
[2019-12-26 09:42:55,813] {logging_mixin.py:112} INFO - [2019-12-26 09:42:55,813] {operations.py:79} INFO - __name__ >>> 2 - INFO logger test