How to configure Celery Worker and Beat for Email Reporting in Apache Superset running on Docker?
I managed to solve it by altering the CeleryConfig
implementation, and adding a beat service to 'docker-compose.yml'
New CeleryConfig
class in 'superset_config.py':
REDIS_HOST = get_env_variable("REDIS_HOST")
REDIS_PORT = get_env_variable("REDIS_PORT")
class CeleryConfig(object):
BROKER_URL = "redis://%s:%s/0" % (REDIS_HOST, REDIS_PORT)
CELERY_IMPORTS = (
'superset.sql_lab',
'superset.tasks',
)
CELERY_RESULT_BACKEND = "redis://%s:%s/1" % (REDIS_HOST, REDIS_PORT)
CELERY_ANNOTATIONS = {
'sql_lab.get_sql_results': {
'rate_limit': '100/s',
},
'email_reports.send': {
'rate_limit': '1/s',
'time_limit': 120,
'soft_time_limit': 150,
'ignore_result': True,
},
}
CELERY_TASK_PROTOCOL = 1
CELERYBEAT_SCHEDULE = {
'email_reports.schedule_hourly': {
'task': 'email_reports.schedule_hourly',
'schedule': crontab(minute='1', hour='*'),
},
}
Changes in 'docker-compose.yml':
superset-worker:
build: *superset-build
command: ["celery", "worker", "--app=superset.tasks.celery_app:app", "-Ofair"]
env_file: docker/.env
restart: unless-stopped
depends_on: *superset-depends-on
volumes: *superset-volumes
superset-beat:
build: *superset-build
command: ["celery", "beat", "--app=superset.tasks.celery_app:app", "--pidfile=", "-f", "/app/celery_beat.log"]
env_file: docker/.env
restart: unless-stopped
depends_on: *superset-depends-on
volumes: *superset-volumes