AWS Elastic Beanstalk: Add custom logs to CloudWatch?

Both bundlelogs.d and taillogs.d are logs retrieved from management console. What you want to do is extend default logs (e.g. eb-activity.log) to CloudWatch Logs. In order to extend the log stream, you need to add another configuration under /etc/awslogs/config/. The configuration should follow the Agent Configuration file Format.

I've successfully extended my logs for my custom ubuntu/nginx/php platform. Here is my extension file FYI. Here is an official sample FYI.

In your case, it could be like

files:
  "/etc/awslogs/config/my_app_log.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/app/current/logs/xxx.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/app/current/logs/xxx.log"]]}`
      log_stream_name = {instance_id}
      file = /var/app/current/logs/xxx.log*

Credits where due go to Sebastian Hsu and Abhyudit Jain.

This is the final config file I came up with for .ebextensions for our particular use case. Notes explaining some aspects are below the code block.

files:
  "/etc/awslogs/config/beanstalklogs_custom.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/tomcat8/catalina.out]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Fn::Select" : [ "1", { "Fn::Split" : [ "-", { "Ref":"AWSEBEnvironmentName" } ] } ] }, "var/log/tomcat8/catalina.out"]]}`
      log_stream_name = `{"Fn::Join":["--", [{ "Ref":"AWSEBEnvironmentName" }, "{instance_id}"]]}`
      file = /var/log/tomcat8/catalina.out*

services:
  sysvinit:
    awslogs:
      files:
        - "/etc/awslogs/config/beanstalklogs_custom.conf"

commands:
  rm_beanstalklogs_custom_bak:
    command: "rm beanstalklogs_custom.conf.bak"
    cwd: "/etc/awslogs/config"
    ignoreErrors: true

log_group_name

We have a standard naming scheme for our EB environments which is exactly environmentName-environmentType. I'm using { "Fn::Split" : [ "-", { "Ref":"AWSEBEnvironmentName" } ] } to split that into an array of two strings (name and type).

Then I use { "Fn::Select" : [ "1", <<SPLIT_OUTPUT>> ] } to get just the type string. Your needs would obviously differ, so you may only need the following:

      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "var/log/tomcat8/catalina.out"]]}`

log_stream_name

I'm using the Fn::Join function to join the EB environment name with the instance ID. Note that the instance ID template is a string that gets echoed exactly as given.

services

The awslogs service is restarted automatically when the custom conf file is deployed.

commands

When the files block overwrites an existing file, it creates a backup file, like beanstalklogs_custom.conf.bak. This block erases that backup file because awslogs service reads both files, potentially causing conflict.

Result

If you log in to an EC2 instance and sudo cat the file, you should see something like this. Note that all the Fn functions have resolved. If you find that an Fn function didn't resolve, check it for syntax errors.

[/var/log/tomcat8/catalina.out]
log_group_name = /aws/elasticbeanstalk/environmentType/var/log/tomcat8/catalina.out
log_stream_name = environmentName-environmentType--{instance_id}
file = /var/log/tomcat8/catalina.out*