AWS - subscribe multiple lambda logs to one elasticsearch service

The problem is that ElasticSearch 6.0.0 made a change that allows indices to only contain a single mapping type. (https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html) I assume you are running an ElasticSearch service instance that is using version 6.0.

The default Lambda JS file if created through the AWS console sets the index type to the log group name. An example of the JS file is on this gist (https://gist.github.com/iMilnb/27726a5004c0d4dc3dba3de01c65c575)

Line 86: action.index._type = payload.logGroup;

I personally have a modified version of that script in use and changed that line to be:

action.index._type = 'cwl';

I have logs from various different log groups streaming through to the same ElasticSearch instance. It makes sense to have them all be the same type since they are all CloudWatch logs versus having the type be the log group name. The name is also set in the @log_group field so queries can use that for filtering.

In my case, I did the following:

1. Deploy modified Lambda
2. Reindex today's index (cwl-2018.03.07 for example) to change the type for old documents from <log group name> to cwl
3. Entries from different log groups will now coexist.

You can also modify the generated Lambda code like below to make it work with multiple CW log groups. If the Lambda function can create different ES index for the different log streams coming under the same log groups, then we can avoid this problem. So, you need to find the Lambda function LogsToElasticsearch_<AWS-ES-DOMAIN-NAME>, then the function function transform(payload), and finally change the index name formation part like below.

    // index name format: cwl-YYYY.MM.DD
    //var indexName = [
        //'cwl-' + timestamp.getUTCFullYear(),              // year
        //('0' + (timestamp.getUTCMonth() + 1)).slice(-2),  // month
        //('0' + timestamp.getUTCDate()).slice(-2)          // day
    //].join('.');

    var indexName = [
        'cwl-' + payload.logGroup.toLowerCase().split('/').join('-') + '-' + timestamp.getUTCFullYear(),              // log group + year
        ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),  // month
        ('0' + timestamp.getUTCDate()).slice(-2)          // day
    ].join('.');