Stream AWS CloudWatch Log Group to Multiple AWS Elasticsearch Services

I've ran into this limitation as well. I have two Lambda's (doing different things) that need to subscribe to the same CloudWatch Log Group.

What I ended up using is to create one Lambda that subscribes to the Log Group and then proxy the events into an SNS topic.

Those two Lambdas are now subscribed to the SNS topic instead of the Log Group.

For filtering events, you could implement them inside the Lambda.

It's not a perfect solution but it's a functioning workaround until AWS allows multiple Lambdas to subscribe to the same CloudWatch Log Group.

Seems like AWS console limitation,

You can do it via command line,

aws logs put-subscription-filter \
    --log-group-name /aws/lambda/testfunc \
    --filter-name filter1 \
    --filter-pattern "Error" \
    --destination-arn arn:aws:lambda:us-east-1:<ACCOUNT_NUMBER>:function:SendToKinesis

You also need to add permissions as well.

Full detailed instructions,

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html

Hope it helps.

I was able to resolve the issue using a bit of a workaround through the Lambda function and also using the response provided by Kannaiyan.

I created the subscription to ES via the console, and then unsubscribed, and modified the Lambda function default code.

I declared two Elasticsearch endpoints:

var endpoint1 = '<ELASTICSEARCH ENDPOINT 1>';
var endpoint2 = '<ELASTICSEARCH ENDPOINT 2>';

Then, declared an array named "endpoint" with the contents of endpoint1 and endpoint2:

var endpoint = [endpoint1, endpoint2];

I modified the "post" function which calls the "buildRequest" function that then references "endpoint"...

function post(body, callback) {
  for (index = 0; index < endpoint.length; ++index) {
    var requestParams = buildRequest(endpoint[index], body);
...

So every time the "post" function is called it cycles through the array of endpoints.

Then, I modified the buildRequest function that is in charge of building the request. This function by default calls the endpoint variable, but since the "post" function cycles through the array, I renamed "endpoint" to "endpoint_xy" to make sure its not calling the global variable and instead takes the variable being inputted into the function:

function buildRequest(endpoint_xy, body) {
  var endpointParts = endpoint_xy.match(/^([^\.]+)\.?([^\.]*)\.?([^\.]*)\.amazonaws\.com$/);
...

Finally, I used the response provided by Kannaiyan on using the AWS CLI to implement the subscription to the logs, but corrected a few variables:

aws logs put-subscription-filter \
--log-group-name <LOG GROUP NAME> \
--filter-name <FILTER NAME> 
--filter-pattern <FILTER PATTERN> 
--destination-arn <LAMBDA FUNCTION ARN>

I kept the filters completely open for now, but will now code the filter directly into the Lambda function like dashmug suggested. At least I can split one log to two ES clusters.

Thank you everyone!