Ansible Cloudwatch rule reports failed invocations

I've lost hours with this too, same error and same confusion (Why there isn't a log for failed invokations?), I'm going to share my ""solution"", it will solve the problem to someone, and will help others to debug and find the ultimate solution.

Note: Be carefull, this could allow any AWS account execute your lambda functions

Since you got invoke the function by creating the rule target manually, I assume you added the invoke permission to the lambda from CloudWatch, however it looks like the Source Account ID is different when the event is created by cli/api and when is created by de AWS dashboard/console

If you are adding the Source Account condition in the lambda invoke permission from principal "events.amazonaws.com" to prevent any AWS account execute your lambdas just remove it (under your responsability!).

So, if your lambda policy looks like this:

{
    "Sid": "<sid>",
    "Effect": "Allow",
    "Principal": {
        "Service": "events.amazonaws.com"
    },
    "Action": "lambda:InvokeFunction",,
    "Condition": {
        "StringEquals": {
            "AWS:SourceAccount": "<account-id>"
        }
    },
    "Resource": "arn:aws:lambda:<region>:<account-id>:function:<lambda-function>"
}

Remove the "Condition" field

{
    "Sid": "sid",
    "Effect": "Allow",
    "Principal": {
        "Service": "events.amazonaws.com"
    },
    "Action": "lambda:InvokeFunction",,
    "Resource": "arn:aws:lambda:<region>:<account-id>:function:<lambda-function>"
}

And "maybe" it will work for you.

I think something weird it is happening with the cloudwatch event owner/creator data when the event is created by cli/api... maybe a bug? Not sure. I will keep working on it

To extend the answered here https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CWE_Troubleshooting.html#LAMfunctionNotInvoked. Since you are creating it via API you should add permission to Lambda as mentioned before. Without compromising security you could do the following:

Add rule with PutRule api call, it will return you

{
   "RuleArn": "string"
}

Use the RuleArn in Lambda AddPermission call

aws lambda add-permission \
--function-name MyFunction \
--statement-id MyId \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn-from-PutRule-request