Why every time Elastic Beanstalk issues a command to its instance it always timed out?

I had this problem a few times. It seems to be affect only particular instances. So it can be solved by terminating the EC2 instance (done via the EC2 page on the Management Console). Thereafter, Elastic Beanstalk will detect that there are 0 healthy instances and automatically launch a new one.

If this is a production environment and you have only 1 instance and you want minimal down time

  1. configure minimum instances to 2, and Beanstalk will launch another instance for you.
  2. terminate the problematic instance via EC2 tab, Beanstalk will launch another instance for you because minimum instance is 2
  3. configure minimum instance back to 1, Beanstalk will remove one of your two instances.

By default Elastic Beanstalk "throws a timeout exception" after 8 minutes (480 seconds defined in settings) if your commands did not complete in time. You can set an higher time up to 30 minutes (1800 seconds).

{
    "Namespace": "aws:elasticbeanstalk:command",
    "OptionName": "Timeout",
    "Value": "1800"
}

Read here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options.html


Beanstalk deployment (and other features like Get Logs) work by sending SQS commands to instances. SQS client is deployed to instances and checks queue about every 20 secs (see /var/log/cfn-hup.log): 2018-05-30 10:42:38,605 [DEBUG] Receiving messages for queue https://sqs.us-east-2.amazonaws.com/124386531466/93b60687a33e19...

If SQS Client crashes or has network problems on t1/t2 instances then it will not be able to receive commands from Beanstalk, and deployment would time out. Rebooting instance restarts SQS Client, and it can receive commands again.

An easier way to fix SQS Client is to restart cfn-hup service:

sudo service cfn-hup restart