Ecs service doesn't update the task definition
As I found out later on, the reason for not updating the task is that the desired count is set to 2 and there are only 2 EC2 instances available. So the ECS agent tries to retain the desired count even though the task has been updated.
Solution - Have one extra EC2 instance (in this case 3 EC2 instances). Or have one extra instance than the preferred number of tasks.
In this way the new task definition can run on the extra instance. After it is stabilized on the extra EC2 instance, the ECS agent will drain the connection on the other two instances for the old task definition, while the load-balancer redirect the traffic to the updated instance. The ECS agent replaces the old task definition with the new ones. And then it maintains the desired count as 2.
An alternative solution is to set the Minimum healthy percent
deployment option of the service to 0
, which results in the existing tasks being stopped prior to the new version being deployed.
This allows single ec2 instance clusters to be used, with the associated cost savings, etc
Not suitable for production as you will have downtime between deployments
To update a task-definition in the "tasks" running in the service You need to delete the tasks and Start a new task.
In this way, I solve the problem of updating task-definition in tasks
I have written the following code :
# Register a new Task definition
aws ecs register-task-definition --family testing-cluster --cli-input-json file://scripts/taskdefinition/testingtaskdef.json --region $AWS_REGION
# Update Service in the Cluster
aws ecs update-service --cluster $CLUSTER_NAME --service $SERVICE --task-definition testing-cluster --desired-count 1 --region $AWS_REGION
DECRIBED_SERVICE=$(aws ecs describe-services --region $AWS_REGION --cluster $CLUSTER_NAME --services $SERVICE);
CURRENT_DESIRED_COUNT=$(echo $DECRIBED_SERVICE | jq --raw-output ".services[0].desiredCount")
# - echo $CURRENT_DESIRED_COUNT
CURRENT_TASK_REVISION=$(echo $DECRIBED_SERVICE | jq -r ".services[0].taskDefinition")
echo "Current Task definition in Service" + $CURRENT_TASK_REVISION
CURRENT_RUNNING_TASK=$(echo $DECRIBED_SERVICE | jq -r ".services[0].runningCount")
echo $CURRENT_RUNNING_TASK
CURRENT_STALE_TASK=$(echo $DECRIBED_SERVICE | jq -r ".services[0].deployments | .[] | select(.taskDefinition != \"$CURRENT_TASK_REVISION\") | .taskDefinition")
echo "Task defn apart from current service Taskdefn" + $CURRENT_STALE_TASK
# - echo $CURRENT_STALE_TASK
tasks=$(aws ecs --region $AWS_REGION list-tasks --cluster $CLUSTER_NAME | jq -r '.taskArns | map(.[40:]) | reduce .[] as $item (""; . + $item + " ")')
echo "Tasks are as follows"
echo $tasks
TASKS=$(aws ecs --region $AWS_REGION describe-tasks --cluster $CLUSTER_NAME --task $tasks);
# - echo $TASKS
OLDER_TASK=$(echo $TASKS | jq -r ".tasks[] | select(.taskDefinitionArn!= \"$CURRENT_TASK_REVISION\") | .taskArn | split(\"/\") | .[1] ")
echo "Older Task running " + $OLDER_TASK
for old_task in $OLDER_TASK; do
aws ecs --region us-east-1 stop-task --cluster $CLUSTER_NAME --task $old_task
done
# Run new tasks with the updated new Task-definition
aws ecs --region $AWS_REGION run-task --cluster $CLUSTER_NAME --task-definition $CURRENT_TASK_REVISION