How to configure autoscaling on docker swarm?

Scaling up is pretty easy. You can just keep on scheduling containers when needed. Then you just create a script that looks for pending container and scales up the cluster. For example if you're using the official CloudFormation template on aws for swarm, you can just change the desired number on the autoscaling group. An example iteration script could looks like this:

services=$(docker service ls --format '{{.ID}}')
for service in $services; do
  tasks=$(docker service ps $service --format '{{.ID}}')
  for task in $tasks; do
    if docker inspect $task --format '{{.Status}}' | grep 'insufficient resources' 1>/dev/null; then
      scale-up-cmd
    fi
  done
done

Short answer: There is no easy way to do this with Docker Swarm for now.

Docker Swarm (or Swarm mode) does not support auto-scaling machines out of the box. You'd need to use another solution for that like docker-machine to create machines (with docker) on your infrastructure and link these to the existing Swarm cluster (with docker swarm join).

This will involve a lot of scripting but the idea is to monitor the cluster for CPU / Memory / Network usage (with top or monit) and once it goes beyond a threshold (say 70% of total cluster resources), you trigger a script calling docker-machine to scale up the cluster. Using the same idea you can also scale down by draining and removing nodes (preferably Agent nodes) from the existing swarm cluster once your are below the lower threshold.

You need to make sure you are monitoring for sustained resource usage if you want to use this criteria or you will have your Infrastructure spawning and destroying nodes from the frequent and sudden changes in resource usage.

You can define a lower bound and an upper bound for machines in the cluster to keep things under control.

Note that Swarm requires at least 3 Manager nodes (recommended 5) to maintain a quorum for the Distributed Consensus algorithm. So the minimum recommended lower bound is 5 nodes (which you can extend with Agent nodes as resources are incrementally being used by services).

To some extent, you can also take a look at Docker InfraKit or Terraform for Infrastructure automation and Health monitoring.

Update: There is now a promising cross-platform autoscaler that supports Swarm Mode task auto-scaling: Orbiter. Although still nothing out-of-the-box yet for service/machine autoscaling.