How to handle specific hostname like -h option in Dockerfile

This isn't generally possible in a Dockerfile.

Depending on the software, you might be able to do some kind of work-around. For example, you could try something like

RUN echo $(grep $(hostname) /etc/hosts | cut -f1) my.host.name >> /etc/hosts && install-software

By setting the hostname within the same RUN command as you install the software, it'll happen inside the same layer of the container. Docker will later overwrite the hostname and you'll have to set it anew when running, but your software might be OK with that.

If you have to do a lot of this, you might try Packer for building containers. It can build Docker containers, but doesn't use multiple layers. This makes it slower to rebuild, faster to download the built images, and makes it more convenient to do multiple operations on an image before freezing it into a container.

As for workaround, you can use docker-compose or docker stack to build your container with specific hostname, e.g.

version: '3'
services:
  all:
    image: testimage
    container_name: myname
    hostname: myhost
    build:
      context: .

Then run as:

docker-compose --build up

or using Docker stacks:

docker stack deploy -c docker-compose.yml mystack