Connection refused on pushing a docker image

More comments about

  • Kubenetes(K8s) / Minikube
  • docker / image / registry, container

If you are using Minikube, and want to pull down an image from 127.0.0.1:5000,

then you meet the errors below:

Failed to pull image "127.0.0.1:5000/nginx_operator:latest": rpc error: code = Unknown desc = Error response from daemon: Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused

Full logs:

$ kubectl describe pod/your_pod
...
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m29s                default-scheduler  Successfully assigned tj-blue-whale-05-system/tj-blue-whale-05-controller-manager-6c8f564575-kwxdv to minikube
  Normal   Pulled     2m25s                kubelet            Container image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0" already present on machine
  Normal   Created    2m24s                kubelet            Created container kube-rbac-proxy
  Normal   Started    2m23s                kubelet            Started container kube-rbac-proxy
  Normal   BackOff    62s (x5 over 2m22s)  kubelet            Back-off pulling image "127.0.0.1:5000/nginx_operator:latest"
  Warning  Failed     62s (x5 over 2m22s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    48s (x4 over 2m23s)  kubelet            Pulling image "127.0.0.1:5000/nginx_operator:latest"
  Warning  Failed     48s (x4 over 2m23s)  kubelet            Failed to pull image "127.0.0.1:5000/nginx_operator:latest": rpc error: code = Unknown desc = Error response from daemon: Get http://127.0.0.1:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused
  Warning  Failed     48s (x4 over 2m23s)  kubelet            Error: ErrImagePull

Possible root cause:

The registry must be setup inside the Minikube side instead of your host side.

i.e.

  • host: registry (127.0.0.1:5000)
  • minikube: no registry (the K8s could not find your image)

How to check?

Step1: check your Minikube container

$ docker ps -a
CONTAINER ID   IMAGE                                           ...   STATUS        PORTS                                                                                                      NAMES
8c6f49491dd6   gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4   ...   Up 15 hours   127.0.0.1:49156->22/tcp, 127.0.0.1:49155->2376/tcp, 127.0.0.1:49154->5000/tcp, 127.0.0.1:49153->8443/tcp   minikube

# your Minikube is under running
# host:49154 <--> minikube:5000
# where: 
#  - port 49154 was allocated randomly by the docker service
#  - port 22: for ssh
#  - port 2376: for docker service
#  - port 5000: for registry (image repository)
#  - port 8443: for Kubernetes

Step2: login to your Minikube

$ minikube ssh

docker@minikube:~$ curl 127.0.0.1:5000
curl: (7) Failed to connect to 127.0.0.1 port 5000: Connection refused

# setup
# =====
# You did not setup the registry.
# Let's try to setup it.
docker@minikube:~$ docker run --restart=always -d -p 5000:5000 --name registry registry:2

# test
# ====
# test the registry using the following commands
docker@minikube:~$ curl 127.0.0.1:5000

docker@minikube:~$ curl 127.0.0.1:5000/v2
<a href="/v2/">Moved Permanently</a>.

docker@minikube:~$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":[]}
# it's successful

docker@minikube:~$ exit
logout

Step3: build your image, and push it into the registry of your Minikube

# Let's take nginx as an example. (You can build your own image)
$ docker pull nginx

# modify the repository (the source and the name)
$ docker tag nginx 127.0.0.1:49154/nginx_operator

# check the new repository (source and the name)
$ docker images | grep nginx
REPOSITORY                       TAG       IMAGE ID       CREATED        SIZE
127.0.0.1:49154/nginx_operator   latest    ae2feff98a0c   3 weeks ago    133MB

# push the image into the registry of your Minikube
$ docker push 127.0.0.1:49154/nginx_operator

Step4: login to your Minikube again

$ minikube ssh

# check the registry
$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":["nginx_operator"]}
# it's successful

# get the image info
$ curl 127.0.0.1:5000/v2/nginx_operator/manifests/latest

docker@minikube:~$ exit
logout



Customize exposed ports of Minikube

if you would like to use the port 5000 on the host side instead of using 49154 (which was allocated randomly by the docker service)

i.e. host:5000 <--> minikube:5000

you need to recreate a minikube instance with the flag --ports

# delete the old minikube instance
$ minkube delete

# create a new one (with the docker driver)
$ minikube start --ports=5000:5000 --driver=docker
# or 
$ minikube start --ports=127.0.0.1:5000:5000 --driver=docker
$ docker ps -a
CONTAINER ID   IMAGE                                           COMMAND                  CREATED              STATUS              PORTS                                                                                                                              NAMES
5d1e5b61a3bf   gcr.io/k8s-minikube/kicbase:v0.0.15-snapshot4   "/usr/local/bin/entr…"   About a minute ago   Up About a minute   0.0.0.0:5000->5000/tcp, 127.0.0.1:49162->22/tcp, 127.0.0.1:49161->2376/tcp, 127.0.0.1:49160->5000/tcp, 127.0.0.1:49159->8443/tcp   minikube

$ docker port minikube
22/tcp -> 127.0.0.1:49162
2376/tcp -> 127.0.0.1:49161
5000/tcp -> 127.0.0.1:49160
5000/tcp -> 0.0.0.0:5000
8443/tcp -> 127.0.0.1:49159
  • you can see: 0.0.0.0:5000->5000/tcp

Re-test your registry in the Minikube

# in the host side
$ docker pull nginx
$ docker tag nginx 127.0.0.1:5000/nginx_operator
$ docker ps -a
$ docker push 127.0.0.1:5000/nginx_operator

$ minikube ssh
docker@minikube:~$ curl 127.0.0.1:5000/v2/_catalog
{"repositories":["nginx_operator"]}

# Great!

Connection refused usually means that the service you are trying to connect to isn't actually up and running like it should. There could be other reasons as outlined in this question, but essentially, for your case, it simply means that the registry is not up yet.

Wait for the registry container to be created properly before you do anything else - docker run -d -p 5000:5000 --restart=always --name registry registry:2 that creates a local registry from the official docker image.

Make sure that the registry container is up by running docker ps | grep registry, and then proceed further.

Tags:

Docker

Ubuntu