Skip to main content

Can't connect to local registry 127.0.0.1:32000

Issue

  • Getting a connection refused when trying to upload images to the local registtry.
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2022-05-26 10:37:08.441849",
    "stderr": "Error: Get \"https://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused; Get \"http://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused",
    "stderr_lines": [
        "Error: Get \"https://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused; Get \"http://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused"
    ],

Environment

  • Red Hat Enterprise Linux 8.6
  • Element On-Premise Installer 2022-05.04

Resolution

Unmount the offending volume from the kubectl describe pod setup:

sudo umount /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863

If this succeeds, then you can issue:

kubectl delete pod -n container-registry registry

and watch the registry come back up.

If you cannot get the mounted volume to unmount, you may need to reboot to completely clear the issue.

Root Cause

The root cause is that the registry container will not start:

$ kubectl get pods -A
NAMESPACE            NAME                                         READY   STATUS             RESTARTS   AGE
kube-system          hostpath-provisioner-566686b959-jl2b4        1/1     Running            1          69m
...
container-registry   registry-9b57d9df8-kmks4                     0/1     ImagePullBackOff   0          44m

To figure out why this won't start, we need to run kubectl describe pod -n container-registry registry:

$ kubectl describe pod -n container-registry registry
Name:         registry-9b57d9df8-k7v2r
Namespace:    container-registry
Priority:     0
Node:         mynode/192.168.122.1
Start Time:   Thu, 26 May 2022 11:33:04 -0700
Labels:       app=registry
              pod-template-hash=9b57dea58
...
  Normal   BackOff           5m41s (x4 over 7m36s)  kubelet            Back-off pulling image "cdkbot/registry-amd64:2.6"
  Warning  Failed            5m41s (x4 over 7m36s)  kubelet            Error: ImagePullBackOff
  Warning  Failed            2m58s                  kubelet            Failed to pull image "cdkbot/registry-amd64:2.6": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/cdkbot/registry-amd64:2.6": failed to extract layer sha256:8aa4fcad5eeb286fe9696898d988dc85503c6392d1a2bd9023911fb0d6d27081: failed to unmount /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863: failed to unmount target /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863: device or resource busy: unknown

Looking at the above, we can seee that /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863 is busy and failing to unmount, thus causing our problem.

Back to top