"msg": "non-zero return code",
"start": "2022-05-26 10:37:08.441849",
"stderr": "Error: Get \"https://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused; Get \"http://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused",
"Error: Get \"https://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused; Get \"http://localhost:32000/v2/\": dial tcp [::1]:32000: connect: connection refused"
Unmount the offending volume from the kubectl describe pod setup:
kubectl describe pod
sudo umount /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863
If this succeeds, then you can issue:
kubectl delete pod -n container-registry registry
and watch the registry come back up.
If you cannot get the mounted volume to unmount, you may need to reboot to completely clear the issue.
The root cause is that the registry container will not start:
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system hostpath-provisioner-566686b959-jl2b4 1/1 Running 1 69m
container-registry registry-9b57d9df8-kmks4 0/1 ImagePullBackOff 0 44m
To figure out why this won't start, we need to run kubectl describe pod -n container-registry registry:
kubectl describe pod -n container-registry registry
$ kubectl describe pod -n container-registry registry
Start Time: Thu, 26 May 2022 11:33:04 -0700
Normal BackOff 5m41s (x4 over 7m36s) kubelet Back-off pulling image "cdkbot/registry-amd64:2.6"
Warning Failed 5m41s (x4 over 7m36s) kubelet Error: ImagePullBackOff
Warning Failed 2m58s kubelet Failed to pull image "cdkbot/registry-amd64:2.6": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/cdkbot/registry-amd64:2.6": failed to extract layer sha256:8aa4fcad5eeb286fe9696898d988dc85503c6392d1a2bd9023911fb0d6d27081: failed to unmount /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863: failed to unmount target /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863: device or resource busy: unknown
Looking at the above, we can seee that /var/snap/microk8s/common/var/lib/containerd/tmpmounts/containerd-mount490181863 is busy and failing to unmount, thus causing our problem.