Installer 2022-08.01 fails to pull element web into the cluster
Issue
TheAfterinstallerrunning the setup, the element web pod isfailingstucktoinenableImagePullBackoffan add-on due to the namespace being deleted.I'm seeing the following messages:
TASK [microk8s : enable addons] *********************************************************************
skipping: [localhost] => (item=dns)
skipping: [localhost] => (item=dashboard)
skipping: [localhost] => (item=helm3)
skipping: [localhost] => (item=host-access)
FAILED - RETRYING: [localhost]: enable addons (3 retries left).
FAILED - RETRYING: [localhost]: enable addons (2 retries left).
FAILED - RETRYING: [localhost]: enable addons (1 retries left).
failed: [localhost] (item=ingress) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "cmd": ["/snap/bin/microk8s.enable", "ingress"], "delta": "0:00:00.838920", "end": "2022-05-03 14:08:03.226221", "item": {"enabled": true, "name": "ingress"}, "msg": "non-zero return code", "rc": 1, "start": "2022-05-03 14:08:02.387301", "stderr": "Warning: Detected changes to resource ingress which is currently being deleted.
Environment
- Element Enterprise Installer
0.6.12022-08.01
Resolution
The following steps should help terminate the namespace thatIt is stucknecessary into theuncomment Terminatingstate.
Givenany inrandom thisvalue example, we have the ingress namespace stuck, allto the following commandsvariables willin usesecret.yaml that namespace. Please replace ingress with your namespace that won't terminate.
Generate a json description of the failing namespace with:
kubectl get namespace ingress -o json > ingress.json
Now edit ingress.json and change this section:
"spec": {
"finalizers": [
kubernetes
]
},
"status": {
"phase": "Terminating"
}
to (basically taking kubernetes out of the finalizers):
dockerhub_username: "spec":anyvalue
{dockerhub_token: "finalizers": [
]
},
"status": {
"phase": "Terminating"
}anyvalue
and then in another terminal on the machine, run:
kubectl proxy
and then back on your first terminal, run:
curl -k -H 'Content-Type: application/json' -X PUT --data-binary @ingress.json http://127.0.0.1:8001/api/v1/namespaces/ingress/finalize
at this point, the namespace should finish terminating and you should be clear to re-run the installer.
To finish, issue a Ctrl+C to the kubectl proxy command on the other terminal.
Root Cause
IfVersion you2022-08.01 haveuses triedan element web image hosted in ems-image-store. A defect appeared during the migration of the image, and the installers looks for the variables dockerhub_username and dockerhub_token to terminateknow if it has to configure docker secrets into the cluster.
Once this secrets are given a namespacevalue, previouslythe that'sinstaller attachedcorrectly uses ems_image_store_username and ems_image_store_token to an add-on and have disabledconfigure the add-ondocker before the namespace properly terminated, it can get stuck.secret.
Full error message:
TASK [microk8s : enable addons] *********************************************************************
skipping: [localhost] => (item=dns)
skipping: [localhost] => (item=dashboard)
skipping: [localhost] => (item=helm3)
skipping: [localhost] => (item=host-access)
FAILED - RETRYING: [localhost]: enable addons (3 retries left).
FAILED - RETRYING: [localhost]: enable addons (2 retries left).
FAILED - RETRYING: [localhost]: enable addons (1 retries left).
failed: [localhost] (item=ingress) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "cmd": ["/snap/bin/microk8s.enable", "ingress"], "delta": "0:00:00.838920", "end": "2022-05-03 14:08:03.226221", "item": {"enabled": true, "name": "ingress"}, "msg": "non-zero return code", "rc": 1, "start": "2022-05-03 14:08:02.387301", "stderr": "Warning: Detected changes to resource ingress which is currently being deleted.\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": serviceaccounts "nginx-ingress-microk8s-serviceaccount" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": roles.rbac.authorization.k8s.io "nginx-ingress-microk8s-role" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": rolebindings.rbac.authorization.k8s.io "nginx-ingress-microk8s" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-load-balancer-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-tcp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-udp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": daemonsets.apps "nginx-ingress-microk8s-controller" is forbidden: unable to create new content in namespace ingress because it is being terminated", "stderr_lines": ["Warning: Detected changes to resource ingress which is currently being deleted.", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": serviceaccounts "nginx-ingress-microk8s-serviceaccount" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": roles.rbac.authorization.k8s.io "nginx-ingress-microk8s-role" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": rolebindings.rbac.authorization.k8s.io "nginx-ingress-microk8s" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-load-balancer-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-tcp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-udp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": daemonsets.apps "nginx-ingress-microk8s-controller" is forbidden: unable to create new content in namespace ingress because it is being terminated"], "stdout": "Enabling Ingress\ningressclass.networking.k8s.io/public unchanged\nnamespace/ingress unchanged\nclusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole unchanged\nclusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s unchanged", "stdout_lines": ["Enabling Ingress", "ingressclass.networking.k8s.io/public unchanged", "namespace/ingress unchanged", "clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole unchanged", "clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s unchanged"]}
skipping: [localhost] => (item=metrics-server)
skipping: [localhost] => (item=rbac)
skipping: [localhost] => (item=registry)
skipping: [localhost] => (item=storage)
skipping: [localhost] => (item=openebs)
PLAY RECAP ******************************************************************************************
localhost : ok=33 changed=0 unreachable=0 failed=1 skipped=5 rescued=0 ignored=0