Skip to main content

Installer 2022-08.01 fails to pull element web into the cluster

Issue

  • The installer is failing to enable an add-on due to the namespace being deleted.
  • I'm seeing the following messages:
TASK [microk8s : enable addons] *********************************************************************
skipping: [localhost] => (item=dns)
skipping: [localhost] => (item=dashboard)
skipping: [localhost] => (item=helm3)
skipping: [localhost] => (item=host-access)
FAILED - RETRYING: [localhost]: enable addons (3 retries left).
FAILED - RETRYING: [localhost]: enable addons (2 retries left).
FAILED - RETRYING: [localhost]: enable addons (1 retries left).
failed: [localhost] (item=ingress) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "cmd": ["/snap/bin/microk8s.enable", "ingress"], "delta": "0:00:00.838920", "end": "2022-05-03 14:08:03.226221", "item": {"enabled": true, "name": "ingress"}, "msg": "non-zero return code", "rc": 1, "start": "2022-05-03 14:08:02.387301", "stderr": "Warning: Detected changes to resource ingress which is currently being deleted.

Environment

  • Element Enterprise Installer 0.6.1

Resolution

The following steps should help terminate the namespace that is stuck in the Terminating state.

Given in this example, we have the ingress namespace stuck, all the following commands will use that namespace. Please replace ingress with your namespace that won't terminate.

Generate a json description of the failing namespace with:

 kubectl get namespace ingress -o json > ingress.json

Now edit ingress.json and change this section:

      "spec": {
         "finalizers": [
           kubernetes
       ]
      },
      "status": {
          "phase": "Terminating"
      }

to (basically taking kubernetes out of the finalizers):

      "spec": {
         "finalizers": [
        ]
      },
      "status": {
          "phase": "Terminating"
      }

and then in another terminal on the machine, run:

kubectl proxy

and then back on your first terminal, run:

curl -k -H 'Content-Type: application/json' -X PUT --data-binary @ingress.json http://127.0.0.1:8001/api/v1/namespaces/ingress/finalize

at this point, the namespace should finish terminating and you should be clear to re-run the installer.

To finish, issue a Ctrl+C to the kubectl proxy command on the other terminal.

Root Cause

If you have tried to terminate a namespace previously that's attached to an add-on and have disabled the add-on before the namespace properly terminated, it can get stuck.

Full error message:

TASK [microk8s : enable addons] *********************************************************************
skipping: [localhost] => (item=dns)
skipping: [localhost] => (item=dashboard)
skipping: [localhost] => (item=helm3)
skipping: [localhost] => (item=host-access)
FAILED - RETRYING: [localhost]: enable addons (3 retries left).
FAILED - RETRYING: [localhost]: enable addons (2 retries left).
FAILED - RETRYING: [localhost]: enable addons (1 retries left).
failed: [localhost] (item=ingress) => {"ansible_loop_var": "item", "attempts": 3, "changed": true, "cmd": ["/snap/bin/microk8s.enable", "ingress"], "delta": "0:00:00.838920", "end": "2022-05-03 14:08:03.226221", "item": {"enabled": true, "name": "ingress"}, "msg": "non-zero return code", "rc": 1, "start": "2022-05-03 14:08:02.387301", "stderr": "Warning: Detected changes to resource ingress which is currently being deleted.\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": serviceaccounts "nginx-ingress-microk8s-serviceaccount" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": roles.rbac.authorization.k8s.io "nginx-ingress-microk8s-role" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": rolebindings.rbac.authorization.k8s.io "nginx-ingress-microk8s" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-load-balancer-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-tcp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-udp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated\nError from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": daemonsets.apps "nginx-ingress-microk8s-controller" is forbidden: unable to create new content in namespace ingress because it is being terminated", "stderr_lines": ["Warning: Detected changes to resource ingress which is currently being deleted.", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": serviceaccounts "nginx-ingress-microk8s-serviceaccount" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": roles.rbac.authorization.k8s.io "nginx-ingress-microk8s-role" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": rolebindings.rbac.authorization.k8s.io "nginx-ingress-microk8s" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-load-balancer-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-tcp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": configmaps "nginx-ingress-udp-microk8s-conf" is forbidden: unable to create new content in namespace ingress because it is being terminated", "Error from server (Forbidden): error when creating "/root/snap/microk8s/3058/tmp/temp.ingress.yaml": daemonsets.apps "nginx-ingress-microk8s-controller" is forbidden: unable to create new content in namespace ingress because it is being terminated"], "stdout": "Enabling Ingress\ningressclass.networking.k8s.io/public unchanged\nnamespace/ingress unchanged\nclusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole unchanged\nclusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s unchanged", "stdout_lines": ["Enabling Ingress", "ingressclass.networking.k8s.io/public unchanged", "namespace/ingress unchanged", "clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole unchanged", "clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s unchanged"]}
skipping: [localhost] => (item=metrics-server)
skipping: [localhost] => (item=rbac)
skipping: [localhost] => (item=registry)
skipping: [localhost] => (item=storage)
skipping: [localhost] => (item=openebs)

PLAY RECAP ******************************************************************************************
localhost                  : ok=33   changed=0    unreachable=0    failed=1    skipped=5    rescued=0    ignored=0
Back to top