synapse-haproxy container in CrashLoopBackOff state

Issue

We are seeing

[karl1@element ~]$ kubectl get pods -n element-onprem
NAME                                        READY   STATUS             RESTARTS   AGE
server-well-known-8c6bd8447-fts78           1/1     Running            2          39h
app-element-web-c5bd87777-745gh             1/1     Running            2          39h
postgres-0                                  1/1     Running            2          39h
instance-synapse-haproxy-5b4b55fc9c-jv7pp   0/1     CrashLoopBackOff   40         39h
instance-synapse-main-0                     1/1     Running            6          39h

and the synapse-haproxy container never leaves the CrashLoopBackOff state.

Environment

Red Hat Enterprise Linux 8.5 Element Enterprise On-Premise PoC Installer 0.6.1

Resolution

Add the following lines to /etc/security/limits.conf:

*              soft    nofile  4096
*              hard    nofile  10240

and reboot the box. After a reboot, the microk8s environment will come back up and the synapse-haproxy container should run without error.

Root Cause

Check the logs of synapse-haproxy with this command:

kubectl logs -n element-onprem instance-synapse-haproxy-5b4b55fc9c-jv7pp

You will want to replace the instance name with your specific instance. See if you have this message:

'[haproxy.main()] Cannot raise FD limit to 80034, limit 65536.'

If so, you have run out of open file descriptors and as such the container cannot start.