synapse-haproxy container in CrashLoopBackOff state
Issue
We are seeing
[karl1@element ~]$ kubectl get pods -n element-onprem
NAME                                        READY   STATUS             RESTARTS   AGE
server-well-known-8c6bd8447-fts78           1/1     Running            2          39h
app-element-web-c5bd87777-745gh             1/1     Running            2          39h
postgres-0                                  1/1     Running            2          39h
instance-synapse-haproxy-5b4b55fc9c-jv7pp   0/1     CrashLoopBackOff   40         39h
instance-synapse-main-0                     1/1     Running            6          39h
and the synapse-haproxy container never leaves the CrashLoopBackOff state.
Environment
- Red Hat Enterprise Linux 8.5
- Element Enterprise On-Premise PoC Installer 0.6.1
Resolution
Add the following lines to /etc/security/limits.conf:
*              soft    nofile  4096
*              hard    nofile  10240
and reboot the box. After a reboot, the microk8s environment will come back up and the synapse-haproxy container should run without error.
Root Cause
Check the logs of synapse-haproxy with this command:
kubectl logs -n element-onprem instance-synapse-haproxy-5b4b55fc9c-jv7pp
You will want to replace the instance name with your specific instance. See if you have this message:
'[haproxy.main()] Cannot raise FD limit to 80034, limit 65536.'
If so, you have run out of open file descriptors and as such the container cannot start.
