synapse-haproxy container in CrashLoopBackOff state
Issue
We are seeing
[karl1@element ~]$ kubectl get pods -n element-onprem
NAME READY STATUS RESTARTS AGE
server-well-known-8c6bd8447-fts78 1/1 Running 2 39h
app-element-web-c5bd87777-745gh 1/1 Running 2 39h
postgres-0 1/1 Running 2 39h
instance-synapse-haproxy-5b4b55fc9c-jv7pp 0/1 CrashLoopBackOff 40 39h
instance-synapse-main-0 1/1 Running 6 39h
and the synapse-haproxy container never leaves the CrashLoopBackOff
state.
Environment
- Red Hat Enterprise Linux 8.5
- Element Enterprise On-Premise PoC Installer 0.6.1
Resolution
Add the following lines to /etc/security/limits.conf:
* soft nofile 4096100000
* hard nofile 10240100000
and reboot the box. After a reboot, the microk8s environment will come back up and the synapse-haproxy container should run without error.
Root Cause
Check the logs of synapse-haproxy with this command:
kubectl logs -n element-onprem instance-synapse-haproxy-5b4b55fc9c-jv7pp
You will want to replace the instance name with your specific instance. See if you have this message:
'[haproxy.main()] Cannot raise FD limit to 80034, limit 65536.'
If so, you have run out of open file descriptors and as such the container cannot start.