ESS LTS 24.04 Change Logs and Upgrade Notes
LTS 24.04 Changelogs and important Update Notes, always check here before upgrading!
Upgrade Notes for the 24.04 LTS
If you are planning on upgrading to the LTS we always recommend upgrading to the latest LTS patch version, however you should be aware of all significant upgrade notes from each prior patch version. They have been collated for convenience below, you can find the full changelogs of each release there after.
24.04.20-gui | The required Python versions are now 3.9, 3.10, 3.11. These are available on all supported OS distributions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.9 packages are available in their package mirrors. Alternatively, Python 3.9, 3.10, or 3.11 can be preinstalled on the server in all situations. |
24.04.05-gui | Major Change: The standalone installer now upgrades microk8s gracefully automatically. The microk8s upgrade procedure does not anymore involve an uninstall/reinstall of microk8s. It now will automatically upgrade microk8s to the expected version, and the flag --upgrade-cluster has been removed.Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will have to be reconfigured. During the upgrade, microk8s will restart, and add-ons will be disabled to force an upgrade. It can induce a small downtime of a couple of minutes. |
24.04.01-gui | This release contains an important Synapse security fix with a backwards incompatible change. Please note that simply reverting this ESS release is not possible. Please ensure to have a working backups before upgrading as downgrading is not a possibility from this release. |
24.04.23-gui
Security Issues
Enterprise | Upgrade Element Web to v1.11.85, fixes CVE-2024-50336, CVE-2024-51749, and CVE-2024-51750. |
Bug Fixes
Enterprise / Starter | Improve the reliability of reverting to the upstream microk8s stop script. |
24.04.22-gui
New Features
Enterprise / Starter | Fix an issue where load would not be properly balanced across Synchrotron workers. |
Bug Fixes
Enterprise / Starter | Fix an issue where setting up Persistent Volumes on microk8s could error and be silently ignored. |
Enterprise / Starter | microk8s stop script is now using the native shell script again. |
Enterprise / Starter | Fix issue where install hung if no usable Python was already installed. |
Enterprise | In Airgapped deployment, mark the microk8s images as pinned to prevent their garbage collection. |
Enterprise / Starter | Fix Snap failing to update microk8s. |
24.04.21-gui
Upgrade Notes
Enterprise / Starter | Upgrade ElementWeb to v1.11.81. |
Bug Fixes
Enterprise / Starter | Fix potential permissions issues during microk8s upgrades. |
Enterprise | Correctly import airgapped registry settings when upgrading from before 24.04. |
Enterprise / Starter | Improve reliability of some microk8s interactions. |
24.04.20-gui
Release Summary
The required Python versions are now 3.9, 3.10, 3.11. These are available on all supported OS distributions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.9 packages are available in their package mirrors. Alternatively, Python 3.9, 3.10, or 3.11 can be preinstalled on the server in all situations.
New Features
Enterprise / Starter | Check for supported Python versions when starting a deployment run. Recreate the virtual environment if it is using the wrong Python version. |
Enterprise / Starter | Speed improvements in the operator/updater reconciliation process. |
Enterprise / Starter | The installer now ensures that the microk8s version on the host is supported before starting the upgrade process. |
Enterprise / Starter | Allow configuration of the operator and updater with debug logs. |
Upgrade Notes
Enterprise | Global upgrade of the monitoring stack. Victoria Metrics is now on version 1.101. |
Enterprise / Starter | Services got renamed, -headless suffixes are all removed. If you are using Network Policies, those will need to be upgraded to the new names. |
Enterprise / Starter | Upgrade microk8s to 1.30. The standalone installer now upgrades microk8s automatically. Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will have to be reconfigured. This upgrade will induce a small downtime of a couple of minutes. |
Enterprise / Starter | Upgrade to cert-manager 1.12.13. |
Security Issues
Enterprise / Starter | Upgrade to Ansible 8 for security fixes. |
Bug Fixes
Enterprise | Fix empty dashboards (NGinx, Kubernetes Workloads, etc) in Grafana. |
Enterprise / Starter | Remove unneeded reconciliations due to bad orphan detection. |
Enterprise / Starter | Fix updater metrics scraping. |
Enterprise / Starter | Fix microk8s stop command not stopping running containers. |
Enterprise / Starter | Improve reliability of some microk8s interactions. |
Enterprise | Fix missing VMAlert component, which is required to gather record metrics. |
Enterprise / Starter | Validate that the node IP is excluded from a HTTP Proxy if one is configured. |
Enterprise / Starter | Improve reliability of changing the Postgres password in the cluster if the password seed changes. |
Enterprise / Starter | Improve reliability of setting up CoreDNS. |
Enterprise | Construct storage for Matrix Content Scanner if deploying on ESS managed microk8s. |
24.04.19-gui
New Features
Enterprise | Backport authenticated media APIs (MSC3916) to Synapse LTS. |
Enterprise / Starter | Scrape Synapse HAProxy metrics. |
Enterprise | Scrape Adminbot and Auditbot HAProxy metrics. |
Enterprise | Set default volume sizes for Matrix Content Scanner volumes. |
Enterprise | Set default volume sizes for Adminbot, Auditbot & Sydent volumes. |
Bug Fixes
Enterprise | Ensure operator and updater metrics are correctly scraped. |
Enterprise | Ensure Telemetry room permissions are consistent. |
Enterprise | Ensure component settings for storageClassName override the global setting. |
24.04.18-gui
Upgrade Notes
Enterprise | Upgrade Auditbot to 6.1.2 to improve overall request handling efficiency, especially at high-loads. |
24.04.17-gui
Bug Fixes
Enterprise / Starter | Fix pulling operator & updater images from behind a proxy. |
24.04.16-gui
Upgrade Notes
Enterprise / Starter | Upgrade ElementWeb to v1.11.75. |
Enterprise | Upgrade Hydrogen to v0.4.1-fix |
Bug Fixes
Enterprise / Starter | Enable MSC 3967 on Synapse to avoid some device verification issues. |
Enterprise | Setup the onprem-admin user as a MAS admin |
24.04.15-gui
Bug Fixes
Enterprise / Starter | Fix proxy variables configuration check preventing the installer from going through. |
Enterprise / Starter | Fix an issue preventing setup when a proxy is configured on the host. On proxy configuration errors, the installer will now continue the setup process after displaying the verification error message. |
24.04.14-gui
Upgrade Notes
Enterprise / Starter | Upgrade ElementWeb to v1.11.73. |
Enterprise | Upgrade SecureBorderGateway to v1.2.0 |
24.04.13-gui
New Features
Enterprise | Adminbot/Auditbot + MAS compatibility |
Upgrade Notes
Enterprise | Update Adminbot & Auditbot to Pipe 6.1.1 |
Enterprise / Starter | Matrix Content Scanner upgrade to 1.0.8 |
Bug Fixes
Enterprise | Fix display of the status of the reconciliation. |
Enterprise | Fix Coturn page causing a memory leak. |
Enterprise / Starter | Increase Matrix Content Scanner ClamAV startup reliability |
Enterprise / Starter | Reduce false positives from Matrix Content Scanner |
Enterprise / Starter | Fix microk8s services subnet parsing. |
24.04.12-gui
New Features
Enterprise / Starter | Speed up initial Synapse deploy |
Enterprise | Add the possibility to configure user deprovisioning and rooms cleanup in GroupSync |
Bug Fixes
Enterprise / Starter | Make sure nf_conntrack module is loaded in the kernel when deploying in standalone mode. |
24.04.11-gui
New Features
Enterprise | Add the possibility to configure a matrix stats endpoint |
Enterprise | Setup the onprem-admin user as a MAS admin |
Enterprise | Allow configuration of empty (no) disallowed IP ranges in Hookshot |
Enterprise | Validate Synapse Telemetry is consistently set |
Enterprise / Starter | Synapse improve worker configuration |
Enterprise / Starter | Allow blocking of non-scanned media |
Upgrade Notes
Enterprise / Starter | On RHEL and derived platforms, it now requires python 3.11 installed. |
Bug Fixes
Enterprise / Starter | On RHEL and derived platforms, the installer should not rely on platform-python for other tasks than Firewalld and SELinux tasks for microk8s setup. |
Enterprise / Starter | Fix some CVEs in the operator/updater/conversion webhook |
Enterprise / Starter | Fix Matrix Content Scanner not working as expected |
Enterprise | Configure max upload size in Secure Border Gateway request body size limit |
24.04.10-gui
Security Issues
Enterprise | Better image signatures, enterprise is now published to sigstore |
Bug Fixes
Enterprise / Starter | Refactor synapse config files to own the priority of each setting managed by ESS |
Enterprise | Sygnal upgrade to 0.15.0 for further Firebase API fixes |
Enterprise | Adminbot and Auditbot are currently incompatible with MAS |
Enterprise | Synapse - override botocore CA bundle to allow pushing against non-AWS S3 providers |
Enterprise | Add support for Element Call configuration in Element Well Known file |
Enterprise | Matrix Authentication Service - fix UI configuration of certificates for ingresses |
Enterprise | Minor speed up to initial setup of Synapse |
Enterprise | Prevent users from editing auditbot and adminbot passphrase in the UI. |
Enterprise | Enforce pattern checks against inputs under options. |
24.04.09-gui
Security Issues
Enterprise | Previous update might have enabled unexpectedly outbound webhooks in Hookshot. If you don't need this feature, make sure that it is disabled in Hookshot integration, under Generic Webhooks settings. |
Bug Fixes
Enterprise | Reduce secrets leaks from operator & updater logs. If you need, for debugging purposes, to enable secrets logging, you must edit the operator & updater deployments and set the environment variable DEBUG_MANIFESTS=1 |
24.04.08-gui
New Features
Enterprise | Add support for Outbound webhooks in Hookshot. |
Enterprise | Synapse OIDC support attribute requirements |
Upgrade Notes
Enterprise | Upgrade Adminbot & Auditbot to using matrix-pipe 5.1.0, based on Rust Crypto SDK. |
Enterprise | Upgrade Sygnal to 0.14.3 to support latest Firebase API. |
Bug Fixes
Enterprise | Fixes an issue where auditbot UI would fail to open because tokens were unable to refresh. |
Enterprise | Fix a critical issue which would prevent users from accessing Adminbot and Auditbot UI. |
Enterprise | Revert change of 24.04.07 which prevented adminbot and auditbot from doing an initial sync. |
Enterprise | Create new devices for adminbot and auditbot to work with the new rust sdk cryptographic libraries. |
24.04.07-gui
New Features
Enterprise | Add support for Outbound webhooks in Hookshot. |
Enterprise | Synapse OIDC support attribute requirements |
Upgrade Notes
Enterprise | Upgrade Adminbot & Auditbot to using matrix-pipe 5.1.0, based on Rust Crypto SDK. |
Enterprise | Upgrade Sygnal to 0.14.3 to support latest Firebase API. |
Bug Fixes
Enterprise / Starter | Fix an issue preventing setup when a proxy is configured on the host. |
Enterprise | Attempt to detect OpenShift and configure operator & updater installation values appropriately |
24.04.06-gui
New Features
Enterprise | Allow configuration of Synapse database connection pool sizes |
Enterprise | Expose Operator & Updater metrics |
Enterprise | Add a ServiceMonitor to scrape metrics of microk8s ingress. |
Upgrade Notes
Enterprise | Upgrade Adminbot for more reliable decryption support |
Enterprise / Starter | Upgrade to cert-manager 1.12.11 |
Bug Fixes
Enterprise | Don't include cert-manager in the airgapped tarball. ESS doesn't install or manage cert-manager in airgapped deploys |
Enterprise / Starter | Allow well-known delegation to omit configuration of the ingress entirely without triggering unknown variable errors |
Enterprise / Starter | Allow configuration of Matrix Content Scanner without a storage class name |
Enterprise / Starter | Mark Postgres configuration as required for all components that use a Postgres database |
Enterprise | Mark the source for GroupSync as required |
Enterprise | Remove workloads and dependent CRs from statuses when they're no longer deployed |
Enterprise | Fix provisioning of users that are not rate-limited |
Enterprise | Better identification for the Telegram and WhatsApp bridges in their respective apps |
Enterprise | Avoid leaking Postgres connections when there are issues provisioning Synapse users |
Enterprise | SIPBridge - Disable Virtual rooms |
Enterprise / Starter | Fix an issue where the cert manager issuer would try to be created but the cert-manager webhook would not be ready. |
Enterprise | Fix monitoring of kube etcd and kube scheduler on microk8s. |
24.04.05-gui
New Features
Enterprise / Starter | Major Change: The standalone installer now upgrades microk8s gracefully automatically. The microk8s upgrade procedure does not anymore involve a uninstall/reinstall of microk8s. It now will automatically upgrade microk8s to the expected version, and the flag --upgrade-cluster has been removed.Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will have to be reconfigured. During the upgrade, microk8s will restart, and addons will be disabled to force an upgrade. It can induce a small downtime of a couple of minutes. |
Enterprise | Status watchers are now golang containers. Resources used by the operator and updater are now reduced. |
Upgrade Notes
Enterprise | Upgrade Telegram bridge to 0.15.1-mod-1 |
Enterprise | Upgrade WhatsApp bridge to 0.10.7-mod-1 |
Bug Fixes
Enterprise / Starter | Fix haproxy failing on ipv4-only nodes. |
Enterprise | Fix inconsistent behaviour when switching between S3/Persistent volume option under media tab. |
Enterprise / Starter | Fix watchers to avoid triggering unneeded reconciliation loops |
Enterprise | GroupSync - Fix issue when LDAP identities contain commas in their names. |
Enterprise / Starter | Fix cert-manager upgrade failing to remove old resources. |
Enterprise | Fix media screen on standalone setup. |
Enterprise / Starter | Remove --upgrade-cluster parameter as microk8s is now upgraded gracefully. |
Enterprise / Starter | Fix operator and updater having permissions issues under Openshift |
Enterprise / Starter | Fix Jitsi JVB fails to get ready when STUN servers list is empty and Coturn is not deployed. |
Enterprise / Starter | The installer does not flake anymore between bootstrap and installer view when the kubernetes cluster is not reachable intermittently. |
Enterprise | Configuring monitoring stack persistent volumes properly in microk8s requires to recreate their statefulsets. |
Enterprise | Fix an ansible error when installing the telemetry script on the local host when user GID != UID. |
Enterprise | Fix missing storage class on some Monitoring PVCs. |
24.04.04-gui
Bug Fixes
Enterprise / Starter | Improve robustness of adding custom well-known delegation configuration |
Enterprise / Starter | Fix missing media tab in the Admin Console when using microk8s. |
Enterprise | Fix Enable DM Admin not being respected for Adminbot |
Enterprise / Starter | Fix failure regenerating installer authentication links. |
24.04.03-gui
New Features
Enterprise | Improve GroupSync performance with large member lists |
Enterprise | Add Azure Blob Storage support to Auditbot |
Enterprise | Config GroupSync memory usage based on resource limits/requests |
Upgrade Notes
Enterprise / Starter | Upgrade Element Web to 1.11.66 |
Bug Fixes
Enterprise | Improve reliability of Synapse user provisioning |
Enterprise | Improve Jitsi timezone validation |
Enterprise / Starter | Improve Postgres shutdown behaviour when using the ESS Postgreses in cluster |
24.04.02-gui
Upgrade Notes
Enterprise | Upgrade airgapped microk8s to 1.27.13 |
Bug Fixes
Enterprise | Fix issue upgrading from 23.10 LTS in an Airgapped environment where images weren't uploaded to the registry anymore |
Enterprise | Synapse HTTP proxy settings can now be edited in the installer. |
Enterprise / Starter | Media volume name and size can now be configured for standalone cluster deployments. |
24.04.01-gui
Release Summary
23.10.29 LTS to 24.04.01 LTS highlights
This release has focused on making deployments on Kubernetes more reliable. A lot of bugs were fixed, and helm charts have been enhanced to allow to deploy webhooks and CRDs together without the operator and updater.
LTS New Features
Enterprise / Starter | The admin app now allows viewing of uploaded media |
Enterprise | Add WhatsApp Bridge support |
Enterprise | Check the health of the deployment or a component using kubectl describe against any Element CRs, in the status. Our documentation describes how to configure ArgoCD to get these informations into your Application health. |
Enterprise | Add the possiblity to configure S3 for Synapse media storage. |
Enterprise | Improve support for non-OIDC compliant upstream identity providers with Matrix Authentication Service, |
Enterprise / Starter | Allow configuration of seLinuxOptions on all workloads. |
Enterprise | Enable simple configuration of whether Element Web generates sharing links with its own URL or matrix.to |
Enterprise | When using Airgapped deployment, it is now possible to login to the target upload registry in the installer UI. |
Enterprise / Starter | A couple of speedups have been implemented both in the operator and the installer. |
Enterprise / Starter | Change deploy order of components to have the core components deployed first by the updater. |
Enterprise / Starter | The operator and the updater are now built based on distroless container, to reduce the image size and contents. |
Enterprise | Auditbot UI does not need any ingress anymore. |
Enterprise / Starter | The installer now contains crictl to allow for local ctr daemon maintenance on microk8s. |
Enterprise | Reduce required resources for Standalone to 2 vCPU and 3Gb of memory. |
Enterprise / Starter | Reduce postgres in cluster requests to 100Mi. |
Enterprise | Add participant limit field in ElementCall configuration. |
Enterprise / Starter | Add support for tolerations and nodeSelectors on workload. |
Enterprise | Coturn is now managed by the UI view, by the updater, alongside ElementCall and Jitsi. It is now possible to deploy Coturn on a Kubernetes cluster. |
Enterprise / Starter | We now configure automatically a CPU Limit of each Operator & Updater to be 25% of the machine vCPUs on standalone. The node still needs at least 2 vCPUs to work properly. On Kubernetes deployment, there's no CPU limit. The number of workers will be adapted relatively to the memory available to the operator/updater. |
LTS Upgrade Notes
This new LTS can be upgraded from 23.10 if you want to get the new latest features of ESS.
LTS Version Updates
Enterprise / Starter | Update operator-sdk to v1.34.1 |
Enterprise | Update Hookshot to 5.2.1 |
Enterprise / Starter | Update ElementWeb to v1.11.64 |
Enterprise / Starter | Update SlidingSync to v0.99.15 |
Enterprise | Update Synapse to v1.99.0 with CVE-2024-31208 fix |
Enterprise | Update Element Call to 0.5.16 and LiveKit to 1.5.1 |
Enterprise | Update Sydent to 2.6.1 |
LTS Synapse security release
This release contains a fix for GHSA-3h7q-rfh9-xm4v / CVE-2024-31208, a high severity Synapse security issue. Upgrading is advised at the soonest possible moment.
Important notes regarding rollback of this release
This release contains an important Synapse security fix with a backwards incompatible change. Please note that simply reverting this ESS release is not possible.
Please ensure to have a working backups before upgrading as downgrading is not a possibility from this release.
New Features
Enterprise | Check the health of the deployment or a component using kubectl describe against any Element CRs, in the status. Our documentation describes how to configure ArgoCD to get these information into your Application health. |
Enterprise | Add the possibility to configure S3 for Synapse media storage. |
Enterprise | Add options under Delegated Auth to configure users profiles editing permissions. |
Enterprise | Improve support for non-OIDC compliant upstream identity providers with Matrix Authentication Service |
Enterprise / Starter | Allow configuration of seLinuxOptions on all workloads |
Enterprise | Enable simple configuration of whether Element Web generates sharing links with its own URL or matrix.to |
Enterprise | Support GCM/FCM API v1 in Sygnal |
Enterprise / Starter | Configure ansible poll interval to 0.01 to reduce CPU load |
Enterprise / Starter | A couple of speedups have been implemented both in the operator and the installer. |
Enterprise / Starter | We now configure automatically a CPU Limit of each Operator & Updater to be 25% of the machine vCPUs on standalone. The node still needs at least 2 vCPUs to work properly. On Kubernetes deployment, there's no CPU limit. The number of workers will be adapted relatively to the memory available to the operator/updater. |
Upgrade Notes
Enterprise / Starter | Update operator-sdk to v1.34.1 |
Enterprise | Update Hookshot to 5.2.1 |
Enterprise / Starter | Update SlidingSync to v0.99.15 |
Enterprise | Update Synapse to v1.99.0 with CVE-2024-31208 fix |
Enterprise / Starter | Upgrade Element Web to v1.11.64. |
Enterprise | Upgrade Matrix Authentication Service to v0.9.0. |
Enterprise | Update Secure Border Gateway to v1.1.1. |
Enterprise | Upgrade Group Sync to v0.13.6. |
Enterprise | Element Call 0.5.16 and LiveKit 1.5.1 |
Enterprise | Sydent 2.6.1 |
Enterprise | Make Jitsi and Element Call STUN configuration consistent with each other to ease the upgrade from 23.10. |
Enterprise | Upgrade Sygnal to v0.14.1. |
Security Issues
Enterprise | Upgrade IRC Bridge to 2.0.0 to fix CVE-2024-32000. |
Bug Fixes
Enterprise / Starter | Correctly install apt package python3-venv on recent ubuntu version. |
Enterprise | Fixes to how Admin/Auditbot configs are maintained in the installer. |
Enterprise / Starter | Improve installer one-time login codes security. |
Enterprise / Starter | Mitigate installer log injections via HTTP headers. |
Enterprise | Fix admin console discovery of OIDC to use MSC2956. |
Enterprise | Update Auditbot S3 object name to one that will not clash with other files. |
Enterprise | Fix issues passing in Coturn external-ip and enabling host mode. |
Enterprise / Starter | Fix an issue where Auditbot S3 storage would prune files too early. |
Enterprise / Starter | Fix an issue with Jitsi where it would not be possible to configure the Sync Power Level in the Restrict Widgets to Synapse configuration. |
Enterprise | AdminBot and Matrix Authentication Service can now be deployed together |
Enterprise | Upgrade Synapse Admin to better support homeservers using SRV delegation |
Enterprise | Fix support for APNS notifications in Sygnal going via a HTTP Forward Proxy |
Enterprise | Fix configuration of multiple TURN servers in Synapse when manually configuring |
Enterprise | Fix Sydent Terms & Conditions having a version that's just a number |
Enterprise / Starter | Fix ServiceMonitors being left behind when components are removed |
Enterprise | Fix SIP Bridge Services clashing |
Enterprise | Fix a bug which could make airgapped impossible to deploy due to microk8s snap refresh being in error state. |
Enterprise | Fix Synapse bootstrap phase getting stuck due to incompatible registration options. |
Enterprise / Starter | Stop displaying NGINX version on error pages. |
Enterprise | Clarify and improve validation of TURN server configuration section. |
Enterprise | Ignore Adminbot/Auditbot users in IRC admin rooms. |
Enterprise | Fix an issue where configuring Coturn would lead to infinite reconciliation. |
Other
Enterprise | Clean up unused Matrix Authentication Service spa HTTP resource. |
Enterprise | Auditbot no longer requires the configuration of a dedicated UI ingress. This is handled by Synapse Admin UI now |
Enterprise | Clarify description of Synapse default room encryption section. |