Skip to main content

ESS LTS 24.04 Change Logs and Upgrade Notes

LTS 24.04 Changelogs and important Update Notes, always check here before upgrading!

                 

Upgrade Notes for the 24.04 LTS

If you are planning on upgrading to the LTS we always recommend upgrading to the latest LTS patch version, however you should be aware of all significant upgrade notes from each prior patch version. They have been collated for convenience below, you can find the full changelogs of each release there after.

24.04.20-gui

The required Python versions are now 3.9, 3.10, 3.11. These are available on all supported OS distributions. The installer will attempt to install the required packages in some scenarios.
Airgapped customers should ensure that Python 3.9 packages are available in their package mirrors.
Alternatively, Python 3.9, 3.10, or 3.11 can be preinstalled on the server in all situations.

24.04.05-gui

Major Change: The standalone installer now upgrades microk8s gracefully automatically. The microk8s upgrade procedure does not anymore involve an uninstall/reinstall of microk8s. It now will automatically upgrade microk8s to the expected version, and the flag --upgrade-cluster has been removed.

Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will have to be reconfigured. During the upgrade, microk8s will restart, and add-ons will be disabled to force an upgrade. It can induce a small downtime of a couple of minutes.

24.04.01-gui

This release contains an important Synapse security fix with a backwards incompatible change. Please note that simply reverting this ESS release is not possible.

Please ensure to have a working backups before upgrading as downgrading is not a possibility from this release.

24.04.21-gui

Upgrade Notes

Enterprise / Starter

Upgrade ElementWeb to v1.11.81.

Bug Fixes

Enterprise / Starter

Fix potential permissions issues during microk8s upgrades.

Enterprise

Correctly import airgapped registry settings when upgrading from before 24.04.

Enterprise / Starter

Improve reliability of some microk8s interactions.

24.04.20-gui

Release Summary

The required Python versions are now 3.9, 3.10, 3.11. These are available on all supported OS distributions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.9 packages are available in their package mirrors. Alternatively, Python 3.9, 3.10, or 3.11 can be preinstalled on the server in all situations.

New Features

Enterprise / Starter

Check for supported Python versions when starting a deployment run. Recreate the virtual environment if it is using the wrong Python version.

Enterprise / Starter

Speed improvements in the operator/updater reconciliation process.

Enterprise / Starter

The installer now ensures that the microk8s version on the host is supported before starting the upgrade process.

Enterprise / Starter

Allow configuration of the operator and updater with debug logs.

Upgrade Notes

Enterprise

Global upgrade of the monitoring stack. Victoria Metrics is now on version 1.101.

Enterprise / Starter

Services got renamed, -headless suffixes are all removed. If you are using Network Policies, those will need to be upgraded to the new names.

Enterprise / Starter

Upgrade microk8s to 1.30. The standalone installer now upgrades microk8s automatically. Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will have to be reconfigured. This upgrade will induce a small downtime of a couple of minutes.

Enterprise / Starter

Upgrade to cert-manager 1.12.13.

Security Issues

Enterprise / Starter

Upgrade to Ansible 8 for security fixes.

Bug Fixes

Enterprise

Fix empty dashboards (NGinx, Kubernetes Workloads, etc) in Grafana.

Enterprise / Starter

Remove unneeded reconciliations due to bad orphan detection.

Enterprise / Starter

Fix updater metrics scraping.

Enterprise / Starter

Fix microk8s stop command not stopping running containers.

Enterprise / Starter

Improve reliability of some microk8s interactions.

Enterprise

Fix missing VMAlert component, which is required to gather record metrics.

Enterprise / Starter

Validate that the node IP is excluded from a HTTP Proxy if one is configured.

Enterprise / Starter

Improve reliability of changing the Postgres password in the cluster if the password seed changes.

Enterprise / Starter

Improve reliability of setting up CoreDNS.

Enterprise

Construct storage for Matrix Content Scanner if deploying on ESS managed microk8s.

24.04.19-gui

New Features

Enterprise

Backport authenticated media APIs (MSC3916) to Synapse LTS.

Enterprise / Starter

Scrape Synapse HAProxy metrics.

Enterprise

Scrape Adminbot and Auditbot HAProxy metrics.

Enterprise

Set default volume sizes for Matrix Content Scanner volumes.

Enterprise

Set default volume sizes for Adminbot, Auditbot & Sydent volumes.

Bug Fixes

Enterprise

Ensure operator and updater metrics are correctly scraped.

Enterprise

Ensure Telemetry room permissions are consistent.

Enterprise

Ensure component settings for storageClassName override the global setting.

24.04.18-gui

Upgrade Notes

Enterprise

Upgrade Auditbot to 6.1.2 to improve overall request handling efficiency, especially at high-loads.

24.04.17-gui

Bug Fixes

Enterprise / Starter

Fix pulling operator & updater images from behind a proxy.

24.04.16-gui

Upgrade Notes

Enterprise / Starter

Upgrade ElementWeb to v1.11.75.

Enterprise

Upgrade Hydrogen to v0.4.1-fix

Bug Fixes

Enterprise / Starter

Enable MSC 3967 on Synapse to avoid some device verification issues.

Enterprise

Setup the onprem-admin user as a MAS admin

24.04.15-gui

Bug Fixes

Enterprise / Starter

Fix proxy variables configuration check preventing the installer from going through.

Enterprise / Starter

Fix an issue preventing setup when a proxy is configured on the host. On proxy configuration errors, the installer will now continue the setup process after displaying the verification error message.

24.04.14-gui

Upgrade Notes

Enterprise / Starter

Upgrade ElementWeb to v1.11.73.

Enterprise

Upgrade SecureBorderGateway to v1.2.0

24.04.13-gui

New Features

Enterprise

Adminbot/Auditbot + MAS compatibility

Upgrade Notes

Enterprise

Update Adminbot & Auditbot to Pipe 6.1.1

Enterprise / Starter

Matrix Content Scanner upgrade to 1.0.8

Bug Fixes

Enterprise

Fix display of the status of the reconciliation.

Enterprise

Fix Coturn page causing a memory leak.

Enterprise / Starter

Increase Matrix Content Scanner ClamAV startup reliability

Enterprise / Starter

Reduce false positives from Matrix Content Scanner

Enterprise / Starter

Fix microk8s services subnet parsing.

24.04.12-gui

New Features

Enterprise / Starter

Speed up initial Synapse deploy

Enterprise

Add the possibility to configure user deprovisioning and rooms cleanup in GroupSync

Bug Fixes

Enterprise / Starter

Make sure nf_conntrack module is loaded in the kernel when deploying in standalone mode.

24.04.11-gui

New Features

Enterprise

Add the possibility to configure a matrix stats endpoint

Enterprise

Setup the onprem-admin user as a MAS admin

Enterprise

Allow configuration of empty (no) disallowed IP ranges in Hookshot

Enterprise

Validate Synapse Telemetry is consistently set

Enterprise / Starter

Synapse improve worker configuration

Enterprise / Starter

Allow blocking of non-scanned media

Upgrade Notes

Enterprise / Starter

On RHEL and derived platforms, it now requires python 3.11 installed.

Bug Fixes

Enterprise / Starter

On RHEL and derived platforms, the installer should not rely on platform-python for other tasks than Firewalld and SELinux tasks for microk8s setup.

Enterprise / Starter

Fix some CVEs in the operator/updater/conversion webhook

Enterprise / Starter

Fix Matrix Content Scanner not working as expected

Enterprise

Configure max upload size in Secure Border Gateway request body size limit

24.04.10-gui

Security Issues

Enterprise

Better image signatures, enterprise is now published to sigstore

Bug Fixes

Enterprise / Starter

Refactor synapse config files to own the priority of each setting managed by ESS

Enterprise

Sygnal upgrade to 0.15.0 for further Firebase API fixes

Enterprise

Adminbot and Auditbot are currently incompatible with MAS

Enterprise

Synapse - override botocore CA bundle to allow pushing against non-AWS S3 providers

Enterprise

Add support for Element Call configuration in Element Well Known file

Enterprise

Matrix Authentication Service - fix UI configuration of certificates for ingresses

Enterprise

Minor speed up to initial setup of Synapse

Enterprise

Prevent users from editing auditbot and adminbot passphrase in the UI.

Enterprise

Enforce pattern checks against inputs under options.

24.04.09-gui

Security Issues

Enterprise

Previous update might have enabled unexpectedly outbound webhooks in Hookshot. If you don't need this feature, make sure that it is disabled in Hookshot integration, under Generic Webhooks settings.

Bug Fixes

Enterprise

Reduce secrets leaks from operator & updater logs. If you need, for debugging purposes, to enable secrets logging, you must edit the operator & updater deployments and set the environment variable DEBUG_MANIFESTS=1

24.04.08-gui

New Features

Enterprise

Add support for Outbound webhooks in Hookshot.

Enterprise

Synapse OIDC support attribute requirements

Upgrade Notes

Enterprise

Upgrade Adminbot & Auditbot to using matrix-pipe 5.1.0, based on Rust Crypto SDK.

Enterprise

Upgrade Sygnal to 0.14.3 to support latest Firebase API.

Bug Fixes

Enterprise

Fixes an issue where auditbot UI would fail to open because tokens were unable to refresh.

Enterprise

Fix a critical issue which would prevent users from accessing Adminbot and Auditbot UI.

Enterprise

Revert change of 24.04.07 which prevented adminbot and auditbot from doing an initial sync.

Enterprise

Create new devices for adminbot and auditbot to work with the new rust sdk cryptographic libraries.

24.04.07-gui

New Features

Enterprise

Add support for Outbound webhooks in Hookshot.

Enterprise

Synapse OIDC support attribute requirements

Upgrade Notes

Enterprise

Upgrade Adminbot & Auditbot to using matrix-pipe 5.1.0, based on Rust Crypto SDK.

Enterprise

Upgrade Sygnal to 0.14.3 to support latest Firebase API.

Bug Fixes

Enterprise / Starter

Fix an issue preventing setup when a proxy is configured on the host.

Enterprise

Attempt to detect OpenShift and configure operator & updater installation values appropriately

24.04.06-gui

New Features

Enterprise

Allow configuration of Synapse database connection pool sizes

Enterprise

Expose Operator & Updater metrics

Enterprise

Add a ServiceMonitor to scrape metrics of microk8s ingress.

Upgrade Notes

Enterprise

Upgrade Adminbot for more reliable decryption support

Enterprise / Starter

Upgrade to cert-manager 1.12.11

Bug Fixes

Enterprise

Don't include cert-manager in the airgapped tarball. ESS doesn't install or manage cert-manager in airgapped deploys

Enterprise / Starter

Allow well-known delegation to omit configuration of the ingress entirely without triggering unknown variable errors

Enterprise / Starter

Allow configuration of Matrix Content Scanner without a storage class name

Enterprise / Starter

Mark Postgres configuration as required for all components that use a Postgres database

Enterprise

Mark the source for GroupSync as required

Enterprise

Remove workloads and dependent CRs from statuses when they're no longer deployed

Enterprise

Fix provisioning of users that are not rate-limited

Enterprise

Better identification for the Telegram and WhatsApp bridges in their respective apps

Enterprise

Avoid leaking Postgres connections when there are issues provisioning Synapse users

Enterprise

SIPBridge - Disable Virtual rooms

Enterprise / Starter

Fix an issue where the cert manager issuer would try to be created but the cert-manager webhook would not be ready.

Enterprise

Fix monitoring of kube etcd and kube scheduler on microk8s.

24.04.05-gui

New Features

Enterprise / Starter

Major Change: The standalone installer now upgrades microk8s gracefully automatically. The microk8s upgrade procedure does not anymore involve a uninstall/reinstall of microk8s. It now will automatically upgrade microk8s to the expected version, and the flag --upgrade-cluster has been removed.

Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will have to be reconfigured. During the upgrade, microk8s will restart, and addons will be disabled to force an upgrade. It can induce a small downtime of a couple of minutes.

Enterprise

Status watchers are now golang containers. Resources used by the operator and updater are now reduced.

Upgrade Notes

Enterprise

Upgrade Telegram bridge to 0.15.1-mod-1

Enterprise

Upgrade WhatsApp bridge to 0.10.7-mod-1

Bug Fixes

Enterprise / Starter

Fix haproxy failing on ipv4-only nodes.

Enterprise

Fix inconsistent behaviour when switching between S3/Persistent volume option under media tab.

Enterprise / Starter

Fix watchers to avoid triggering unneeded reconciliation loops

Enterprise

GroupSync - Fix issue when LDAP identities contain commas in their names.

Enterprise / Starter

Fix cert-manager upgrade failing to remove old resources.

Enterprise

Fix media screen on standalone setup.

Enterprise / Starter

Remove --upgrade-cluster parameter as microk8s is now upgraded gracefully.

Enterprise / Starter

Fix operator and updater having permissions issues under Openshift

Enterprise / Starter

Fix Jitsi JVB fails to get ready when STUN servers list is empty and Coturn is not deployed.

Enterprise / Starter

The installer does not flake anymore between bootstrap and installer view when the kubernetes cluster is not reachable intermittently.

Enterprise

Configuring monitoring stack persistent volumes properly in microk8s requires to recreate their statefulsets.

Enterprise

Fix an ansible error when installing the telemetry script on the local host when user GID != UID.

Enterprise

Fix missing storage class on some Monitoring PVCs.

24.04.04-gui

Bug Fixes

Enterprise / Starter

Improve robustness of adding custom well-known delegation configuration

Enterprise / Starter

Fix missing media tab in the Admin Console when using microk8s.

Enterprise

Fix Enable DM Admin not being respected for Adminbot

Enterprise / Starter

Fix failure regenerating installer authentication links.

24.04.03-gui

New Features

Enterprise

Improve GroupSync performance with large member lists

Enterprise

Add Azure Blob Storage support to Auditbot

Enterprise

Config GroupSync memory usage based on resource limits/requests

Upgrade Notes

Enterprise / Starter

Upgrade Element Web to 1.11.66

Bug Fixes

Enterprise

Improve reliability of Synapse user provisioning

Enterprise

Improve Jitsi timezone validation

Enterprise / Starter

Improve Postgres shutdown behaviour when using the ESS Postgreses in cluster

24.04.02-gui

Upgrade Notes

Enterprise

Upgrade airgapped microk8s to 1.27.13

Bug Fixes

Enterprise

Fix issue upgrading from 23.10 LTS in an Airgapped environment where images weren't uploaded to the registry anymore

Enterprise

Synapse HTTP proxy settings can now be edited in the installer.

Enterprise / Starter

Media volume name and size can now be configured for standalone cluster deployments.

24.04.01-gui

Release Summary

23.10.29 LTS to 24.04.01 LTS highlights

This release has focused on making deployments on Kubernetes more reliable. A lot of bugs were fixed, and helm charts have been enhanced to allow to deploy webhooks and CRDs together without the operator and updater.

LTS New Features

Enterprise / Starter

The admin app now allows viewing of uploaded media

Enterprise

Add WhatsApp Bridge support

Enterprise

Check the health of the deployment or a component using kubectl describe against any Element CRs, in the status. Our documentation describes how to configure ArgoCD to get these informations into your Application health.

Enterprise

Add the possiblity to configure S3 for Synapse media storage.

Enterprise

Improve support for non-OIDC compliant upstream identity providers with Matrix Authentication Service,

Enterprise / Starter

Allow configuration of seLinuxOptions on all workloads.

Enterprise

Enable simple configuration of whether Element Web generates sharing links with its own URL or matrix.to

Enterprise

When using Airgapped deployment, it is now possible to login to the target upload registry in the installer UI.

Enterprise / Starter

A couple of speedups have been implemented both in the operator and the installer.

Enterprise / Starter

Change deploy order of components to have the core components deployed first by the updater.

Enterprise / Starter

The operator and the updater are now built based on distroless container, to reduce the image size and contents.

Enterprise

Auditbot UI does not need any ingress anymore.

Enterprise / Starter

The installer now contains crictl to allow for local ctr daemon maintenance on microk8s.

Enterprise

Reduce required resources for Standalone to 2 vCPU and 3Gb of memory.

Enterprise / Starter

Reduce postgres in cluster requests to 100Mi.

Enterprise

Add participant limit field in ElementCall configuration.

Enterprise / Starter

Add support for tolerations and nodeSelectors on workload.

Enterprise

Coturn is now managed by the UI view, by the updater, alongside ElementCall and Jitsi. It is now possible to deploy Coturn on a Kubernetes cluster.

Enterprise / Starter

We now configure automatically a CPU Limit of each Operator & Updater to be 25% of the machine vCPUs on standalone. The node still needs at least 2 vCPUs to work properly. On Kubernetes deployment, there's no CPU limit. The number of workers will be adapted relatively to the memory available to the operator/updater.

LTS Upgrade Notes

This new LTS can be upgraded from 23.10 if you want to get the new latest features of ESS.

LTS Version Updates

Enterprise / Starter

Update operator-sdk to v1.34.1

Enterprise

Update Hookshot to 5.2.1

Enterprise / Starter

Update ElementWeb to v1.11.64

Enterprise / Starter

Update SlidingSync to v0.99.15

Enterprise

Update Synapse to v1.99.0 with CVE-2024-31208 fix

Enterprise

Update Element Call to 0.5.16 and LiveKit to 1.5.1

Enterprise

Update Sydent to 2.6.1

LTS Synapse security release

This release contains a fix for GHSA-3h7q-rfh9-xm4v / CVE-2024-31208, a high severity Synapse security issue. Upgrading is advised at the soonest possible moment.

Important notes regarding rollback of this release

This release contains an important Synapse security fix with a backwards incompatible change. Please note that simply reverting this ESS release is not possible.

Please ensure to have a working backups before upgrading as downgrading is not a possibility from this release.

New Features

Enterprise

Check the health of the deployment or a component using kubectl describe against any Element CRs, in the status. Our documentation describes how to configure ArgoCD to get these information into your Application health.

Enterprise

Add the possibility to configure S3 for Synapse media storage.

Enterprise

Add options under Delegated Auth to configure users profiles editing permissions.

Enterprise

Improve support for non-OIDC compliant upstream identity providers with Matrix Authentication Service

Enterprise / Starter

Allow configuration of seLinuxOptions on all workloads

Enterprise

Enable simple configuration of whether Element Web generates sharing links with its own URL or matrix.to

Enterprise

Support GCM/FCM API v1 in Sygnal

Enterprise / Starter

Configure ansible poll interval to 0.01 to reduce CPU load

Enterprise / Starter

A couple of speedups have been implemented both in the operator and the installer.

Enterprise / Starter

We now configure automatically a CPU Limit of each Operator & Updater to be 25% of the machine vCPUs on standalone. The node still needs at least 2 vCPUs to work properly. On Kubernetes deployment, there's no CPU limit. The number of workers will be adapted relatively to the memory available to the operator/updater.

Upgrade Notes

Enterprise / Starter

Update operator-sdk to v1.34.1

Enterprise

Update Hookshot to 5.2.1

Enterprise / Starter

Update SlidingSync to v0.99.15

Enterprise

Update Synapse to v1.99.0 with CVE-2024-31208 fix

Enterprise / Starter

Upgrade Element Web to v1.11.64.

Enterprise

Upgrade Matrix Authentication Service to v0.9.0.

Enterprise

Update Secure Border Gateway to v1.1.1.

Enterprise

Upgrade Group Sync to v0.13.6.

Enterprise

Element Call 0.5.16 and LiveKit 1.5.1

Enterprise

Sydent 2.6.1

Enterprise

Make Jitsi and Element Call STUN configuration consistent with each other to ease the upgrade from 23.10.

Enterprise

Upgrade Sygnal to v0.14.1.

Security Issues

Enterprise

Upgrade IRC Bridge to 2.0.0 to fix CVE-2024-32000.

Bug Fixes

Enterprise / Starter

Correctly install apt package python3-venv on recent ubuntu version.

Enterprise

Fixes to how Admin/Auditbot configs are maintained in the installer.

Enterprise / Starter

Improve installer one-time login codes security.

Enterprise / Starter

Mitigate installer log injections via HTTP headers.

Enterprise

Fix admin console discovery of OIDC to use MSC2956.

Enterprise

Update Auditbot S3 object name to one that will not clash with other files.

Enterprise

Fix issues passing in Coturn external-ip and enabling host mode.

Enterprise / Starter

Fix an issue where Auditbot S3 storage would prune files too early.

Enterprise / Starter

Fix an issue with Jitsi where it would not be possible to configure the Sync Power Level in the Restrict Widgets to Synapse configuration.

Enterprise

AdminBot and Matrix Authentication Service can now be deployed together

Enterprise

Upgrade Synapse Admin to better support homeservers using SRV delegation

Enterprise

Fix support for APNS notifications in Sygnal going via a HTTP Forward Proxy

Enterprise

Fix configuration of multiple TURN servers in Synapse when manually configuring

Enterprise

Fix Sydent Terms & Conditions having a version that's just a number

Enterprise / Starter

Fix ServiceMonitors being left behind when components are removed

Enterprise

Fix SIP Bridge Services clashing

Enterprise

Fix a bug which could make airgapped impossible to deploy due to microk8s snap refresh being in error state.

Enterprise

Fix Synapse bootstrap phase getting stuck due to incompatible registration options.

Enterprise / Starter

Stop displaying NGINX version on error pages.

Enterprise

Clarify and improve validation of TURN server configuration section.

Enterprise

Ignore Adminbot/Auditbot users in IRC admin rooms.

Enterprise

Fix an issue where configuring Coturn would lead to infinite reconciliation.

Other

Enterprise

Clean up unused Matrix Authentication Service spa HTTP resource.

Enterprise

Auditbot no longer requires the configuration of a dedicated UI ingress. This is handled by Synapse Admin UI now

Enterprise

Clarify description of Synapse default room encryption section.