ESS LTS 24.10 Change Logs and Upgrade Notes
LTS 24.10 Changelogs and important Update Notes, always check here before upgrading!
Upgrade Notes for the 24.10 LTS
If you plan on upgrading to this LTS we always recommend upgrading to the latest patch version of your current LTS and then updating to the latest version of this LTS.
If you plan on updating, we recommend installing the latest patch version.
Whether upgrading or updating, you should be aware of all significant upgrade notes from each prior patch version. Any highlighted patch notes for this specific LTS have been collated for convenience below, you can find the full changelogs of each release thereafter.
24.10.01-gui | The required Python versions are now 3.10, 3.11, 3.12. As a result, Ubuntu 24.04 is now supported but Ubuntu 20.04 support is dropped. Please consult the Ubuntu documentation for upgrading between Ubuntu LTS versions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.12 packages are available in their package mirrors. Alternatively, Python 3.10, 3.11, or 3.12 can be preinstalled on the server in all situations. |
24.10.01-gui
Release Summary
The required Python versions are now 3.10, 3.11, 3.12. As a result, Ubuntu 24.04 is now supported but Ubuntu 20.04 support is dropped. Please consult the Ubuntu documentation for upgrading between Ubuntu LTS versions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.12 packages are available in their package mirrors. Alternatively, Python 3.10, 3.11, or 3.12 can be preinstalled on the server in all situations.
New Features
Enterprise | XMPP Bridge and IRC Bridge both support Authenticated Medias. Their signing key is generated automatically by the installer UI. |
Enterprise / Starter | Authenticated Media is now enforced by default. All components but Matrix Content Scanner are compatible with it. If you need to disable it, please add enable_authenticated_media: false to Synapse -> Additional YAML. |
Enterprise / Starter | Add the possibility to allow/deny rooms and log events for Auditbot. |
Enterprise / Starter | Support overriding just the server and path in the image digest ConfigMap. |
Enterprise / Starter | Support Element Call in Element X. |
Enterprise / Starter | Matrix Authentication Service and Synapse only use internal paths to communicate, removing the need for hostAliases setup between the two. |
Enterprise | All ESS Images are now hosted behind registry.element.io. |
Enterprise | Synapse workers supporting multiple replicas can now be configured for automatic horizontal scaling. |
Enterprise / Starter | Expose images_digests.yml file in the Download screen for Airgapped customers who want to sync their registry directly with registry.element.io. |
Upgrade Notes
Enterprise / Starter | Upgrade to cert-manager 1.15.3. |
Enterprise / Starter | Operator - Upgrade Python to 3.12, Ansible to 2.17. |
Enterprise / Starter | Upgrade Synapse to v1.116.0. |
Enterprise / Starter | Upgrade Element Web to v1.11.82. |
Enterprise | Update XMPP Bridge to 2.0.1. |
Enterprise | Update Adminbot and Auditbot to 6.3.1. |
Enterprise | Update IRC Bridge to 3.0.2. |
Enterprise | Update Hydrogen to 0.5.0. |
Enterprise / Starter | Update Admin Console to v16.105.4. |
Enterprise / Starter | Upgrade microk8s to 1.31. As per 24.10 releases, the standalone installer only supports upgrading microk8s installed from 23.10 releases. As per 23.10.35/24.04.05/24.05.01, the standalone installer now upgrades microk8s automatically. The microk8s upgrade procedure does not involve an uninstall/reinstall of microk8s anymore. It now will automatically upgrade microk8s to the expected version, and as such, the --upgrade-cluster flag has been removed.Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will have to be reconfigured.During the upgrade, microk8s & workloads will restart several times. Managed addons that require upgrading will be temporarily disabled to be upgraded. This all will induce a small downtime of a couple of minutes. |
Enterprise / Starter | The installer now makes sure the upgrade comes from a supported version. |
Security Issues
Enterprise / Starter | Upgrade to Ansible 9 for security fixes and Python compatibility. |
Bug Fixes
Enterprise | Allow only one VoIP platform (Jitsi or Element Call) to be enabled. |
Enterprise | Fix migration of authentication settings from <24.07.01 with Matrix Authentication Service installed. |
Enterprise / Starter | Fix an issue where, after update, the installer UI would ask to save for changes on the Host screen when the user actually did not click anything. |
Enterprise | Fix monitoring integration tab not rendering. |
Enterprise | Fix Auditbot logs viewer when Matrix Authentication Service is setup. |
Deprecations
Starter | Matrix Content Scanner is not available anymore in Starter Edition. |
Non-LTS Monthly Release Changes
This section summarises all the changes between the previous LTS and this one during the monthly non-LTS releases. Duplicate entries where individual components received upgrades have been removed so only the latest version is mentioned.
You can then compare the below changelog against the above LTS releases for an accurate overall changelog if upgrading from a previous LTS.
Some changes added to non-LTS monthly releases are backported into older LTS releases if required. As such, some of the below features may already be present in a previous LTS. You can check the associated LTS books' respective changelog page to compare.
Release Summary
The required Python versions are now 3.9, 3.10, 3.11. These are available on all supported OS distributions. The installer will attempt to install the required packages in some scenarios. Airgapped customers should ensure that Python 3.9 packages are available in their package mirrors. Alternatively, Python 3.9, 3.10, or 3.11 can be preinstalled on the server in all situations.
Enterprise | This release adds the possibility to enable Matrix Authentication Service during initial setup. Enabling Matrix Authentication Service is experimental; a couple of features do not work yet with it (Auditbot, Adminbot, Element Call, GroupSync, Admin UI). Enabling MAS allows you to use Element X with OIDC or LDAP login. |
Enterprise | This release now makes ESS ElementX ready by default. Any new installation will deploy Matrix Authentication Service. Existing setups will not profit from this change, migration paths are planned later in the future. |
New Features
General | Support knocking with generic_worker federation. |
Enterprise / Starter | Major Change: The standalone installer now upgrades microk8s gracefully and automatically. The microk8s upgrade procedure no longer involves an uninstall/reinstall of microk8s. It now automatically upgrades microk8s to the expected version, and the --upgrade-cluster flag has been removed.Any customization to CNI Configuration in /var/snap/microk8s/current/args/cni-network/cni.yaml will need to be reconfigured. During the upgrade, microk8s will restart, and addons will be disabled to force an upgrade. This process may induce a small downtime of a couple of minutes. |
Enterprise | Status watchers are now golang containers, reducing resources used by the operator and updater. |
Enterprise | Allow configuration of Synapse database connection pool sizes. |
Enterprise | Add a ServiceMonitor to scrape metrics of microk8s ingress. |
Enterprise | Expose Operator & Updater metrics. |
Enterprise | Add support for Outbound webhooks in Hookshot. |
Enterprise | Synapse OIDC support attribute requirements. |
Enterprise | Add a new experimental feature to enable Matrix Authentication Service during ESS bootstrap. |
Enterprise | Simplification of the OIDC provider configuration. After upgrading, please make sure that your OIDC settings were properly migrated to the new view. |
Enterprise | It is now possible to enable the new Matrix Authentication Service when bootstrapping a new ESS setup. It is an experimental feature, incompatible with Groupsync, Element Call, Auditbot, and Adminbot at this time. It is required to try out Element X with OIDC login. |
Enterprise | It is now possible to use LDAP with Matrix Authentication Service. |
Enterprise / Starter | Properly enforce patterns check in UI inputs under cards that can be enabled/disabled. |
Enterprise | Display deployment availability in the UI, in addition to the reconciliation status. |
Enterprise | Element Call is now MAS-Compatible. |
Enterprise | Add the possibility to configure a matrix stats endpoint. |
Enterprise | Setup the onprem-admin user as a MAS admin. |
Enterprise | Allow configuration of empty (no) disallowed IP ranges in Hookshot. |
Enterprise | Validate Synapse Telemetry is consistently set. |
Enterprise / Starter | Synapse improve worker configuration. |
Enterprise / Starter | Allow blocking of non-scanned media. |
Enterprise | Adminbot/Auditbot + MAS compatibility. |
Enterprise / Starter | The UI now properly marks secrets as required when necessary. |
Enterprise / Starter | The reconciliation process now ensures that all secrets are present and shows missing secrets if necessary. |
Enterprise | Add Hookshot permissions configuration. |
Enterprise | Add the possibility to manage Federation dynamically from the Admin Console when Secure Border Gateway is enabled. |
Enterprise / Starter | Speed up initial Synapse deployment. |
Enterprise | Add the possibility to configure user deprovisioning and room cleanup in GroupSync. |
Enterprise | Synapse auto invite: use Synapse native feature, run on background worker if it exists. |
Enterprise | Allow to override a container image without configuring a new digest. |
Enterprise / Starter | Support MSC4186 / Simplified Sliding Sync natively in Synapse. |
Enterprise / Starter | Support authenticated media APIs (MSC3916) in Synapse. |
Enterprise / Starter | Scrape Synapse HAProxy metrics. |
Enterprise | Scrape Adminbot and Auditbot HAProxy metrics. |
Enterprise | Set default volume sizes for Matrix Content Scanner volumes. |
Enterprise | Set default volume sizes for Adminbot, Auditbot & Sydent volumes. |
Enterprise / Starter | The administration interface can now manage users on deployments using Matrix OIDC. |
Enterprise | Administrators can now configure the SBG allowlist within the Admin UI. |
Enterprise / Starter | The user management page now allows admins to toggle the locked status of users. |
Enterprise / Starter | The user management page now displays the primary email address of users. |
Enterprise / Starter | The user management page will now default to showing locked and deactivated users when searching by name. |
Enterprise | Enabling MAS is not experimental anymore, and is now the default setup mode. |
Enterprise | Allow to override a container image without configuring a new digest. |
Enterprise / Starter | Allow configuration of the operator and updater with debug logs. |
Enterprise / Starter | Check for supported Python versions when starting a deployment run. Recreate the virtual environment if it is using the wrong Python version. |
Enterprise / Starter | The installer now makes sure that the microk8s version on the host is supported before starting the upgrade process. |
Enterprise / Starter | Speed improvements in the operator/updater reconciliation process. |
Upgrade Notes
Enterprise | Upgrade Telegram bridge to 0.15.1-mod-1. |
Enterprise | Upgrade WhatsApp bridge to 0.10.7-mod-1. |
Enterprise | Upgrade Sygnal to 0.14.3 to support the latest Firebase API. |
Enterprise | Update Synapse Admin to v16.92.0. |
Enterprise | Update Adminbot to Pipe 6.1.1. |
Enterprise / Starter | Matrix Content Scanner upgrade to 1.0.8. |
Enterprise / Starter | On RHEL and derived platforms, it now requires python 3.11 installed. |
Enterprise | Upgrade SecureBorderGateway to v1.2.0. |
Enterprise | Upgrade Auditbot to 6.1.2 to improve overall request handling efficiency, especially at high-loads. |
Enterprise / Starter | Upgrade to Synapse 1.114.0. |
Enterprise | Upgrade to Element Call 0.6.3 with improved call layout. |
Enterprise | Upgrade to Matrix Authentication Service 0.11.0 and support password auth. |
Enterprise | Synapse registration and password policy settings are now moved to Authentication configuration, under Local Password Database mode. |
Enterprise | Upgrade Hydrogen to v0.4.1-fix. |
Enterprise / Starter | Upgrade to cert-manager 1.12.13. |
Enterprise / Starter | Upgrade ElementWeb to v1.11.81. |
Enterprise / Starter | Services got renamed, -headless suffixes are all removed. If you are using Network Policies, those will need to be upgraded to the new names. |
Enterprise | Global upgrade of the monitoring stack. Victoria Metrics is now on version 1.101. |
Enterprise | Now that Synapse brings native Sliding Sync protocol, the Sliding Sync proxy has been discontinued. Its PostgreSQL cluster instance is being cleaned-up. |
Security Issues
Enterprise | Previous update might have enabled unexpectedly outbound webhooks in Hookshot. If you don't need this feature, make sure that it is disabled in Hookshot integration, under Generic Webhooks settings. |
Enterprise | Better image signatures, enterprise is now published to sigstore. |
Enterprise / Starter | Upgrade to Ansible 8 for security fixes. |
Bug Fixes
Enterprise / Starter | Fix Remove button not working for some integrations. |
Enterprise / Starter | Fix cert-manager upgrade failing to remove old resources. |
Enterprise / Starter | Fix operator and updater having permissions issues under Openshift. |
Enterprise / Starter | Fix Jitsi JVB failing to get ready when STUN servers list is empty and Coturn is not deployed. |
Starter | Fix upgrade failing. |
Enterprise | Fix missing storage class on some Monitoring PVCs. |
Enterprise | Fix media screen on standalone setup. |
Enterprise / Starter | Remove --upgrade-cluster parameter as microk8s is now upgraded gracefully. |
Enterprise | Fix inconsistent behavior when switching between S3/Persistent volume option under the media tab. |
Enterprise / Starter | Fix watchers to avoid triggering unneeded reconciliation loops. |
Enterprise | GroupSync: Fix issue when LDAP identities contain commas in their names. |
Enterprise | Configuring monitoring stack persistent volumes properly in microk8s requires recreating their statefulsets. |
Starter / Enterprise | Fix haproxy failing on IPv4-only nodes. |
Enterprise / Starter | The installer no longer flakes between bootstrap and installer view when the Kubernetes cluster is intermittently unreachable. |
Enterprise | Fix an Ansible error when installing the telemetry script on the local host when user GID != UID. |
Enterprise / Starter | Allow well-known delegation to omit configuration of the ingress entirely without triggering unknown variable errors. |
Enterprise / Starter | Allow configuration of Matrix Content Scanner without a storage class name. |
Enterprise / Starter | Mark Postgres configuration as required for all components that use a Postgres database. |
Enterprise | Mark the source for GroupSync as required. |
Enterprise | Remove workloads and dependent CRs from statuses when they're no longer deployed. |
Enterprise | Fix provisioning of users that are not rate-limited. |
Enterprise | Better identification for the Telegram and WhatsApp bridges in their respective apps. |
Enterprise / Starter | Fix an issue where the cert-manager issuer would try to be created but the cert-manager webhook would not be ready. |
Starter / Enterprise | Fix haproxy failing on IPv4-only nodes. |
Enterprise | Fix monitoring of kube etcd and kube scheduler on microk8s. |
Enterprise | Don't include cert-manager in the airgapped tarball. ESS doesn't install or manage cert-manager in airgapped deploys. |
Enterprise | Avoid leaking Postgres connections when there are issues provisioning Synapse users. |
Enterprise | SIPBridge - Disable Virtual rooms. |
Enterprise | Attempt to detect OpenShift and configure operator & updater installation values appropriately. |
Enterprise / Starter | Fix an issue preventing setup when a proxy is configured on the host. |
Enterprise | Fix a critical issue which would prevent users from accessing Adminbot and Auditbot UI. |
Enterprise | Fixes an issue where Auditbot UI would fail to open because tokens were unable to refresh. |
Enterprise | Revert change of 24.04.07 which prevented Adminbot and Auditbot from doing an initial sync. |
Enterprise | Create new devices for Adminbot and Auditbot to work with the new Rust SDK cryptographic libraries. |
Enterprise | Reduce secrets leaks from operator & updater logs. If you need, for debugging purposes, to enable secrets logging, you must edit the operator & updater deployments and set the environment variable DEBUG_MANIFESTS=1. |
Enterprise / Starter | Refactor Synapse config files to own the priority of each setting managed by ESS. |
Enterprise | Sygnal upgrade to 0.15.0 for further Firebase API fixes. |
Enterprise | Adminbot and Auditbot are currently incompatible with MAS. |
Enterprise | Synapse - Override botocore CA bundle to allow pushing against non-AWS S3 providers. |
Enterprise | Add support for Element Call configuration in Element Well Known file. |
Enterprise | Matrix Authentication Service - Fix UI configuration of certificates for ingresses. |
Enterprise | Minor speed up to initial setup of Synapse. |
Starter | Fix MAU Limit, which was configured at 250 instead of 200. |
Enterprise | Prevent users from manually editing the Auditbot/Adminbot passphrase. |
Enterprise | Fix display of the status of the reconciliation. |
Enterprise | Fix Coturn page causing a memory leak. |
Enterprise / Starter | Ensure the nf_conntrack module is loaded in the kernel when deploying in standalone mode. |
Enterprise / Starter | Fix microk8s services subnet parsing. |
Enterprise / Starter | Fix some CVEs in the operator/updater/conversion webhook. |
Enterprise / Starter | Fix Matrix Content Scanner not working as expected. |
Enterprise | Configure max upload size in Secure Border Gateway request body size limit. |
Enterprise | Prevent users from editing Auditbot and Adminbot passphrases in the UI. |
Enterprise | Enforce pattern checks against inputs under options. |
Enterprise / Starter | Increase Matrix Content Scanner ClamAV startup reliability. |
Enterprise / Starter | Reduce false positives from Matrix Content Scanner. |
Enterprise / Starter | On RHEL and derived platforms, the installer should not rely on platform-python for tasks other than Firewalld and SELinux tasks for microk8s setup. |
Enterprise / Starter | Fix proxy variables configuration check preventing the installer to go through. |
Enterprise / Starter | Fix an issue preventing setup when a proxy is configured on the host. On a proxy configuration errors, the installer will now continue the setup process after displaying the verification error message. |
Enterprise / Starter | Enable MSC 3967 on Synapse to avoid some device verification issues. |
Enterprise | Setup the onprem-admin user as a MAS admin. |
Enterprise / Starter | Fix pulling operator & updater images from behind a proxy. |
Enterprise / Starter | Expired sessions are now automatically logged out of the admin interface. |
Enterprise / Starter | OIDC sessions are now refreshed correctly when the token expires. |
Enterprise | An error is now displayed when the standalone admin UI cannot load the audit/admin interface configuration. |
Enterprise | Ensure operator and updater metrics are correctly scraped. |
Enterprise | Ensure Telemetry room permissions are consistent. |
Enterprise | Ensure component settings for storageClassName override the global setting. |
Enterprise / Starter | Removing an item from a list field will now only delete one item. |
Enterprise | Setup the onprem-admin user as a MAS admin. |
Enterprise / Starter | Fix Synapse being stuck with registration closed even if explicitly allowed. |
Enterprise / Starter | Improve reliability of changing the Postgres password in cluster if the password seed changes. |
Enterprise / Starter | Fix potential permissions issues during microk8s upgrades. |
Enterprise | Construct storage for Matrix Content Scanner if deploying on ESS managed microk8s. |
Enterprise | Correctly import airgapped registry settings when upgrading from before 24.04. |
Enterprise / Starter | Remove unneeded reconciliations due to bad orphan detection. |
Enterprise / Starter | Fix updater metrics scraping. |
Enterprise / Starter | Improve reliability of setting up CoreDNS. |
Enterprise / Starter | Validate that the node IP is excluded from an HTTP Proxy if one is configured. |
Enterprise | Fix empty dashboards (NGinx, Kubernetes Workloads, etc) in Grafana. |
Enterprise | Fix missing VMAlert component which is required to gather record metrics. |
Enterprise / Starter | Fix microk8s stop command not stopping running containers. |
Enterprise / Starter | Improve reliability of some microk8s interactions. |
Deprecations
Enterprise | Element Call participants limits feature is deprecated. The option has been removed from the UI. |
Enterprise | Jitsi and Element Call can not be deployed together. |