Administration

Migrating? Automate your deployment? Configuring Backups? Guides for Administrators here!

Authentication Configuration Examples

Authentication Configuration Examples

Provided below are some configuration examples covering how you can set up various types of Delegated Authentication.

LDAP on Windows AD

OpenID on Microsoft Azure

Before configuring within the installer, you have to configure Microsoft Azure Active Directory.

Set up Microsoft Azure Active Directory

Screenshot 2023-05-03 at 16.30.06.png

For the bridge to be able to operate correctly, navigate to API permissions, add Microsoft Graph APIs, choose Delegated Permissions and add:

Remember to grant the admin consent for those.

To setup the installer, you'll need:

Configure the installer

OpenID on Microsoft AD FS

Install Microsoft AD FS

Before starting the installation, make sure:

You can find a checklist here.

Steps to follow:

Install AD CS

You need to install the AD CS Server Role.

Obtain and Configure an SSL Certificate for AD FS

Before installing AD FS, you are required to generate a certificate for your federation service. The SSL certificate is used for securing communications between federation servers and clients.

Install AD FS

You need to install the AD FS Role Service.

Configure the federation service

AD FS is installed but not configured.

Screenshot 2023-06-22 at 15.55.57.png

Screenshot 2023-06-22 at 15.57.41.png

Screenshot 2023-06-22 at 15.59.27.png

Screenshot 2023-06-22 at 16.04.13.png

Screenshot 2023-06-22 at 16.05.50.png

Add AD FS as an OpenID Connect identity provider

To enable sign-in for users with an AD FS account, create an Application Group in your AD FS.

To create an Application Group, follow theses steps:

Screenshot 2023-06-22 at 16.39.52.png

Screenshot 2023-06-22 at 16.45.44.png

Screenshot 2023-06-22 at 16.56.40.png

Screenshot 2023-06-23 at 09.48.07.png

Screenshot 2023-06-23 at 09.51.06.png

Export Domain Trusted Root Certificate

Configure the installer

Add an OIDC provider in the 'Synapse' configuration after enabling Delegated Auth and set the following fields in the installer:

Screenshot 2023-05-04 at 10.45.23.png

Screenshot 2023-05-03 at 17.27.00.png

Other configurations are documented here.

SAML on Microsoft Azure

Before setting up the installer, you have to configure Microsoft Entra ID.

Set up Microsoft Entra ID

With an account with enough rights, go to : Enterprise Applications

  1. Click on New Application

  2. Click on Create your own application on the top left corner

  3. Choose a name for it, and select `Integrate any other application you don't find in the

gallery`

  1. Click on "Create"

  2. Select Set up single sign on

  3. Select SAML

  4. Edit on Basic SAML Configuration

  5. In Identifier , add the following URL : https://synapse_fqdn/_synapse/client/saml2/metadata.xml

  6. Remove the default URL

  7. In Reply URL , add the following URL : https://synapse_fqdn/_synapse/client/saml2/authn_response

  8. Click on Save

  1. Make a note of the App Federation Metadata Url under SAML Certificates as this will be required in a later step.

  2. Edit on Attributes & Claims

  3. Remove all defaults for additional claims

  4. Click on Add new claim to add the following (suggested) claims (the UID will be used as the MXID):

    • Name: uid , Transformation : ExtractMailPrefix , Parameter 1 : user.userprincipalname

    • Name: email , Source attribute : user.mail

    • Name: displayName , Source attribute : user.displayname

  5. Click on Save

  1. In the application overview screen select Users and Groups and add groups and users which may have access to element

Configure the installer

Add a SAML provider in the 'Synapse' configuration after enabling Delegated Auth and set the following (suggested) fields in the installer:

Troubleshooting

Redirection loop on SSO

Synapse needs to have the X-Forwarded-For and X-Forwarded-Proto headers set by the reverse proxy doing the TLS termination. If you are using a Kubernetes installation with your own reverse proxy terminating TLS, please make sure that the appropriate headers are set.

Backup and Restore

Welcome, ESS Administrators. This guide is crafted for your role, focusing on the pragmatic aspects of securing crucial data within the Element Server Suite (ESS). ESS integrates with external PostgreSQL databases and persistent volumes and is deployable in standalone or Kubernetes mode. To ensure data integrity, we recommend including valuable, though not strictly consistent, data in backups. The guide also addresses data restoration and a straightforward disaster recovery plan.

Software Overview

ESS provides Synapse and Integrations which require an external PostgreSQL and persistent volumes. It offers standalone or Kubernetes deployment.

You'll find below a description of the content of each component data and db backup.

Synapse

Adminbot

Auditbot

Matrix Authentication Service

Sliding Sync

Sydent

Integrator

Bridges (XMPP, IRC, Whatsapp, SIP, Telegram)

Backup Policy & Backup Procedure

There is no particular prerequisite to do before executing an ESS backup. Only Synapse and MAS Databases should be backed up in sync and stay consistent. All other individual components can be backed up on it's own lifecycle.

Backups frequency and retention periods must be defined according to your own SLAs and SLIs.

Data restoration

The following ESS components should be restored first in case of complete restoration. Other components can be restore on their distinctively, on their own time:

Disaster Recovery Plan

In case of disaster recovery, the following components are critical for your system recovery:

The following systems will recover features subsets, and might involve reset & data loss if not recovered :

Security Considerations

Some backups will contain sensitive data, Here is a description of the type of data and the risks associated to it. When available, make sure to enable encryption for your stored backups. You should use appropriate access controls and authentication for your backup processes.

Synapse

Synapse media and db backups should be considered sensitive.

Synapse media backups will contain all user media (avatar, photos, video, files). If your organization is enforcing encrypted rooms, the media will be stored encrypted with each user e2ee keys. If you are not enforcing encryption, you might have media stored in cleartext here, and appropriate measures should be taken to ensure that the backups are safely secured.

Synapse postgresql backups will contain all user key backup storage, where their keys are stored safely encrypted with each user passphrase. Synapse DB will also store room states and events. If your organization is enforcing encrypted rooms, these will be stored encrypted with each user e2ee keys.

The Synapse documentation contains further details on backup and restoration. Importantly the e2e_one_time_keys_json table should not be restored from backup.

Adminbot

Adminbot PV backup should be considered sensitive.

Any user accessing it could read the content of your organization rooms. Would such an event occur, revoking the bot tokens would prevent logging in as the AdminBot and stop any pulling of the room messages content.

Auditbot

Auditbot PV backup should be considered sensitive.

Any user accessing it could read the content of your organization rooms. Would such an event occur, revoking the bot tokens would prevent logging in as the AuditBot and stop any pulling of the room messages content.

Logs stored by the AuditBot for audit capabilities are not encrypted, so any user able to access it will be able to read any logged room content.

Sliding Sync

Sliding-Sync DB Backups should be considered sensitive.

Sliding-Sync database backups will contain Users Access tokens, which are encrypted with Sliding Sync Secret Key. The tokens are only refreshed regularly if you are using Matrix Authentication Services. These tokens give access to user messages-sending capabilities, but cannot read encrypted messages without user keys.

Sydent

Sydent DB Backups should be considered sensitive.

Sydent DB Backups contain association between user matrix accounts and their external identifiers (mails, phone numbers, external social networks, etc).

Matrix Authentication Service

Matrix Authentication Service DB Backups should be considered sensitive.

Matrix Authentication Service database backups will contain user access tokens, so they give access to user accounts. It will also contain the OIDC providers and confidential OAuth 2.0 Clients configuration, with secrets stored encrypted using MAS encryption key.

IRC Bridge

IRC Bridge DB Backups should be considered sensitive.

IRC Bridge DB Backups contain user IRC passwords. These passwords give access to users IRC account, and should be reinitialized in case of incident.

Standalone Deployment Guidelines

General storage recommentations for single-node instances

Adminbot storage:

Auditbot storage:

Synapse storage:

Postgres (in-cluster) storage:

Backup Guidance:

Calculate monthly active users

Take great care when modifying and running queries in your database. Ensure you understand what the queries do and double check that your query is correct.

Incorrect queries can cause irrecoverable data loss.

We recommend you familiarize yourself with Transactions. That way, changes are not immediately written and you can undo any errors.

  1. Connect to your Synapse database
  2. Get the UNIX timestamps in milliseconds for the time frame you are interested in. You want the time set to 00:00:00 GMT. https://www.epochconverter.com/ is a great tool to convert to/from UNIX timestamps.
    a. If you are interested in the current MAU number, pick the date 30 days ago. Note that if you have MAU metrics enabled, this information is also available in Grafana (or your metrics system of choice)
    b. If you want a specific month, get the timestamps for 1st of that month and 1st of the following month
  3. Modify and run the appropriate query below

Get your current MAU number. This uses the timestamp for 30 days ago. For example, if you're running this on January 7, 2025, you would use December 8 2024. This is similar to the query used by Synapse to calculate user count for phone-home stats (Synapse sourse).

SELECT COUNT(*) FROM (
    SELECT user_id
    FROM user_ips
    WHERE
        last_seen >= 1733616000000 AND -- Sunday, 8 December 2024 00:00:00 GMT
        user_id NOT IN (
            SELECT name
            FROM users
            WHERE user_type = 'support'
        )
    GROUP BY user_id
) AS temp;

For reference, this is equal to

SELECT COUNT(*) FROM (
    SELECT
        user_id,
        MAX(timestamp) AS timestamp
    FROM user_daily_visits
    WHERE
        timestamp >= 1733616000000 AND -- Sunday, 8 December 2024 00:00:00 GMT
        user_id NOT IN (
            SELECT name
            FROM users
            WHERE user_type = 'support'
        )
    GROUP BY user_id
) AS temp;

To get retrospective statistics, use this query instead

SELECT COUNT(*) FROM (
    SELECT
        user_id,
        MAX(timestamp) AS timestamp
    FROM user_daily_visits
    WHERE
        timestamp >= 1730419200000 AND -- Friday, 1 November 2024 00:00:00 GMT
        timestamp < 1733011200000 AND -- Sunday, 1 December 2024 00:00:00 GMT
        user_id NOT IN (
            SELECT name
            FROM users
            WHERE user_type = 'support'
        )
    GROUP BY user_id
) AS temp;

Configuring Element Desktop

Element Desktop is a Matrix client for desktop platforms with Element Web at its core.

You can download Element Desktop for Mac, Linux or Windows from the Element downloads page.

See https://web-docs.element.dev/ for the Element Web and Desktop documentation.

Aligning Element Desktop with your ESS deployed Element Web

By default, Element Desktop will be configured to point to the Matrix.org homeserver, however this is configurable by supplying a User Specified config.json.

As Element Desktop is mainly Element Web, but packaged as a Desktop application, this config.json is identical to the config.json ESS will configure and deploy for you at https://<element_web_fqdn>/config.json, so it is recommended to setup Element Desktop using that file directly.

How you do this will depend on your specific environment, but you will need to ensure the config.json is placed in the correct location to be used by Element Desktop.

In the paths above, $NAME is typically Element, unless you use --profile $PROFILE in which case it becomes Element-$PROFILE.

As Microsoft Windows File Explorer by default hides file extensions, please double check to ensure the config.json does indeed have the .json file extension, not .txt.

Customising your desktop configuration

You may wish to further customise Element Desktop, if the changes you wish to make should not also apply to your ESS deployed Element Web, you will need to add them in addition to your existing config.json.

You can find Desktop specific configuration options, or just customise using any options from the Element Web Config docs.

The Element Desktop MSI

Where to download

Customers who have a subscription to the Enterprise edition of the Element Server Suite (ESS) can download a MSI version of Element Desktop. This version of Element Desktop is by default installed into Program Files (instead of per user) and can be used to deploy into enterprise environments. To download, login to your EMS Accoutn and access from the same download page you'd find the enterprise installer, https://ems.element.io/on-premise/download.

Using the Element Desktop MSI

The Element Desktop MSI can be used to install Element Desktop to all desired machines in your environment, unlike the usual installer, you can customise it's install directory (which now defaults to Program Files).

You can customise the installation directory by installing the MSI using, or just generally configuring the APPLICATIONFOLDER:

msiexec /i "Element 1.11.66.msi" APPLICATIONFOLDER="C:\Element"
MSI and config.json

Once users run Element for the first time, an Element folder will be created in their AppData profile specific to that user. By using Group Policy, Logon Scripts, SCCM or whatever other method you like, ensure the desired config.json is present within %APPDATA%\Element. (The config.json can be present prior to the directories creation.)

Guidance on High Availability

ESS makes use of Kubernetes for deployment so most guidiance on high-availability is tied directly with general Kubernetes guidance on high availability.

Kubernetes

High-Level Overview

It is strongly advised to make use of the Kubernetes documentation to ensure your environment is setup for high availability, see links above. At a high-level, Kubernetes achieves high availability through:

How does this tie into ESS

As ESS is deployed into a Kubernetes cluster, if you are looking for high availability you should ensure your environment is configured with that in mind. One important factor is to ensure you deploy using the Kubernetes deployment option, whilst Standalone mode will deploy to a Kubernetes cluster, by definition it exists solely on a single node so options for high availability will be limited.

PostgreSQL

High-Level Overview

To ensure a smooth failover process for ESS, it is crucial to prepare a robust database topology. The following list outline the necessary element to take into consideration:

By carefully preparing the database topology as described, you can ensure that the failover process for ESS is efficient and reliable, minimizing downtime and maintaining data integrity.

How does this tie into ESS

As ESS relies on PostgreSQL for its database if you are looking for high availability you should ensure your environment is configured with that in mind. The database replicas can be achieved the same way in both Kubernetes and Standalone deployment, as the database is not managed by ESS.

ESS failover plan

This document outlines a high-level, semi-automatic, failover plan for ESS. The plan ensures continuity of service by switching to a secondary data center (DC) in the event of a failure in the primary data center.

Prerequisites

ESS Architecture for failover capabilities based on 3 datacenters

DC1 (Primary)

DC2

DC3

Failover Process

When DC1 experiences downtime and needs to be failed over to DC2, follow these steps:

You should decline your own failover procedure based on this high-level failover overview. By doing so, you can ensure that ESS continues to operate smoothly and with minimal downtime, maintaining service availability even when the primary data center goes down.

Migrating from Self-Hosted to ESS

This document is currently work-in-progress and might not be accurate. Please speak with your Element contact if you have any questions.

Preparation

This section outlines what you should do ahead of the migration in order to ensure the migration goes as quickly as possible and without issues.

Note that the database and media may be duplicated/stored twice on your ESS host during the import process depending on how you do things.

Setup your new ESS server

Follow the ESS docs for first-time installation, configuring to match your existing homeserver before proceeding with the below.

The Domain Name on the Domains page during the ESS initial setup wizard must be the same as you have on your current setup. The other domains can be changed if you wish.

To make the import later easier, we recommend you select the following Synapse Profile. You can change this as required after the import.

After the ESS installation, you can check your ESS Synapse version on the Admin -> Server Info page:

Export your old Matrix server

SSH to your old Matrix server

You might want to run everything in a tmux or a screen session to avoid disruption in case of a lost SSH connection.

Upgrade your old Synapse to the same version EES is running

Follow https://element-hq.github.io/synapse/latest/upgrade.html

Please be aware that ESS, especially our LTS releases may not run the latest available Synapse release. Please speak with your Element contact for advice on how to resolve this issue. Note that Synapse does support downgrading, but occationally a new Synapse version includes database schema changes and this limits downgrading. See https://element-hq.github.io/synapse/latest/upgrade.html#rolling-back-to-older-versions for additional details and compatible versions.

Start Synapse, make sure it's happy.
Stop Synapse

Create a folder to store everything

mkdir -p /tmp/synapse_export
cd /tmp/synapse_export

The guide from here on assumes your current working directory is /tmp/synapse_export.

Set restrictive permissions on the folder

If you are working as root: (otherwise set restrictive permissions as needed):

chmod 700 /tmp/synapse_export

Copy Synapse config

Get the following files :

Stop Synapse

Once Synapse is stopped, do not start it again after this

Doing so can cause issues with federation and inconsistent data for your users.

While you wait for the database to export or files to transfer, you should edit or create the well-known files and DNS records to point to your new EES host. This can take a while to update so should be done as soon as possible in order to ensure your server will function properly when the migration is complete.

Database export

Dump your database:

pg_dump -Fc -O -h <dbhost> -U <dbusername> -d <dbname> -W -f synapse.dump

Import to your ESS server

Database import

Enter a bash shell on the Synapse postgres container:

Stop Synapse

kubectl .... replicas=0

Note that this might differ depending on how you have your Postgres managed. Please consult the documentation for your deployment system.

kubectl exec -it -n element-onprem synapse-postgres-0 --container postgres  -- /bin/bash

Then on postgres container shell run:

psql -U synapse_user synapse

The following command will erase the existing Synapse Database without warning or confirmation. Please ensure that is is the correct database and there is no production data on it.

DO $$ DECLARE
r RECORD;
BEGIN
  FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = current_schema()) LOOP
    EXECUTE 'DROP TABLE ' || quote_ident(r.tablename) || ' CASCADE';
  END LOOP;
END $$;

DROP sequence cache_invalidation_stream_seq;
DROP sequence state_group_id_seq;
DROP sequence user_id_seq;
DROP sequence account_data_sequence;
DROP sequence application_services_txn_id_seq;
DROP sequence device_inbox_sequence;
DROP sequence event_auth_chain_id;
DROP sequence events_backfill_stream_seq;
DROP sequence events_stream_seq;
DROP sequence presence_stream_sequence;
DROP sequence receipts_sequence;
DROP sequence un_partial_stated_event_stream_sequence;
DROP sequence un_partial_stated_room_stream_sequence;

Use \q to quit, then back on the host run:

gzip -d synapse_export.sql.gz
sudo cp synapse_export.sql /data/postgres/synapse/
# or
kibectl --namespace element-onprem cp synapse_export.sql element-onprem synapse-postgres-0:/tmp

Finally on the pod:

cd /var/lib/postgresql/data
# or
cd /tmo

pg_restore <connection> --no-owner --role=<new role> -d <new db name> dump.sql

Mobile client provisioning

The default sign-in flow in the mobile app requires the user to enter the correct account provider (homeserver) which is error prone and also not effortless.

In order to improve the user experience, and to skip this step, there are 3 options to specify the account provider or other configuration for the user:

This is a new feature available in since 25.05.2 (iOS) and 25.06.0 (Android).

Deeplinking

Deeplinking is the simplest option (unless the customer is already using MDM for their users or has a custom app).

How does it work?

The customer would send an on-boarding email to its users which contains two major steps:

  1. Download the app
  2. Sign in to the app

The second step then contains the deeplink which opens the app and sets the desired account provider. Optionally, a QR code of the link would be included in case the user can't access the mail on their phone.

Here's an example of how such e-mail could look like. Note that currently ESS does not have means for composing and sending these emails to users automatically.

The configuration set by opening the app with the deeplink is retained only until the app is running. If the app is killed and opened without using the deeplink, the default behaviour will apply (e.g. user has to enter account provider).

If account provider is already set by MDM or customer publishes their own app which does not allow using an arbitrary account provider, the account provider in the deeplink is ignored.

If the user tries to open the app with the deeplink, but does not have the app installed yet on their phone, the user lands on a web page that guides them to install it.

How to Create the Deeplink?

The format for the deeplink is:

https://mobile.element.io/<app>/?account_provider=<example.com>&login_hint=<alice>

The <app> specifies the mobile app to open:

Note: while Element X is supported, it is expected that customers use the Element Pro app.

The <example.com> specifies the server name of the account provider aka homeserver, e.g. example.com. Note that for backward compatibility purposes the hs_url works as an alternative for the Element (classic) app, but it expects an URL, e.g. https://example.com.

The <alice> is intended to populate the username for the login. Since this is implemented by providing it as the login_hint parameter in the OIDC protocol, the exact value/format of it depends on which means of authentication is used.

When MAS is used without an upstream identity provider (e.g. no SSO for users), the format mxid:@alice:example.org is expected. An example deeplink/URL for @alice:example.com to sign to Element Pro in such a case is: https://mobile.element.io/element-pro/?account_provider=example.com&login_hint=mxid:@alice:example.org

When MAS is used with an upstream identity provider, you need to set forward_login_hint: true in your MAS configuration for the hint to be forwarded. See the MAS documentation for more details. The hint itsef depends on what the provider expects, but in many cases the email address is used as the user ID, e.g. alice@example.com. Also, in case of SSO providing the username is less critical, as user is very likely already signed in because they have used other apps with the SSO.

To create a QR code of the link, online services can be used.

Please refer to Apple and Google documentation how to use badges for the download buttons.

Note: Redirect from Element Web

Note: when a user tries to access Element web app with a browser on a mobile device (e.g. they go to https://chat.example.com), the user is directed to a page which also guides them to download the native mobile app and has a button to open the app with the deeplink. In such a case the account provider is set to the one that the Element web installation specifies.

Mobile Device Management (MDM)

If the customer is using MDM, the account provider can be delivered to the user's phone via the AppConfig.

When applying the MDM configuration to Element Pro, the Application/Bundle ID is io.element.enterprise. The key for specifying the account provider is accountProvider. The value must be the server name, e.g. example.com.

The end user just needs to download the app (if not installed by the MDM solution already) and open it.

Note that when MDM is used for specifying the account provider, other means like deeplink are ignored.

Custom app

If the customer is publishing their own custom app in the stores, the account provider(s) is built into the app.

The account provider(s) are specified by the accountProviders parameter of the custom app's build pipeline (by Element who sets up the pipeline). Details are provided in the pipeline inputs documentation.

With custom app it is possible to specify more than one account provider. In such a case the user would still need to make a selection but would not need to input manually.

Note that the account provider(s) in the custom app are overriden by the one from MDM (if exists).

If a custom app is opened with a deeplink that also specifies account provider, the one in the deeplink is only applied when the accountProviders parameter of the custom app contains *. The latter means that users of the custom app are allowed to use arbitrary account providers.

Starting and Stopping ESS Services

Stopping a component

To stop a component, such as Synapse, it is necessary to stop the operator :

kubectl scale deploy/element-operator-controller-manager -n operator-onprem --replicas 0

Once the operator is stopped, you can delete the Synapse resource to remove all Synapse workloads :

kubectl delete synapse/first-element-deployment -n element-onprem

To get a list of resources that you can remove, you can look at the following command :

kubectl get elementdeployment/first-element-deployment -n element-onprem  --template='{{range $key, $value := .status.dependentCRs}}{{$key}}{{"\n"}}{{end}}'

Example :

ElementWeb/first-element-deployment
Hookshot/first-element-deployment
Integrator/first-element-deployment
MatrixAuthenticationService/first-element-deployment
Synapse/first-element-deployment
SynapseAdminUI/first-element-deployment
SynapseUser/first-element-deployment-adminuser-donotdelete
SynapseUser/first-element-deployment-telemetry-donotdelete
WellKnownDelegation/first-element-deployment

Starting a component

To stop a component, such as Synapse, it is necessary to start the operator :

kubectl scale deploy/element-operator-controller-manager -n operator-onprem --replicas 1

Because the Synapse resource will automatically have been recreated by the updater, the operator on startup will automatically detect it and recreate all synapse workloads.