Element On-Premise Documentation

Introduction to Element Enterprise

What is Element Enterprise?

Element Enterprise provides an enterprise-grade secure communications platform that can be run either on your own premise or in our Element Cloud. Element Enterprise includes all of the security and privacy features that you get with Element:

and combines them with the following unique Enterprise specific features:

Given the flexibility afforded by this platform, ours has a number of moving parts to configure. This documentation will step you through architecting and deploying Element Enterprise On-Premise.

Deploying to a Single Node or Multiple Nodes?

Element Enterprise On-Premise can be deployed both to a single node or a set of multiple nodes. In the case of the multiple node deployment, this requires Kubernetes, a container orchestration platform. In the case of our single node deployment, our installer deploys microk8s (a smaller lightweight distribution of Kubernetes) and deploys our application to that microk8s instance.

In general, regardless of if you pick a single node deployment or a multiple node deployment, you will need a base level of hardware to support the application.

For scenarios that utilise closed federation, Element recommends a minimum of 4 vCPUs/CPUs and 16GB ram for the host(s) running synapse pods.

For scenarios that utilise open federation, Element recommends a minimum of 8 vCPUs/CPUs and 32GB ram for the host(s) running synapse pods.

Architecture

This document gives an overview of our secure communications platform architecture:

Matrix Architecture - Generic Kubernetes Deployment.png

(Please click on the image to view it at 100%.)

Comprising our secure communications platform are the following components:

For each of the components in this list (excluding postgresql, groupsync, adminbot, auditbot, and prometheus), you must provide a hostname on your network that meets this criteria:

It is possible to deploy Element Enterprise On-Premise with self-signed certificates and without proper DNS in place, but this is not ideal as the mobile clients and federation do not work with self-signed certificates. Information on how to use self-signed certificates and hostname mappings instead of DNS can be found in How to Setup Local Host Resolution Without DNS

In addition to hostnames for the above, you will also need a hostname and PEM encoded certificate key/cert pair for your base domain. If we were deploying a domain called example.com and wanted to deploy all of the software, we would have the following hostnames in our environment that needed to meet the above criteria:

As mentioned above, this list excludes postgresql, groupsync, adminbot, auditbot, and prometheus.

Further, if you want to do voice or video across network boundaries (ie: between people not on the same local network), you will need a TURN server. If you already have one, you do not have to set up coturn. If you do not already have a TURN server, you will want to set up coturn and if your server is behind NAT, you will need to have an external IP in order for coturn to work.

Installation

Multiple Nodes

For a multiple node installation, make sure you have a Kubernetes platform deployed that you have access to and head over to Kubernetes Installations

Single Node

For a single node installation, please note that we support these on the following platforms:

Once you have a server with one of these installed, please head over to Single Node Installations

Kubernetes Installations

How to Use the Installer

This video covers the single node installer, but many of the concepts are also applicable to our multi-node installer.

Overview

Our Element Enterprise Kubernetes Installer can handle the installation of Element Enterprise into your production kubernetes (k8s) environment.

To get started with a kubernetes installation, there are several things that need to be considered and this guide will work through them:

Once these areas have been covered, you'll be able to install a production environment!

Unpacking the Installer

Please make sure that you unpack element-enterprise-installer onto a system that has access to your k8s environment. The directory that it unpacks into will be referenced in this document as the installer directory.

You will also need to create a directory for holding the configurations for the installer. This will be referenced as the config directory going forward.

mkdir  ~/.element-onpremise-config

k8s Environments

Element Enterprise Installer allows you to either deploy directly into a kubernetes environment or to render a set of manifests for a future deployment in a kubernetes environment.

To configure your kubernetes environment for a direct deployment, you need to :

An example k8s.yml file would look like:

provider_storage_class_name: gp8-delete  # select an available storage class
ingress_annotations: ## below are expected annotations for an aws deployment
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/group.name: global
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/ip-address-type: ipv4
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
synapse_ingress_annotations:  # below are required annotations if using the NGINX ingress controller
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
tls_managed_externally: true  # true if the certificates are managed externaly to k8s
security_context_force_uid_gid: true  # true to enable pod runAsUser and fsGroup in security context. false if it should not be used, in the case of openshift for example.
security_context_set_seccomp: true  # true to enable RuntimeDefault pod seccomp. false if it should not be used, in the case of openshift for example.
operator_namespace: <namespace to create to deploy the operator>
element_namespace: <namespace to create to deploy the element resources>
k8s_auth_context: <the k8s auth context>
out_dir: # Absolute path to the directory where to render manifests, if render mode is used
# operator_manager_limits:  # Can be used to defined upper limits if the default one are not large enough for your operator deployment
#   cpu: "2"
#   memory: 8Gi

If you do not want to deploy directly to kubernetes, but wish to render manifests instead, set all of the above mentioned variables except for k8s_auth_context and define a value for the parameter out_dir, which specifies where to write the kubernetes manifests. Further, when you go to run the installer, you need to invoke it as such:

bash install.sh  ~/.element-onpremise-config --target render

Using the above syntax, you will have a set of manifests written out to out_dir that you can then deploy into your kubernetes environment.

N.B. You will need to set your ingress controller's upload size to be at least 50 Mb to match synapse's default upload size if you wish to be able to have users upload files up to 50 Mb in size. Instructions for doing this with nginx are included in the parameters.yml section below.

Postgresql Database

The installation requires that you have a postgresql database with a locale of C and UTF8 encoding set up. See https://matrix-org.github.io/synapse/latest/postgres.html#set-up-database for further details.

Please make note of the database hostname, database name, user, and password as you will need these to begin the installation.

TURN Server

For installations in which you desire to use video conferencing functionality, you will need to have a TURN server installed and available for Element to use.

If you do not have an existing TURN server, our installer can configure one for you by following the extra steps in Setting Up Jitsi and TURN With the Installer.

If you have an existing TURN server, please create a file called synapse/turn.yml in your config directory and put the following in it:

turn_uris: [ "turn:turn.matrix.org?transport=udp", "turn:turn.matrix.org?transport=tcp" ]
turn_shared_secret: "n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons"
turn_user_lifetime: 86400000
turn_allow_guests: True

based on your TURN server specifics. This will allow the installer to configure synapse to use your TURN server.

A few notes on TURN servers:

SSL Certificates

For SSL Certificates, you have three options:

In the case of LetsEncrypt, your hostnames must be accessible on the internet.

You will need to configure certificates for the following names:

Using our example hosts, this would mean that we need certificates for:

Certificates without LetsEncrypt

If you have certificates for all of the aforementioned host names, then you can simply place the .crt and .key files in the certs directory under the config directory. Certificates in the certs directory must take the form of fqdn.cert and fqdn.key.

Certificates with LetsEncrypt

Our installer also supports using LetsEncrypt to build certificates for your host names and automatically install them into your environment. If your hosts are internet accessible, this is the easiest method and only requires an admin email address to provide to LetsEncrypt.

parameters.yml

Now it is time to set parameters.yml. A sample has been provided and to get started, it is easiest to do:

cp config-sample/parameters.yml.sample ~/.element-onpremise-config/parameters.yml

Using the example hostnames of element.local and synapse.local (not resolvable on the internet), we would set the following parameters first in parameters.yml:

domain_name: local
element_fqdn: element.local
synapse_fqdn: synapse.local

Next, we need to set the variables related to Postgres. As you are installing into kubernetes, you will need to set the following for your Postgres database:

postgres_create_in_cluster: false
postgres_fqdn: `Postgres Server`
postgres_user: `Postgres User`
postgres_db: `Postgres Database for Element`

The next line states:

media_size: "50Gi"

You wll want to adjust that to match the size of storage you've allocated for your media. It must be at least 50Gb.

The next section pertains to certmanager. If you are using your own certificates, please leave these items both blank, as such:

certmanager_issuer:
certmanager_admin_email:

If you have chosen to use letsencrypt, please specify “letsencrypt” for the certmanager_issue and an actual email address for who should manage the certificates for certmanager_admin_email:

certmanager_issuer: 'letsencrypt'
certmanager_admin_email: 'admin@mydomain.com'

Starting with installer 2022-08.02, we have added two mandatory variables related to telemetry data. These are max_mau_users and strict_mau_users_limit. You should set max_mau_users to the value defined in your contract with Element. If you set this number above your contractual limit, then the software will allow you to exceed your contractual limit and Element will bill you appropriately for the overage.

Setting strict_mau_users_limit to true forces synapse to cap the number of monthly active users to the value defined in max_mau_users. Say for example, you've paid Element for 1,000 monthly active users and don't want to exceed that, you would set:

max_mau_users: 1000
strict_mau_users_limit: true

Let's say that you paid Element for 1,000 monthly active users, but didn't mind going over provided that you didn't exceed 2,000 monthly active users. In this scenario, you would set:

max_mau_users: 2000
strict_mau_users_limit: true

For more information on the data that Element collects, please see: What Telemetry Data is Collected by Element?

You will also see two paths:

# media_host_data_path: "/mnt/data/synapse-media"
# postgres_data_path: "/mnt/data/synapse-postgres"

For kubernetes installations, please make sure these are commented out.

You will also notice two lines towards the end regarding synapse_registration and tls_managed_externally. In most cases, you can leave these alone, but if you wish to close synapse registration or have your TLS managed externally, you may set them at this time.

If you are using nginx as your ingress controller and wish to send files up to 50 Mb in size, please add these two lines to parameters.yml:

synapse_ingress_annotations:
  nginx.ingress.kubernetes.io/proxy-body-size: "50m"

secrets.yml

Now we move on to configuring secrets.yml. You will need the following items here:

To build a secrets.yml with the macaroon key, the registration shared secret, the generic shared secret, and the signing key already filled in, please run:

sh build_secrets.sh
mv secrets.yml  ~/.element-onpremise-config/

You will need to uncomment and set your postgres_password field to the proper password for your database.

Do not forget to also set the values for ems_image_store_username and ems_image_store_token, which will both be provided by Element.

If you have a paid docker hub account, you can specify your username and password to avoid being throttled in the dockerhub_username and dockerhub_token fields. This is optional.

Extra Configuration Items

It is possible to configure anything in Synapse's homeserver.yaml or Element’s config.json.

To do so, you need to create json or yaml files in the appropriate directory under the config directory. These files will be merged to the target configuration file.

Samples are available in config-sample under the installer directory.

To configure synapse:

To configure element:

For specifics on configuring permalinks for Element, please see Setting up Permalinks.

For specifics on setting up Delegated Authentication, please see Setting up Delegated Authentication With the Installer

For specifics on setting up Group Sync, please see Setting up Group Sync

For specifics on setting up the Integration Manager, please see Setting Up the Integration Manager With the Installer

For specifics on setting up GitLab, GitHub, and JIRA integrations, please see Setting up GitLab, GitHub, and JIRA Integrations With the Installer

For specifics on setting up Chatterbox, please see: Setting Up Chatterbox

For specifics on setting up Adminbot and Auditbot, please see: Setting up Adminbot and Auditbot

For specifics on setting up the Enterprise Admin Dashboard, please see: Configuring the Enterprise Admin Dashboard

For specifics on pointing your installation at an existing Jitsi instance, please see Setting Up Jitsi and TURN With the Installer

For specifics on configuring the Teams Bridge, please see Setting Up the Teams Bridge

For specifics on configuring the Telegram Bridge, please see Setting Up the Telegram Bridge

For specifics on configuring the IRC Bridge, please see Setting Up the IRC Bridge

For specifics on configuring the XMPP Bridge, please see Setting Up the XMPP Bridge

Installation

Let’s review! Have you considered:

Once you have the above sections taken care of and your parameters.yml and secrets.yml files are in order, you are ready to begin the actual installation.

From the installer directory, run:

bash install.sh  ~/.element-onpremise-config

The first run should go for a little while and then exit, instructing you to log out and back in.

Please log out and back in and re-run the installer from the installer directory again:

bash install.sh  ~/.element-onpremise-config

Upgrading between installer versions

Installer 2022-08.02

Added two mandatory variables related to telemetry data:
max_mau_users and strict_mau_users_limit

They need to be set (ie: max_mau_users = 1000 and strict_mau_users_limit = true).

Single Node Installations

How to Use the Installer

Overview

Our Element Enterprise Single Node Installer can handle the installation of environments in which only one server is available. Our single node environment consists of a single server with microk8s running that we deploy our Element Enterprise Operator to, resulting in a fully functioning Synapse server with Element Web.

To get started with a single node installation, there are several things that need to be considered and this guide will work through them:

Once these areas have been covered, you’ll be able to install a single node environment!

Operating System

To get started, we have tested on Ubuntu 20.04 and Red Hat Enterprise Linux 8.5 and suggest that you start there as well. For x86_64, you can grab an Ubuntu iso here:

https://releases.ubuntu.com/20.04.3/ubuntu-20.04.3-live-server-amd64.iso

or you can get Red Hat Enterprise Linux 8 with a Developer Subscription

https://developers.redhat.com/content-gateway/file/rhel-8.6-x86_64-dvd.iso

Note that future references in this document to EL reference Enterprise Linux.

Ubuntu Specific Directions

Make sure to select docker as a package option. Do set up ssh.

Once you log in, please run:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-signedjson pwgen -y

The installer requires that you run it as a non-root user who has sudo permissions. Please make sure that you have a user who can use sudo. If you wanted to make a user called element-demo that can use sudo, the following commands (run as root) would achieve that:

useradd element-demo
gpasswd -a element-demo sudo

EL Specific directions

Make sure to select "Container Management" in the "Additional Software" section.

Once you log in, please run:

sudo yum update -y
sudo yum install podman-docker python-pip -y
sudo yum install
https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y
sudo update-alternatives --config python3

Add the following lines to /etc/security/limits.conf:

*              soft    nofile  100000
*              hard    nofile  100000

Then, run:

sudo yum install make gcc python3-devel pwgen -y
pip3 install signedjson --user

The installer requires that you run it as a non-root user who has sudo permissions. Please make sure that you have a user who can use sudo. If you wanted to make a user called element-demo that can use sudo, the following commands (run as root) would achieve that:

useradd element-demo
gpasswd -a element-demo wheel

Setting up the Configuration Directory

You should have the installer unpacked in a directory on your server. We will refer to this as the installer directory. You will also need to create a configuration directory that we will call the config directory. Both the parameters.yml and secrets.yml file live in the config directory.

To create the configuration directory, run the following:

mkdir ~/.element-onpremise-config

Network Specifics

Element Enterprise On-Premise needs to bind and serve content over:

microk8s needs to bind and serve content over:

For more information, see https://microk8s.io/docs/ports.

In a default Ubuntu installation, these ports are allowed through the firewall. You will need to ensure that these ports are passed through your firewall.

For EL, you need to explicitly open the above ports and enabling masquerading:

sudo firewall-cmd --add-service={http,https} --permanent
sudo firewall-cmd --add-port=16443/tcp --add-port=10250/tcp --add-port=10255/tcp --add-port=25000/tcp --add-port=12379/tcp --add-port=10257/tcp --add-port=10259/tcp --add-port=19001/tcp --permanent
sudo firewall-cmd --add-masquerade --permanent
sudo firewall-cmd --reload 

Further, you need to make sure that your host is able to access the following hosts on the internet:

Further, you will also need to make sure that your host can access your distributions' package repositories. As these hostnames can vary, it is beyond the scope of this documentation to enumerate them.

Network Proxies

We also cover the case where you need to use a proxy to access the internet. Please see this article for more information: Configuring a microk8s Single Node Instance to Use a Network Proxy

Unpacking the Installer

Please make sure that you unpack element-enterprise-installer onto your single node system. The directory that it unpacks into will be referenced in this document as the installer directory.

Postgresql Database

The installation requires that you have a postgresql database with a locale of C and UTF8 encoding set up. See https://github.com/matrix-org/synapse/blob/develop/docs/postgres.md#set-up-database for further details.

If you have this already, please make note of the database name, user, and password as you will need these to begin the installation.

If you do not already have a database, then the single node installer will set up PostgreSQL on your behalf.

TURN Server

For installations in which you desire to use video conferencing functionality, you will need to have a TURN server installed and available for Element to use.

If you do not have an existing TURN server, our installer can configure one for you by following the extra steps in Setting Up Jitsi and TURN With the Installer

If you have an existing TURN server, please create a file called synapse/turn.yml in your config directory and put the following in it:

turn_uris: [ "turn:turn.matrix.org?transport=udp", "turn:turn.matrix.org?transport=tcp" ]
turn_shared_secret: "n0t4ctuAllymatr1Xd0TorgSshar3d5ecret4obvIousreAsons"
turn_user_lifetime: 86400000
turn_allow_guests: True

based on your TURN server specifics. This will allow the installer to configure synapse to use your TURN server.

A few notes on TURN servers:

SSL Certificates

For SSL Certificates, you have three options:

In the case of Signed certificates or LetsEncrypt, your hostnames must be accessible on the internet.

In the case of self-signed certificates, these are acceptable for a PoC (proof of concept) environment, but will not be supported in a production environment as the security risk would be too high. Configuring mobile clients and federation will not be possible with self-signed certificates.

You will need to configure certificates for the following names:

Using our example hosts, this would mean that we need certificates for:

Certificates without LetsEncrypt

If you have certificates for all of the aforementioned host names, then you can simply place the PEM encoded .crt and .key files in the certs directory under the configuration directory. Certificates in the certs directory must take the form of fqdn.crt and fqdn.key.

Self-signed certificates with mkcert

For information on using self-signed certificates with mkcert, please see this article: Using Self-Signed Certificates with mkcert

Certificates with LetsEncrypt

Our installer also supports using LetsEncrypt to build certificates for your host names and automatically install them into your environment. If your hosts are internet accessible, this is the easiest method and only requires an admin email address to provide to LetsEncrypt.

parameters.yml

Now it is time to set parameters.yml. A sample has been provided and to get started, it is easiest to do:

cp config-sample/parameters.yml.sample ~/.element-onpremise-config/parameters.yml

Using the example hostnames of element.local and synapse.local (not resolvable on the internet), we would set the following parameters first in parameters.yml:

domain_name: local
element_fqdn: element.local
synapse_fqdn: synapse.local

Next, we need to set the variables related to Postgres. If you do not have an existing Postgres server, do not make any changes. If you have an existing Postgres server, set the following:

postgres_create_in_cluster: false
postgres_fqdn: `Postgres Server`
postgres_user: `Postgres User`
postgres_db: `Postgres Database for Element`

The next line states:

media_size: "50Gi"

You wll want to adjust that to match the size of storage you've allocated for your media. It must be at least 50Gb.

The next section pertains to certmanager. If you are using your own certificates, please leave these items both blank, as such:

certmanager_issuer:
certmanager_admin_email:

If you have chosen to use letsencrypt, please specify “letsencrypt” for the certmanager_issue and an actual email address for who should manage the certificates for certmanager_admin_email:

certmanager_issuer: 'letsencrypt'
certmanager_admin_email: 'admin@mydomain.com'

Starting with installer 2022-08.02, we have added two mandatory variables related to telemetry data. These are max_mau_users and strict_mau_users_limit. You should set max_mau_users to the value defined in your contract with Element. If you set this number above your contractual limit, then the software will allow you to exceed your contractual limit and Element will bill you appropriately for the overage.

Setting strict_mau_users_limit to true forces synapse to cap the number of monthly active users to the value defined in max_mau_users. Say for example, you've paid Element for 1,000 monthly active users and don't want to exceed that, you would set:

max_mau_users: 1000
strict_mau_users_limit: true

Let's say that you paid Element for 1,000 monthly active users, but didn't mind going over provided that you didn't exceed 2,000 monthly active users. In this scenario, you would set:

max_mau_users: 2000
strict_mau_users_limit: true

For more information on the data that Element collects, please see: What Telemetry Data is Collected by Element?

You will also see two paths:

media_host_data_path: "/mnt/data/synapse-media"
# postgres_data_path: "/mnt/data/synapse-postgres"

For all installations, media_host_data_path should be uncommented. For installations in which you are letting the installer install postgresql for you, please uncomment the postgres_data_path line.

The next lines concern images_dir and local_registry. These are only needed in an air-gapped environment. If you are installing into an air-gapped environment, please see: Using the Single Node Installer in an Air-Gapped Environment

The next item in the configuration is the microk8s DNS resolvers. By default, the installer will use Google's publicly available DNS servers. If you have defined your hosts on a non-publicly available DNS server, then you should use your DNS servers instead of the publicly available Google DNS servers. Let's assume that your local dns servers are 192.168.122.253 and 192.168.122.252. To use those servers, you would need to add this line:

microk8s_dns_resolvers: "192.168.122.253,192.168.122.252"

You will also notice two lines towards the end regarding synapse_registration and tls_managed_externally. In most cases, you can leave these alone, but if you wish to close synapse registration or have your TLS managed externally, you may set them at this time.

Further, if you are not using DNS for hostname mapping, you will need to configure the host_aliases parameter in this file and that is documented in How to Setup Local Host Resolution Without DNS.

secrets.yml

Now we move on to configuring secrets.yml. You will need the following items here:

To build a secrets.yml with the macaroon key, the registration shared secret, the generic shared secret, and the signing key already filled in, please run:

sh build_secrets.sh
mv secrets.yml  ~/.element-onpremise-config/

If you are using your own Postgres server, you will need to uncomment and fill in the postgres_passwd. If you are letting the installer install Postgres for you, then you will need to set a random password. You can generate a random password with:

pwgen 32 1

and then insert that value in the postgres_passwd field, making sure that you uncomment the line.

Do not forget to also set the values for ems_image_store_username and ems_image_store_token, which will both be provided by Element.

If you have a paid docker hub account, you can specify your username and password to avoid being throttled in the dockerhub_username and dockerhub_token fields. This is optional.

Extra Configuration Items

It is possible to configure anything in Synapse's homeserver.yaml or Element’s config.json.

To do so, you need to create json or yaml files in the appropriate directory under the config directory. These files will be merged to the target configuration file.

Samples are available in config-sample under the installer directory.

To configure synapse:

To configure element:

For specifics on configuring permalinks for Element, please see Setting up Permalinks With the Installer

For specifics on setting up Delegated Authentication, please see Setting up Delegated Authentication With the Installer

For specifics on setting up Group Sync, please see Setting up Group Sync with the Installer

For specifics on setting up the Integration Manager, please see Setting Up the Integration Manager With the Installer

For specifics on setting up GitLab, GitHub, and JIRA integrations, please see Setting up GitLab, GitHub, and JIRA Integrations With the Installer

For specifics on setting up Chatterbox, please see: Setting Up Chatterbox

For specifics on setting up Adminbot and Auditbot, please see: Setting up Adminbot and Auditbot

For specifics on setting up the Enterprise Admin Dashboard, please see: Configuring the Enterprise Admin Dashboard

For specifics on pointing your installation at an existing Jitsi instance, please see Setting Up Jitsi and TURN With the Installer

For specifics on configuring the Teams Bridge, please see Setting Up the Teams Bridge

For specifics on configuring the Telegram Bridge, please see Setting Up the Telegram Bridge

For specifics on configuring the IRC Bridge, please see Setting Up the IRC Bridge

For specifics on configuring the XMPP Bridge, please see Setting Up the XMPP Bridge

Installation

Let’s review! Have you considered:

Once you have the above sections taken care of and your parameters.yml and secrets.yml files are in order, you are ready to begin the actual installation.

From the installer directory, run: (Note: You can replace ~/.element-onpremise-config with whatever you have specified for your config directory.)

bash install.sh  ~/.element-onpremise-config

The first run should go for a little while and then exit, instructing you to log out and back in.

Please log out and back in and re-run the installer from the installer directory again:

bash install.sh  ~/.element-onpremise-config

Once this has finished, you can run:

kubectl get pods -n element-onprem

And you should get similar output to:

NAME                                        READY   STATUS    RESTARTS   AGE
app-element-web-c5bd87777-rqr6s             1/1     Running   1          29m
server-well-known-8c6bd8447-wddtm           1/1     Running   1          29m
postgres-0                                  1/1     Running   1          40m
instance-synapse-main-0                     1/1     Running   2          29m
instance-synapse-haproxy-5b4b55fc9c-hnlmp   1/1     Running   0          20m

At this time, you should also be able to browse to: https://fqdn and create a test account with Element on your new homeserver. Using our example values, I am able to go to https://element.local/ and register an account, sign in and begin testing!

Upgrading between installer versions

Installer 2022-08.02

Added two mandatory variables related to telemetry data:
max_mau_users and strict_mau_users_limit

They need to be set (ie: max_mau_users = 1000 and strict_mau_users_limit = true).

Using the Installer in an Air-Gapped Environment

Defining Air-Gapped Environments

An air-gapped environment is any environment in which the running hosts will not have access to the greater internet. This proposes a situation in which these hosts are unable to get access to various needed bits of software from Element and also are unable to share telemetry data back with Element.

For some of these environments, they can be connected to the internet from time to time and updated during those connection periods. In other environments, the hosts are never connected to the internet and everything must be moved over sneaker net.

This guide will cover running the microk8s installer when only sneaker net is available as that is the most restrictive of these environments.

Preparing the media to sneaker net into the air gapped environment

You will need our airgapped dependencies .tar.gz file which you can get from Element:

Running the installer in the air gapped environment

Extract the airgapped dependencies to the airgapped directory at the root of the installer folder. You obtain the following directories :

Your airgapped machine will still require access to airgapped linux repositories depending on your OS.

Add the following parameters in your parameters.yml :

The installer will upload the images automatically to your local registry, and use these references to start the workloads.

When running the install script, add the parameter --airgapped so that it installs its pip and galaxy dependencies from the airgapped folder.

If you are using the kubernetes installer (instead of the single node installer), please note that once the image upload has been done, you will need to copy the airgapped/images/images_digests.yml file to the same path on the machine which will be used to render or deploy element services. Doing this, the new image digests will be used correctly in the kubernetes manifests used for deployment.

Setting Up Jitsi and TURN With the Installer

Configure the Installer to install Jitsi and TURN

Prerequisites

Firewall

You will have to open the following ports to your microk8s host to enable coturn and jitsi :

For jitsi :

For coturn, allow the following ports :

You will also have to allow the following port range, depending on the settings you define in coturn.yml (see below) :

DNS

The jitsi and coturn domain names must resolve to the VM access IP. You must not use host_aliases for these hosts to resolve to the private IP locally on your setup.

Coturn

Jitsi

Element

Restart the install script once everyting is set.

Configure the installer to use an existing Jitsi instance

{
      "jitsi": {
            "preferredDomain": "your.jitsi.example.org"
      }
}

replacing your.jitsi.example.org with the hostname of your Jitsi server.

Configure the installer to use an existing Coturn instance

Follow the instructions here: https://ems-docs.element.io/books/element-on-premise-documentation/page/single-node-installations#bkmrk-turn-server

Setting up Permalinks With the Installer

Element Extra Configurations

{
    "permalinkPrefix": "https://<element fqdn>"
}

Setting up Delegated Authentication With the Installer

On Element Enterprise

cp -r config-sample/synapse ~/.element-onpremise-config/synapse

On the provider

Here we cover Azure ADFS and Keycloak.

Other SAML providers can be configured for use with Element Enterprise. Please contact Element for further information in the event that you are not using one of the above providers.

Azure ADFS

Keycloak

Setting up Group Sync with the Installer

What is Group Sync?

Group Sync allows you to use the ACLs from your identity infrastructure in order to set up permissions on Spaces and Rooms in the Element Ecosystem. Please note that the initial version we are providing only supports a single node, non-federated configuration.

General settings

mkdir ~/.element-onpremise-config/groupsync
cp config-sample/groupsync/gsync.yml ~/.element-onpremise-config/groupsync/

Configuring the source

LDAP Servers

cp config-sample/groupsync/ldap.yml ~/.element-onpremise-config/groupsync/

edit the following variables :

MS Graph (Azure AD)

Space Mapping

The space mapping mechanism allows us to configure additional spaces that Group Sync will maintain, beyond the ones that it creates by default. It is optional – the configuration can be skipped if no additional spaces are to be created.

This is especially useful when used with bridges other than LDAP, which would normally not create any spaces other than the company-wide one. When used with the LDAP backend, the spaces created from LDAP OrgUnits will be added to the list of subspaces of the toplevel space.

Space mapping also replaces the group power level configuration and group filtering, being a superset of their functionality. It is recommended to use space mapping in their stead, as they might eventually be deprecated and removed.

Note: transitioning to space mapping

If you're using Group Sync already and want to transition to space mapping, make sure to match the root space ID with the one that Group Sync has already created by default -- otherwise it will create a brand new space and forget about the old one.

For LDAP, the default ID is the DN (distinguished name) of your main OrgUnit.

For MS Graph, the ID should be your tenant ID.

For SCIM, use scim:<client-id>, where the client-id is what you have defined in client.id in your configuration.

You can verify what the ID of the existing space is by running Group Sync in dry-run mode (-n1 launch parameter).

Configuration

We define each space giving it a name (which will be displayed in Element), a unique ID (which allows Group Sync to track the Space even if it gets renamed), and a list of groups whose users will become the members of the Space. Users needs to be a member of any configured group, not all of them.

You can pick any ID you want (taking note of the section above), but if you change it later Group Sync will create a brand new space and abandon the old ones, likely confusing the users.

Each group may optionally include a powerLevel setting, allowing specific groups to have elevated permissions in the space.

A special group ID of '' (an empty string) indicates that all users from the server, regardless of their group membership, should become the members of the Space.

An optional list of subspaces may also be configured, each using the same configuration format and behaviour (recursively).

The default Group Sync behaviour is equivalent to the following Space Mapping:

spaces:
  id: root
  name: 'Company'
  groups:
    - externalId: '' # include all users, not limited to any group

In order to limit space membership to a specific Group, we include its Group ID. This is equivalent to the group_filter configuration option.

spaces:
  id: root
  name: 'Company'
  groups:
    - externalId: 'element-users'

With powerLevel option allows us to give users extra permissions. This is equivalent to the group_power_level setting[^note].

[^note]: In the LDAP bridge group_power_level is the only way to assign permissions to spaces automatically generated from LDAP OrgUnits. If you define both space mapping and group_power_level in your configuration, group_power_level will only be used for the automatically generated spaces, it will not be taken into account for the spaces defined manually in your space mapping config.

spaces:
  id: root
  name: 'Company'
  groups:
    # regular users
    - externalId: 'element-users'
    # moderators
    - externalId: 'element-moderators'
      powerLevel: 50

In case of Power Level conflicts, the highest power level will be used. With the following configuration:

spaces:
  id: root
  name: 'Company'
  groups:
    - externalId: 'moderators'
      powerLevel: 50
    - externalId: 'admins'
      powerLevel: 100

A user who's a member of both moderators and admins will end up with Power Level of 100.

Subspaces can be configured analogically:

spaces:
  id: shared
  name: "Element Corp"
  groups:
  - externalId: 'matrix-mods'
    powerLevel: 50
  - externalId: ''
  subspaces:
  - id: london
    name: "London Office"
    groups:
    - externalId: 'london-matrix-mods'
      powerLevel: 50
    - externalId: 'london-employees'

Setting Up the Integration Manager With the Installer

Dimension is Deprecated

Starting with Element Enterprise Installer 2022-09.06, we are now shipping Integrator, our next generation integration manager. When installing 2022-09.06 or later, you should install Integrator. If you still have Dimension installed, please follow this step to remove it:

kubectl delete dimension/instance -n element-onprem

Now you will need to delete the two configuration files for dimension. Assuming your configuration directory is ~/.element-onpremise-config/, the commands would be:

rm -rf ~/.element-onpremise-config/dimension
rm ~/.element-onpremise-config/element/dimension.json

Once you have finished these steps you may continue with the directions to configure integrator.

If you are on a release prior to 2022-09.06 and need documentation on dimension, please see: Documentation Covering Installers From 2022.07.03 to 2022.09.05

On the hosting machine

Setting up GitLab, GitHub, and JIRA Integrations With the Installer

In Element Enterprise On-Premise, our GitLab, GitHub, and JIRA integrations are provided by the hookshot package. This documentation explains how to configure the installer to install hookshot and then how to interact with hookshot once installed.

Configuring Hookshot with the Installer

Enabling GitHub Integration

On GitHub

On the installation

In Element's room

Enabling GitLab integration

On GitLab

On the installation

In Element's room

Enabling JIRA integration

On JIRA

Enable OAuth

The JIRA service currently only supports atlassian.com (JIRA SaaS) when handling user authentication. Support for on-prem deployments is hoping to land soon.

On the installation

In Element's room

Enabling generic webhooks integration

On the installation

In Element's room

Setting up Adminbot and Auditbot

Overview

Starting with Installer version 2022.07-03, we have enabled the configuration of our Adminbot and Auditbot products, which are available as add-ons to our Enterprise customers.

Adminbot allows for an Element Administrator to become admin in any existing room or space on a managed homeserver. This enables you to delete rooms for which the room administrator has left your company and other useful administration actions.

Auditbot allows you to have the ability to export any communications in any room that the auditbot is a member of, even if encryption is in use. This is important in enabling you to handle compliance requirements that require chat histories be obtainable.

This document details how to configure the Adminbot and Auditbot themselves, but you will also need to install and configure our Enterprise Admin Dashboard so that an Element Administrator can log in and then log in as the Adminbot or Auditbot and perform specific functions.

Configuring Admin Bot

Start by copying config-sample/adminbot/adminbot.yml into your configuration directory, by running these commands from your installer directory:

mkdir ~/.element-onpremise-config/adminbot
cp config-sample/adminbot/adminbot.yml ~/.element-onpremise-config/adminbot/

The above assumes that ~/.element-onpremise-config is your configuration directory. Change it as necessary.

The config starts with these items:

bot_backup_phrase: # your secret storage backup phrase
bot_data_path: /mnt/data/adminbot
bot_data_size: 10M

enable_dm_admin: false

Let's discuss them:

Once this configuration is in place, you can re-run the installer and watch adminbot come up and then start joining rooms on your server. You may also choose to continue configuring audit bot and then the Enterprise Admin Dashboard prior to re-running the installer.

Configuring Audit Bot

Start by copying config-sample/auditbot/auditbot.yml into your configuration directory, by running these commands from your installer directory:

mkdir ~/.element-onpremise-config/auditbot
cp config-sample/auditbot/auditbot.yml ~/.element-onpremise-config/auditbot/

The above assumes that ~/.element-onpremise-config is your configuration directory. Change it as necessary.

The config starts with these items:

bot_backup_phrase: # your secret storage backup phrase
bot_data_path: /mnt/data/auditbot
bot_data_size: 10M

enable_dm_audit: false

### optional :the S3 bucket where to store the audit logs
#s3_bucket:
#s3_access_key_id:
#s3_secret_access_key:
#s3_key_prefix:
#s3_region:
#s3_endpoint:

### optional : the local logfile settings. Used if s3 bucket is not enabled.
logfile_size: 1M
logfile_keep: 3

Let's discuss them:

Once this configuration is in place, you can re-run the installer and watch auditbot come up and then start joining rooms on your server. You may also choose to continue configuring the Enterprise Admin Dashboard prior to re-running the installer.

Adminbot Federation

On the central admin bot server

On the remote admin bot server

Auditbot Federation

On the central auditbot server

On the remote audit bot server

Enterprise Admin Dashboard

Please see this document on Configuring the Enterprise Admin Dashboard

Configuring the Enterprise Admin Dashboard

Overview

Our Enterprise Admin Dashboard gives you the ability to manage users, rooms, the Adminbot, and the Auditbot. In the future, we will be expanding the functionality of this dashboard.

Configuring the Admin Dashboard

Start by copying config-sample/synapseadminui/synapseadminui.yml into your configuration directory, by running these commands from your installer directory:

mkdir ~/.element-onpremise-config/synapseadminui
cp config-sample/synapseadminui/synapseadminui.yml ~/.element-onpremise-config/synapseadminui/

The above assumes that ~/.element-onpremise-config is your configuration directory. Change it as necessary.

The config has these items:

synapseadmin_fqdn: <admin fqdn>
admin_elementweb_fqdn: <special elementwen web admin fqdn>

Let's discuss them:

For each of these FQDNs, you will need to make sure that a PEM encoded .crt and .key pair are in the certs directory of the configuration directory.

If you are not using delegated authentication, you will also need to set one more variable in your secrets.yml in the configuration directory and that is:

adminuser_password: <password>

Replacing <password> with the actual password that you want to use to be able to login to the Admin Dashboard with the

onprem-admin-donotdelete user.

If you are using delegated authentication, then you will need to give synapse admin privileges to one of your users. Let's say that your user who needs to have admin is named bob@local. To give this user

kubectl exec -n element-onprem -it pods/postgres-0 -- /usr/bin/psql -d synapse -U synapse_user -c "update users set admin = 1 where name = '@bob:server.name';"

Once you have done this, re-run the installer and after the pods have come up, you will be able to access the Enterprise Admin Dashboard at the provided FQDN.

Setting Up Chatterbox

What is Chatterbox?

Chatterbox allows for the embedding of a light-weight matrix-based chat client into any standard website. Chatterbox can be configured in two main modes:

How to set up Chatterbox

Copy config-sample/chatterbox/chatterbox.yml.sample into a file called chatterbox/chatterbox.yml in your configuration directory.

Create a certificate for the fqdn of chatterbox (chatterbox.example) and add that PEM based .crt/.key pair to your certs/ directory.

Edit the values of chatterbox/chatterbox.yml and set the following:

Deploying Chatterbox to Your Website

On the website that you'd like chatterbox set up on, add the following code:

    <script>
        window.CHATTERBOX_CONFIG_LOCATION = "https://<chatterbox_fqdn>/chatterbox-webconfig/config.json";
    </script>
    <script src="https://<chatterbox_fqdn>/assets/parent.js" type="module" id="chatterbox-script"></script>
    </body>

replacing <chatterbox_fqdn> with the value specified in the config file.

Setting up On-Premise Metrics

Setting up prometheus and grafana (Starting from installer 2022-08.02)

After running the installer, open the FQDN of Grafana. The initial login user is admin and password is admin. You'll be required to set a new password, please define one secured and keep it in a safe place.

Setting Up the Telegram Bridge

Configuring Telegram bridge

On Telegram platform

Basic config

Usage

Setting Up the Teams Bridge

Configuring Teams Bridge

Register with Microsoft Azure

You will first need to generate an "Application" to serve connect your Teams bridge with Microsoft.

Permissions

You will need to set some API permissions.

For each of the list below click Add permission > Microsoft Graph > and then set the Delegated permissions.

For each of the list below click Add permission > Microsoft Graph > and then set the Application permissions:

Once you are done, click Grant admin consent

Setting up the bot user

The bridge requires a Teams user to be registered as a "bot" to send messages on behalf of Matrix users. You just need to allocate one user from the Teams interface to do this.

Getting the groupId

The groupId can be found by opening Teams, clicking ... on a team, and clicking "Get link to team". The groupId is included in the URL 12345678-abcd-efgh-ijkl-lmnopqrstuvw in this example.

https://teams.microsoft.com/l/team/19%3XXX%40thread.tacv2/conversations?groupId=12345678-abcd-efgh-ijkl-lmnopqrstuvw&tenantId=87654321-dcba-hgfe-lkji-zyxwvutsrqpo

On the hosting machine

Generate teams registration keys

openssl genrsa -out teams.key 1024
openssl req -new -x509 -key teams.key -out teams.crt -days 365

Configure Teams Bridge

teams_client_id: # teams app client id
teams_client_secret: # teams app secret
teams_tenant_id: # teams app tenant id
teams_bot_username: # teams bot username
teams_bot_password: # teams bot password
teams_cert_file: teams.crt
teams_cert_private: teams.key
teams_fqdn: <teams bridge fqdn>
teams_bridged_groups:
- group_id: 218b0bfe-05d3-4a63-8323-846d189f1dc1 #change me
  properties:
    autoCreateRooms:
      public: true
      powerLevelContent:
        users:
          "@alice:example.com": 100 # This will add <alice> account as admin
          "@teams-bot:example.com": 100 # the Teams bot mxid <bot_sender_localpart>:<domain_name>
    autoCreateSpace: true
    limits:
      maxChannels: 25
      maxTeamsUsers: 25
# repeat "- group_id:" section above for each Team you want to bridge

     
bot_display_name: Teams Bridge Bot
bot_sender_localpart: teams-bot
enable_welcome_room: true
welcome_room_text: |
 Welcome, your Element host is configured to bridge to a Teams instance.

 This means that Microsoft Teams messages will appear on your Element
 account and you can send messages in Element rooms to have them appear
 on teams.

 To allow Element to access your Teams account, please say `login` and
 follow the steps to get connected. Once you are connected, you can open
 the 🧭 Explore Rooms dialog to find your Teams rooms.
# namespaces_prefix_user: OPTIONAL: default to _teams_
# namespaces_prefix_aliases: OPTIONAL: default to teams_

Setting Up the IRC Bridge

Overview

The IRC bridge allows you to bridge IRC servers into your Element server.

To configure the irc bridge, beging by copying config-sample/ircbridge/bridge.yml to CONFIG_DIR/ircbridge/bridge.yml. Then edit the file and set the following settings:

Next, we have the provisioning rules section, which will make sure that rooms are not bridged if a match is made on these rules. This is useful for preventing bad actors on Matrix from flooding IRC. This section looks like:

provisioning_rules:
 # The bridge checks the joined members of a propective room and checks to see
 # if any users matching these regex sets are in the room. `exempt` users never
 # match, and will be ignored. If any user matches `conflict`, the room will not
 # be allowed to be bridged until the user is removed. Both sets take a regular expression.
 userIds:
   exempt:
     # These users never conflict, even if matching
     - "@doubleagent:badguys.com"
   conflict:
     # These users will deny a room from being bridged.
     - "@.*:badguys.com"
provisioning_room_limit: 50

For other settings that can also be applied to this config file, please see:https://github.com/matrix-org/matrix-appservice-irc/blob/develop/config.sample.yaml#L52

Setting Up the SIP Bridge

Configuring SIP bridge

Basic config

Setting Up the XMPP Bridge

Configuring the XMPP Bridge

The XMPP bridge relies on the xmpp "component" feature. It is an equivalent of matrix application services. You need to configure an XMPP Component on an XMPP Server that the bridge will use to bridge matrix and xmpp user.

On the hosting machine

Prosody Example

If you are configuring prosody, you need the following component configuration (for the sample xmpp server, xmpp.lab.element.com):

    Component "matrix.xmpp.lab.element.com"
        ssl = {
          certificate = "/etc/prosody/certs/tls.crt";
          key = "/etc/prosody/certs/tls.key";
        }
      component_secret = "eeb8choosaim3oothaeGh0aequiop4ji"

And then with that configured, you would pass the following into xmpp.yml:

xmpp_service: xmpp://xmpp.lab.element.com:5347
xmpp_domain: matrix.xmpp.lab.element.com
xmpp_component_password: eeb8choosaim3oothaeGh0aequiop4ji # xmpp component password

Note: We've used pwgen 32 1 to generate the component_secret.

Joining an XMPP Room

Once you have the XMPP bridge up, you need to map an XMPP room to a Matrix ID. To do this, if the room on XMPP is named:

#iwotevo@conference.xmpp.lab.element.com

(conference is the fqdn of the component's hosting rooms on our xmpp test instance)

then on Matrix, you would join:

#_xmpp_iwotevo_conference.xmpp.lab.element.com:<your server name>

The command to do that from within the Element client would be: (assuming your homeserver domain is example.com)

/join #_xmpp_iwotevo_conference.xmpp.lab.element.com:example.com

Single Node Installs: Storage and Backup Guidelines

General storage recommentations for single-node instances

Adminbot storage:

Auditbot storage:

Chatterbox storage:

Dimension storage :

Synapse storage:

Postgres (in-cluster) storage:

Backup Guidance:

On-Premise Support Scope of Coverage

For Element Enterprise On-Premise, we support the following:

For Element On-Premise, we support the following:

The following items are not included in support coverage:

For single node setups, the following also applies:

For kubernetes deployments, the following also applies:

Troubleshooting

Introduction to Troubleshooting

Troubleshooting the Element Installer comes down to knowing a little bit about kubernetes and how to check the status of the various resources. This guide will walk you through some of the initial steps that you'll want to take when things are going wrong.

install.sh problems

Sometimes there will be problems when running the ansible-playbook portion of the installer. When this happens, you can increase the verbosity of ansible logging by editing .ansible.rc in the installer directory and setting:

export ANSIBLE_DEBUG=true
export ANSIBLE_VERBOSITY=4

and re-running the installer. This will generate quite verbose output, but that typically will help pinpoint what the actual problem with the installer is.

Problems post-installation

Checking Pod Status and Getting Logs

Other Commands of Interest

Some other commands that may yield some interesting data while troubleshooting are:

Archived Documentation Repository

Archived Documentation Repository

Documentation Covering Installers From 2022.07.03 to 2022.09.05

element-on-premise-documentation-0703-0905.pdf

Archived Documentation Repository

Documentation covering v1 and installers prior to 2022-07.03

element-on-premise-documentation-july28-2022.pdf