Kubernetes Installations
Kubernetes Installations
Overview
Our Installer can handle the installation of Element Enterprise into your production kubernetes (k8s) environment.
Prerequisites
Before beginning the installation, there are a few things that must be prepared to ensure a successful deployment and functioning installation.
Kubectl environment
The installer uses your currently active kubectl context which can be determined with kubectl config current-context - make sure this is the correct context as all subsequent operations will be performed under this.
More information on configuring this can be found in the upstream kubectl docs
Be sure to export K8S_AUTH_CONTEXT=<kube context name> for the Installer if you need to use a context aside from your currently active one.
PostgreSQL
Before you can begin with the installation you must have a PostgreSQL database instance available. The installer does not manage databases itself.
The database you use must be set to a locale of C and use UTF8 encoding - see https://matrix-org.github.io/synapse/latest/postgres.html#set-up-database for further details as they relate to Synapse. If the locale / encoding are incorrect, Synapse will fail to initialize the database and get stuck in a CrashLoopBackoff cycle.
Please make note of the database hostname, database name, user, and password as you will need these to begin the installation.
For testing and evaluation purposes, you can deploy PostgreSQL to k8s before you begin the installation process - see Kubernetes Installations - Quick Start - Deploying PostgreSQL to Kubernetes for more information.
Kubernetes Ingress Controller
The installer does not manage cluster Ingress capabilities since this is typically a cluster-wide concern - You must have this available prior to installation. Without a working Ingress Controller you will be unable to route traffic to your services without manual configuration.
If you do not have an Ingress Controller deployed please see Kubernetes Installations - Quick Start - Deploying ingress-nginx to Kubernetes for information on how to set up a bare-bones ingress-nginx installation to your cluster.
Use an existing Ingress Controller
If you have an Ingress Controller deployed already and it is set to the default class for the cluster, you shouldn't have to do anything else.
If you're unsure you can see which providers are available in your cluster with the following command:
$ kubectl get IngressClass
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 40d
And you can check to see whether an IngressClass is set to default using kubectl, for example:
$ kubectl describe IngressClass nginx
Name: nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.1.1
argocd.argoproj.io/instance=ingress-nginx
helm.sh/chart=ingress-nginx-4.0.17
Annotations: ingressclass.kubernetes.io/is-default-class: true
Controller: k8s.io/ingress-nginx
Events: <none>
In this example cluster there is only an nginx IngressClass and it is already default, but depending on the cluster you are deploying to this may be something you must manually set.
Migrating from our older installer
If you have previously used installer versions 2023-03.01 and earlier, you will need to run our migration script to convert your previous configuration to the new format that is used with our UI based installer. This script became available in 2023-03.02, so you must have at least that version or higher of the graphical installer for this to work.
NOTE: Before running the migration script, we highly recommend that you take a backup or snapshot of your working environment. While we have tested the migration script against several configurations at this point, we have not tested for all of the combinations of configuration that the previous installer allowed. We expect that migration will be a quick process for most customers, but in the event that something goes wrong, you'll want to be able to get back to a known good state through a backup or snapshot.
NB: If you are using group sync, you cannot presently migrate to the graphical installer. We are working to address the issues with migrating group sync and will remove this note once we have those addressed.
If you have not used our installer before, you may safely ignore this section.
To run the migration script, please do the following:
chmod +x ./element-enterprise-graphical-installer-YYYY-MM.VERSION-gui.bin
./element-enterprise-graphical-installer-YYYY-MM.VERSION-gui.bin --import ~/.element-onpremise-config
Make sure to replace ~/.element-onpremise-config with the path that your actual configuration exists in. Further, replace YYYY-MM.VERSION with the appropriate tag for the installer you downloaded.
Once the import has finished, the GUI will start and you will be able to browse to the installer at one of the provided URLs, much as if you had started the installer without doing a migration as detailed in the following section.
Beginning the Installation
Head to https://ems.element.io/on-premise/download and download the latest installer. The installer will be called element-enterprise-graphical-installer-YYYY-MM.VERSION-gui.bin. You will take this file and copy it to the machine where you will be installing the Element Server Suite. Once you have this file on the machine in a directory accessible to your sudo-enabled user, you will run:
chmod +x ./element-enterprise-graphical-installer-YYYY-MM.VERSION-gui.bin
replacing the YYYY-MM.VERSION with the appropriate tag for the installer you downloaded.
If you have multiple kubernetes clusters configured in your kubeconfig, you will have to export the K8S_AUTH_CONTEXT variable before running the installer :
export K8S_AUTH_CONTEXT=<kube context name>
Once you have done this, you will run:
./element-enterprise-graphical-installer-YYYY-MM.VERSION-gui.bin
replacing the YYYY-MM.VERSION with the appropriate tag for the installer you downloaded, and this will start a web server with the installer loaded.
You will see a message similar to:
[user@element-demo ~]$ ./element-enterprise-graphical-installer-2023-02.02-gui.bin
Testing network...
Using self-signed certificate with SHA-256 fingerprint:
F3:76:B3:2E:1B:B3:D2:20:3C:CD:D0:72:A3:5E:EC:4F:BC:3E:F5:71:37:0B:D7:68:36:2E:2C:AA:7A:F2:83:94
To start configuration open:
https://192.168.122.47:8443 or https://10.1.185.64:8443 or https://127.0.0.1:8443
At this point, you will need to open a web browser and browse to one of these IPs. You may need to open port 8443 in your firewall to be able to access this address from a different machine.
If you are unable to open port 8443 or you are having difficulty connecting from a different machine, you may want to try ssh port forwarding in which you would run:
ssh <host> -L 8443:127.0.0.1:8443
replacing host with the IP address or hostname of the machine that is running the installer. At this point, with ssh connected in this manner, you should be able to use the https://127.0.0.1:8443 link as this will then forward that request to the installer box via ssh.
Upon loading this address for the first time, you may be greeted with a message informing you that your connection isn't private such as this:
In this case, you'll need to click "Advanced" and then "Continue to
The Hosts Screen
The very first page that you come to is the host screen.
You will want to make sure that "Kubernetes Application" is selected. You can opt to skip the update setup or the operator setup, but unless you know why you are doing that, you should leave them alone.
The very next prompt that you come to is for an EMS Image Store Username and Token. These are provided to you by element as access tokens for our enterprise container registries. If you have lost your token, you can always generate a new token at https://ems.element.io/on-premise/subscriptions.
Here, we find the ability to set the namespaces that the application will be deployed into.
The final options on the hostpage are related to connectivity. For this guide, we are assuming "Connected" and you can leave that be. If you are doing "Airgapped", you would pick airgapped at this point and then please see the section on airgapped installations.
You are presented with the option to provide docker hub credentials. These are optional, but if you do not provide them, you may be rate limited by Docker and this could cause issues pulling container images.
The Domains Screen
On this page, we get to specify the domains for our installation. In this example, we have a domain name of airgap.local and this would mean our MXIDs would look like @kabbott:airgap.local.
Our domain page has checking to ensure that the host names resolve. Once you get green checks across the board, you can click continue.
The Certificates Screen
On the Certificates screen, you will provide SSL certificate information for well-known delegation, Synapse, Element Web, Synapse Admin, and Integrator.
If you are using Let's Encrypt, then each of the sections should look like:
If you are using certificate files, then you will see a screen like:
which allows you to upload a .crt and .key file for each host. These files must be in PEM encoding. Our installer does accept wildcard certificates.
Once you have completed the certificate section for each host on the page, you may click continue.
The Database Screen
As you must use an external PostgreSQL database with our kubernetes install, on this page, we provide the option to specify the database name, the database host name, the port to connect to, the SSL mode to use, and finally, the username and password to connect with. Once you have completed this section, you may click continue.
The Media Screen
On this page, you can specify the size of your synapse media volume. Please leave "Create New Volume" checked and specify the size of the volume that you wish to allocate. You must have this space available in /data/element-deployment or whatever you specified back on the hosts screen. If you wish to create a 50G volume, you would need to specify 50Gi for the Volume size.
The Cluster Screen
Most deployments can ignore this, however, if you want to change any kubernetes cluster parameters, this is where to do it.
If you are in an environment where you have self-signed certificates, you will want to disable TLS verification, by clicking "Advanced" and then scrolling down and unchecking Verify TLS:
Please bear in mind that disabling TLS verification and using self-signed certificates is not recommended for production deployments.
If your host names are not DNS resolvable, you need to use host aliases and this can be set up here. You will also click "Advanced" and scroll down to the "Host Aliases" section in "k8s". In here, you will click "Add Host Aliases" and then you will specify an IP and host names that resolve to that IP as such:
Important: If you are not using OpenShift, you will need to set "Force UID GID" and "Set Sec Comp" to "Enable" under the section "Security Context" so that it looks like:
If you are using OpenShift, you should leave the values of "Force UID GID" and "Set Sec Comp" set to "Auto".
When you are finished with this page, you can click continue.
The Synapse Screen
The first setting that you will see is whether you want to auto accept invites. The default of "Manual" will fit most use cases, but you are welcome to change this value.
The next setting is the maximum number of monthly active users (MAU) that you have purchased for your server. Your server will not allow you to go past this value. If you set this higher than your purchased MAU and you go over your purchased MAU, you will need to true up with Element to cover the cost of the unpaid users.
The next setting concerns registration. A server with open registration on the open internet can become a target, so we default to closed registration. You will notice that there is a setting called "Custom" and this requires explicit custom settings in the additional configuration section. Unless instructed by Element, you will not need the "Custom" option and should instead pick "Closed" or "Open" depending on your needs.
After this, you will see that the installer has picked an admin password for you. You will want to use the eye icon to view the password and copy this down as you will use this with the user onprem-admin-donotdelete to log into the admin panel after installation.
Continuing, we see telemetry. You should leave this enabled as you are required to report MAU to Element. In the event that you are installing into an enviroment without internet access, you may disable this so that it does not continue to try talking to Element. That said, you are still required to generate an MAU report at regular intervals and share that with Element.
For more information on the data that Element collects, please see: What Telemetry Data is Collected by Element?
Next, we have two advanced buttons. The top one is for synapse and gives you a text box to inject additional synapse configs (homeserver.yaml), access to the macaroon, registration shared secret, and signing key. Further, you can add and configure multiple synapse workers, external appservices, and federation. These topics are presently beyond the scope of this install guide, but we will write them up in due time. Further, you also have the ability to set a STUN shared secret and set TURN URIs for any existing TURN servers that you may have in your environment.
The second advanced button allows you to explicitly set any kubernetes cluster settings that you would like just for the synapse pods. Most users will be able to ignore this.
You can hit continue to go to the next screen.
The Element Web Screen
Most users will be able to simply click "Continue" here.
The Advanced section allows you to set any custom element web configurations you would like (config.json).
A common custom configuration would be configuring permalinks for Element, which we have documented here: Setting up Permalinks With the Installer
Further, it provides access to the k8s section, allowing you to explicitly set any kubernetes cluster settings that you would like just for the element-web pod.
The Enterprise Admin Dashboard
Most users will be able to simply click "Continue" here. The Advanced section allows you to explicitly set any kubernetes cluster settings that you would like just for the synapse-admin-ui pod.
One word to note here is that if you are not using delegated authentication, then the initial username that an administrator will use to log into this dashboard post-installation is onprem-admin-donotdelete. You can find the password for this user on the Synapse page in the "Admin Password" field.
If you are using delegated authentication, you will need to assign a user admin rights as detailed in this article: How do I give a user admin rights when I am using delegated authentication and cannot log into the admin console?
The Integrator Screen
On this page, you can set up Integrator, the integrations manager.
The first option allows you to choose whether users can add custom widgets to their rooms with the integrator or not.
The next option allows you to specify which Jitsi instance the Jitsi widget will create conferences on.
The verify TLS option allows you to set this specifically for Integrator, regardless of what you set on the cluster screen.
The logging section allows you to set the log level and whether the output should be structured or not.
The Advanced section allows you to explicitly set any kubernetes cluster settings that you would like just for the integrator pods.
Click "Continue to go to the next screen".
The Integrations Screen
This screen is where you can install any available integrations.
Some of these integrations will have "YAML" next to them. When you see this designation, this integration requires making settings in YAML, much like the old installer. However, with this installer, these YAML files are pre-populated and often only involve a few changes.
If you do not see a "YAML" designation next to the integration then this means that will use regular GUI elements to configure this integration.
Over time, we will do the work required to move the integrations with "YAML" next to them to the new GUI format.
For specifics on configuring well known delegation, please see Setting Up Well Known Delegation
For specifics on setting up Delegated Authentication, please see Setting up Delegated Authentication With the Installer
For specifics on setting up Group Sync, please see Setting up Group Sync with the Installer
For specifics on setting up GitLab, GitHub, and JIRA integrations, please see Setting up GitLab, GitHub, and JIRA Integrations With the Installer
For specifics on setting up Adminbot and Auditbot, please see: Setting up Adminbot and Auditbot
For specifics on setting up Hydrogen, please see: Setting Up Hydrogen
For specifics on pointing your installation at an existing Jitsi instance, please see Setting Up Jitsi and TURN With the Installer
If you do not have an existing TURN server or Jitsi server, our installer can configure these for you by following the extra steps in Setting Up Jitsi and TURN With the Installer
For specifics on configuring the Teams Bridge, please see Setting Up the Teams Bridge
For specifics on configuring the Telegram Bridge, please see Setting Up the Telegram Bridge
For specifics on configuring the IRC Bridge, please see Setting Up the IRC Bridge
For specifics on configuring the XMPP Bridge, please see Setting Up the XMPP Bridge
Once you have configured all of the integrations that you would like to configure, you can click "Continue" to head to the installation screen.
The Installation Screen
On the installation screen, you should see a blank console and a start button:
Click Start.
After a moment, you will notice the installer hang. If you go back to the prompt where you are running the installer, you will see that you are being asked for the sudo password:
Go ahead and enter the sudo password and the installation will continue.
On the very first time that you run the installer, you will be prompted to log out and back in again to allow Linux group membership changes to be refreshed. This means that you will need to issue a ctrl-C in the terminal running your installer and actually log all the way out of your Linux session, log back in, restart the installer, navigate back to the installer screen, click start again, and then re-enter your sudo password. You will only have to perform this step once per server.
Verifying Your Installation
Once the installation has finished, it can take as much as 15 minutes on a first run for everything to be configured and set up. If you use:
kubectl get pods -n element-onprem
You should see similar output to:
NAME READY STATUS RESTARTS AGE
app-element-web-c5bd87777-rqr6s 1/1 Running 1 29m
server-well-known-8c6bd8447-wddtm 1/1 Running 1 29m
postgres-0 1/1 Running 1 40m
instance-synapse-main-0 1/1 Running 2 29m
instance-synapse-haproxy-5b4b55fc9c-hnlmp 1/1 Running 0 20m
Once the admin console is up and running:
first-element-deployment-synapse-admin-ui-564cbf5665-dn8nv 1/1 Running 1 (4h4m ago) 3d1h
and synapse:
first-element-deployment-synapse-redis-59548698df-gqkcq 1/1 Running 1 (4h4m ago) 3d2h
first-element-deployment-synapse-haproxy-7587dfd6f7-gp6wh 1/1 Running 2 (4h3m ago) 2d23h
first-element-deployment-synapse-appservice-0 1/1 Running 3 (4h3m ago) 3d
first-element-deployment-synapse-main-0 1/1 Running 0 3h19m
then you should be able to log in at your admin panel (in our case https://admin.airgap.local/) with the onprem-admin-donotdelete user and the password that was specified on the "Synapse" screen.
A word about Configuration Files
In the new installer, all configuration files are placed in the directory .element-enterprise-server. This can be found in your user's home directory. In this directory, you will find a subdirectory called config that contains the actual configurations.
Running the Installer without the GUI
It is possible to run the installer without using the GUI provided that you have a valid set of configuration files in the .element-enterprise-server/config directory. Directions on how to do this are available at: https://ems-docs.element.io/books/ems-knowledge-base/page/how-do-i-run-the-installer-without-using-the-gui. Using this method, you could use the GUI as a configuration editor and then take the resulting configuration and modify it as needed for further installations.
This method also makes it possible to set things up once and then run future updates without having to use the GUI.
End-User Documentation
After completing the installation you can share our User Guide to help orient and onboard your users to Element!