Using the PoC Installer in an Air-Gapped Environment

Defining Air-Gapped Environments

An air-gapped environment is any environment in which the running hosts will not have access to the greater internet. This proposes a situation in which these hosts are unable to get access to various needed bits of software from Element and also are unable to share telemetry data back with Element.

For some of these environments, they can be connected to the internet from time to time and updated during those connection periods. In other environments, the hosts are never connected to the internet and everything must be moved over sneaker net.

This guide will cover running the microk8s installer when only sneaker net is available as that is the most restrictive of these environments.

Preparing the media to sneaker net into the air gapped environment

First, you will need to get the latest On-Premise installer. We will unpack that into an installation directory and then you will create a configuration directory like this:

tar xvzf /path/to/installer.tar.gz
mkdir -p ~/.element-onprem-config/airgapped/images

Now, let's set up a minimal parameters.yml and secrets.yml. Run these commands in the installer directory:

sh build_secrets.sh
mv secrets.yml ~/.element-onprem-config/
cp config-sample/parameters.yml.sample ~/.element-onprem-config/paramters.yml

Edit ~/.element-onprem-config/parameters.yml and set the following two parameters:

images_dir: ~/.element-onprem-config/airgapped/images
local_registry: 127.0.0.1:32000

Start by downloading the container images in the installer directory:

bash download_images.sh ~/.element-onprem-config/ -o ~/.element-onprem-config/airgapped/images

Now, in the installer directory, go get the pip artifacts:

mkdir ~/.element-onprem-config/airgapped/pip
pip3 download -r requirements.txt -d ~/.element-onprem-config/airgapped/pip
pip3 download wheel setuptools_scm pyyaml MarkupSafe cryptography simplejson -d ~/.element-onprem-config/airgapped/pip

Now, we need to get the necessary ansible-galaxy collections:

mkdir ~/.element-onprem-config/airgapped/galaxy
pip3 install ansible --upgrade --user
~/.local/bin/ansible-galaxy collection download -r requirements.yml -p ~/.element-onprem-config/airgapped/galaxy

Now, let's download the requiste items from Snapcraft:

mkdir ~/.element-onprem-config/airgapped/snaps
cd ~/.element-onprem-config/airgapped/snaps
snap download microk8s --channel=1.21
snap download core18
snap download snapd

Now, let's grab helm3:

cd ~/.element-onprem-config/airgapped
wget https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz
tar xvzf helm-v3.5.0-linux-amd64.tar.gz

Now let's tar all of this up:

cd ~
tar cvzf installer-sneaker-net.tar.gz ./.element-onprem-config ./<path to installer>

And now we take installer-sneaker-net.tar.gz into our air-gapped environment.

Running the installer in the air gapped environment

Make sure that the machine is set up per the instructions in our PoC Installer Guide up through the Unpacking the Installer section.

Sneaker net over your tarball and unpack it in your user's home directory.

Patch install.sh, changing this line:

pip3 install -r requirements.txt --upgrade --user

to:

pip3 install -r requirements.txt --upgrade --user --no-index --find-links ~/.element-onprem-config/airgapped/pip

and this line:

ansible-galaxy collection install -r requirements.yml

to:

cd ~/.element-onprem-config/airgapped/galaxy && ansible-galaxy collection install -r requirements.yml && cd -

Make sure you finish setting up parameters.yml and secrets.yml per the PoC Installer Guide as well.

Now we need to install microk8s. To do that, we need to finish configuring snapd:

sudo systemctl start snapd
sudo systemctl enable snapd
cd ~/.element-onprem-config/airgapped/snaps
sudo snap ack core18_2409.assert
sudo snap install core18_2409.snap
sudo snap ack snapd_15904.assert
sudo snap install snapd_15904.snap
sudo snap ack microk8s_3202.assert
sudo ln -sf /var/lib/snapd/snap /snap
sudo snap install microk8s_3202.snap --classic
sudo usermod -a -G microk8s karl1
sudo chown -f -R karl1 ~/.kube
sudo mkdir /var/snap/microk8s/current/bin/
sudo cp ~/.element-onprem-config/airgapped/linux-amd64/helm /var/snap/microk8s/current/bin/helm3
sudo systemctl reboot

You should now be ready to run the installer, which is going to fail at a few key points:

calico-node tends to hang and needs to be killed for the installer to proceed -- this holds up port 9099.
The registry doesn't actually start and this causes port 32000 to not be accessible. This happens due to kube-system calico-kube-controllers-f7868dd95-864wz 0/1 ErrImagePull 0 16m

not being available. We are missing all of these images : (re-enabling the internet for a brief moment allowed all these images to pull)

kube-system          hostpath-provisioner-566686b959-bpj9l        1/1     Running   0          4m21s
kube-system          dashboard-metrics-scraper-78d7698477-jf2wc   1/1     Running   0          4m21s
kube-system          coredns-7f9c69c78c-c2rb4                     1/1     Running   0          4m21s
kube-system          metrics-server-8bbfb4bdb-8twg6               1/1     Running   0          4m21s
ingress              nginx-ingress-microk8s-controller-nt9c7      1/1     Running   0          4m21s
container-registry   registry-9b57d9df8-7mr5j                     1/1     Running   0          4m21s
kube-system          calico-kube-controllers-f7868dd95-9fsdc      1/1     Running   0          32s
kube-system          kubernetes-dashboard-85fd7f45cb-bsc8r        1/1     Running   0          19s

bits and bobs and random notes

wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.3/mkcert-v1.4.3-linux-amd64

EPEL via satellite on RHEL. Ubuntu via some local repo as well.

EPEL is needed to install pwgen, snapd, snap-confine, and snapd-selinux in RHEL.

If you install snapd manually, you need to then start snapd (and enable snapd) or the installer will puke.

ansible and pip installed artifacts

~~On your system that is connected to the internet, download the appropriate pip packages for offline use in the installer directory:~~

mkdir pip
mkdir galaxy
pip download -r requirements.txt -d ./pip
pip download wheel -d ./pip
pip download setuptools_scm -d ./pip
ansible-galaxy collection download -r requirements.yml -p galaxy

~~Need to then modify this line in install.sh to force it to use the local pip repository:~~

pip3 install -r requirements.txt --upgrade --user

~~with:~~

pip3 install -r requirements.txt --upgrade --user --no-index --find-links ./pip

~~Also modify:~~

ansible-galaxy collection install -r requirements.yml

~~to read:~~

cd galaxy && ansible-galaxy collection install -r requirements.yml && cd -

~~pip install --no-index --find-links /path/to/download/dir/ -r requirements.txt~~

microk8s and helm3

~~On a machine connected to the internet, you will need to download the appropriate snaps for microk8s. Our installer presently uses the 1.21 version of microk8s from the stable branch.~~

snap download microk8s --channel=1.21
snap download core18
snap download snapd

~~This will download 6 files -- a~~ .assert ~~and a~~ .snap ~~for each requested package. In my case, I have the following files:~~

core18_2409.assert  
core18_2409.snap  
microk8s_3202.assert  
microk8s_3202.snap
snapd_15904.assert
snapd_15904.snap

~~Your list will be similar, but the revision numbers will possibly be different.~~

Now, we need to go grab the helm3 binary. As this is a dependency of microk8s, we need to actually check which version our microk8s will need. As such, you will need to have the microk8s that you downloaded installed for this to work.

~~If you have not previously installed microk8s, please do the following:~~

sudo snap ack core18_2409.assert
sudo snap install core18_2409.snap
sudo snap ack snapd_15904.assert
sudo snap install snapd_15904.snap
sudo snap ack microk8s_3202.assert
sudo snap install microk8s_3202.snap --classic
sudo usermod -a -G microk8s karl1
sudo chown -f -R karl1 ~/.kube
newgrp microk8s
REBOOT

cat /var/lib/snapd/snap/microk8s/current/actions/enable.helm3.sh | awk -F 'HELM_VERSION=' '{print $2}' - | xargs | sort -b | uniq

In my case, this yields:

v3.5.0

With this knowledge, we can now construct a wget command to grab the proper helm3 binary:

wget https://get.helm.sh/helm-<VERSION>-linux-amd64.tar.gz

which with our output from above would be:

wget https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz

Now we need to unpack this tarball:

tar xvzf helm-v3.5.0-linux-amd64.tar.gz

and now we will find the proper helm binary for microk8s in the linux-amd64 directory.

Now, we need to take the snap artifacts and the helm binary to our disconnected environment.

Microk8s must be running and the installer will have failed to enable helm3 to do this:

sudo cp /path/to/helm /var/snap/microk8s/current/bin/helm3
microk8s.enable helm3

Once you have helm3 enabled, you can opt to re-run the installer as it will now get past the enabling of helm3.

Getting Element's Container Images into your air-gapped environment.

~~Workflow on connected host:~~

~~Get installer from Element. Run download_images.sh and point this at an images directory. Tar up the installer that download_images.sh has run on and the images directory.~~

~~Take this installer and the images directory to the air gapped environment and deploy.~~