Using the PoC Installer in an Air-Gapped Environment

Defining Air-Gapped Environments

An air-gapped environment is any environment in which the running hosts will not have access to the greater internet. This proposes a situation in which these hosts are unable to get access to various needed bits of software from Element and also are unable to share telemetry data back with Element.

For some of these environments, they can be connected to the internet from time to time and updated during those connection periods. In other environments, the hosts are never connected to the internet and everything must be moved over sneaker net.

This guide will cover running the microk8s installer when only sneaker net is available as that is the most restrictive of these environments.

Preparing the media to sneaker net into the air gapped environment

First,Run youthe script download_images.sh to run download the images to a local directory. The script will needalso compute your images digest in a file named images_digests.yml. Copy the whole directory containing the images and the images_digests.yml file over the airgapped environment.

It is also required to get the latestairgap On-Premisedependencies. installer.They Weare willmade unpackof thattwo into an installation directory and then you will create a configuration directory like this:

tar xvzf /path/to/installer.tar.gz mkdirfiles -p ~/.element-onprem-config/airgapped/images

Now, let's set up a minimal parameters.yml and secrets.yml. Run these commands in the installer directory::

sh

build_secrets.sh
mv secrets.yml ~/
element-enterprise-poc-installer-airgapped-deps-<version>.element-onprem-config/
cp config-sample/parameters.yml.sample ~/.element-onprem-config/paramters.yml

Edit ~/.element-onprem-config/secrets.yml and set ems_image_store_username and ems_image_store_token. You should have this username/token pair provided to you by Element. If not, contact your Element representative.

Edit ~/.element-onprem-config/parameters.yml and set the following two parameters:

images_dir: ~/.element-onprem-config/airgapped/images
local_registry: 127.0.0.1:32000

Start by downloading the container images in the installer directory:

bash download_images.sh ~/.element-onprem-config/ -o ~/.element-onprem-config/airgapped/images

Now, in the installer directory, go get the pip artifacts:

mkdir ~/.element-onprem-config/airgapped/pip
pip3 download -r requirements.txt -d ~/.element-onprem-config/airgapped/pip
pip3 download wheel setuptools_scm pyyaml MarkupSafe cryptography simplejson -d ~/.element-onprem-config/airgapped/pip

Now, we need to get the necessary ansible-galaxy collections:

mkdir ~/.element-onprem-config/airgapped/galaxy
pip3 install ansible --upgrade --user
~/.local/bin/ansible-galaxy collection download -r requirements.yml -p ~/.element-onprem-config/airgapped/galaxy

Now, let's download the requiste items from Snapcraft:

mkdir ~/.element-onprem-config/airgapped/snaps
cd ~/.element-onprem-config/airgapped/snaps
snap download microk8s --channel=1.24
snap download core18
snap download snapd

Now, let's grab helm3:

cd ~/.element-onprem-config/airgapped
wget https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz
tar xvzf helm-v3.5.0-linux-amd64.
element-enterprise-poc-installer-airgapped-microk8-<version>.tar.gz

Now let's tar all of this up:

cd ~
tar cvzf installer-sneaker-net.tar.gz ./.element-onprem-config ./<path to installer>

And now we take installer-sneaker-net.tar.gz into our air-gapped environment.

Running the installer in the air gapped environment

Make sure thatExtract the machineairgapped isdependencies set up perto the instructionsairgapped indirectory our PoC Installer Guide up throughat the Unpackingroot of the Installerinstaller section.folder. You obtain the following directories :

• airgapped/pip
• airgapped/galaxy
• airgapped/snaps
• airgapped/containerd

You airgapped machine will still require access to airgapped linux repositories depending on your OS.

SneakerAdd netthe overfollowing your tarball and unpack itparameters in your user's home directory.

Patch install.sh, changing this line:

pip3 install -r requirements.txt --upgrade --user

to:

pip3 install -r requirements.txt --upgrade --user --no-index --find-links ~/.element-onprem-config/airgapped/pip

and this line:

ansible-galaxy collection install -r requirements.yml

to:

cd ~/.element-onprem-config/airgapped/galaxy && ansible-galaxy collection install -r requirements.yml && cd -

Make sure you finish setting up parameters.yml and:

secrets.yml

per
• local_registry: 127.0.0.1:32000
• images_dir: <absolute path to the PoCcopied Installerimage Guidedirectory> as
well.

Now we need to install microk8s. To do that, we need to finish configuring snapd:

sudo systemctl start snapd
sudo systemctl enable snapd
cd ~/.element-onprem-config/airgapped/snaps
sudo snap ack core18_2409.assert
sudo snap install core18_2409.snap
sudo snap ack snapd_15904.assert
sudo snap install snapd_15904.snap
sudo snap ack microk8s_3202.assert
sudo ln -sf /var/lib/snapd/snap /snap
sudo snap install microk8s_3202.snap --classic
sudo usermod -a -G microk8s karl1
sudo chown -f -R karl1 ~/.kube
sudo mkdir /var/snap/microk8s/current/bin/
sudo cp ~/.element-onprem-config/airgapped/linux-amd64/helm /var/snap/microk8s/current/bin/helm3
sudo systemctl reboot

You should now be ready to run the installer, which is going to fail at a few key points:

1. calico-node tends to hang and needs to be killed for the installer to proceed -- this holds up port 9099.
2. The registry doesn't actually start and this causes port 32000 to not be accessible. This happens due to kube-system calico-kube-controllers-f7868dd95-864wz 0/1 ErrImagePull 0 16m

not being available. We are missing all of these images : (re-enabling the internet for a brief moment allowed all these images to pull)

kube-system          hostpath-provisioner-566686b959-bpj9l        1/1     Running   0          4m21s
kube-system          dashboard-metrics-scraper-78d7698477-jf2wc   1/1     Running   0          4m21s
kube-system          coredns-7f9c69c78c-c2rb4                     1/1     Running   0          4m21s
kube-system          metrics-server-8bbfb4bdb-8twg6               1/1     Running   0          4m21s
ingress              nginx-ingress-microk8s-controller-nt9c7      1/1     Running   0          4m21s
container-registry   registry-9b57d9df8-7mr5j                     1/1     Running   0          4m21s
kube-system          calico-kube-controllers-f7868dd95-9fsdc      1/1     Running   0          32s
kube-system          kubernetes-dashboard-85fd7f45cb-bsc8r        1/1     Running   0          19s

bits and bobs and random notes

wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.3/mkcert-v1.4.3-linux-amd64

EPEL via satellite on RHEL. Ubuntu via some local repo as well.

EPEL is needed to install pwgen, snapd, snap-confine, and snapd-selinux in RHEL.

If you install snapd manually, you need to then start snapd (and enable snapd) or the installer will puke.

upload

catthe /var/lib/snapd/snap/microk8s/current/actions/enable.helm3.shimages | awk -F 'HELM_VERSION=' '{print $2}' - | xargs | sort -b | uniq 

In my case, this yields:

v3.5.0

With this knowledge, we can now construct a wget commandautomatically to grabyour local registry, and use these references to start the proper helm3 binary:workloads.

When running the install script, add the parameter wget--airgapped https://get.helm.sh/helm-<VERSION>-linux-amd64.tar.gzso 

that

whichit withinstalls ourits outputpip and galaxy dependencies from abovethe wouldairgapped be:folder.

wget https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz

Now we need to unpack this tarball:

tar xvzf helm-v3.5.0-linux-amd64.tar.gz

and now we will find the proper helm binary for microk8s in the linux-amd64 directory.

Now, we need to take the snap artifacts and the helm binary to our disconnected environment.

Microk8s must be running and the installer will have failed to enable helm3 to do this:

sudo cp /path/to/helm /var/snap/microk8s/current/bin/helm3
microk8s.enable helm3

Once you have helm3 enabled, you can opt to re-run the installer as it will now get past the enabling of helm3.

ContainerD Registry export for non-Element bits: (on connected host)

 for i in $(microk8s.ctr images list -q | grep -v sha256); do tarfile=$(echo $i | awk -F '/' '{print $NF}' -); microk8s.ctr images export $tarfile.tar $i && echo "microk8s.ctr images import --base-name $i $tarfile.tar" >> import-script; done

This creates import-script which can be run by moving the script and tar files to disconnected box and running:

chmod +x ./import-script
./import-script 
unpacking docker.io/calico/cni:v3.19.1 (sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4)...done
unpacking docker.io/calico/kube-controllers:v3.17.3 (sha256:072e5f661f9c1af9ab49d127743fdbfe5d8ba16bac7c2616b92e64dd304ec3c4)...done
unpacking docker.io/calico/node:v3.19.1 (sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55)...done
unpacking docker.io/calico/pod2daemon-flexvol:v3.19.1 (sha256:bcac7dc4f1301b062d91a177a52d13716907636975c44131fb8350e7f851c944)...done
unpacking docker.io/cdkbot/hostpath-provisioner:1.1.0 (sha256:317ed1b7324dc5f20db5dfdc0647a26a1ac44df426e9caad9cfdd6ba84c47697)...done
unpacking docker.io/coredns/coredns:1.8.0 (sha256:cc8fb77bc2a0541949d1d9320a641b82fd392b0d3d8145469ca4709ae769980e)...done
unpacking docker.io/kubernetesui/dashboard:v2.3.0 (sha256:11e006ccc6399880360f6bafaf8545892fe3c92959074297205937496f8baa86)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.6 (sha256:1f977343873ed0e2efd4916a6b2f3075f310ff6fe42ee098f54fc58aa7a28ab7)...done
unpacking docker.io/library/registry:2.7.1 (sha256:169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375)...done
unpacking k8s.gcr.io/ingress-nginx/controller:v1.1.0 (sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a)...done
unpacking k8s.gcr.io/metrics-server/metrics-server:v0.5.2 (sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10)...done
unpacking k8s.gcr.io/pause:3.1 (sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea)...done

microk8s.ctr content fetch --all-platforms docker.io/library/busybox:1.34.1

and need pause as well.