Using the PoC Installer in an Air-Gapped Environment
Defining Air-Gapped Environments
An air-gapped environment is any environment in which the running hosts will not have access to the greater internet. This proposes a situation in which these hosts are unable to get access to various needed bits of software from Element and also are unable to share telemetry data back with Element.
For some of these environments, they can be connected to the internet from time to time and updated during those connection periods. In other environments, the hosts are never connected to the internet and everything must be moved over sneaker net.
This guide will cover running the microk8s installer when only sneaker net is available as that is the most restrictive of these environments.
Preparing the media to sneaker net into the air gapped environment
First, you will need to get the latest On-Premise installer. We will unpack that into an installation directory and then you will create a configuration directory like this:
tar xvzf /path/to/installer.tar.gz
mkdir -p ~/.element-onprem-config/airgapped/images
Now, let's set up a minimal parameters.yml and secrets.yml. Run these commands in the installer directory:
sh build_secrets.sh
mv secrets.yml ~/.element-onprem-config/
cp config-sample/parameters.yml.sample ~/.element-onprem-config/paramters.yml
Edit ~/.element-onprem-config/secrets.yml and set ems_image_store_username and ems_image_store_token. You should have this username/token pair provided to you by Element. If not, contact your Element representative.
Edit ~/.element-onprem-config/parameters.yml and set the following two parameters:
images_dir: ~/.element-onprem-config/airgapped/images
local_registry: 127.0.0.1:32000
Start by downloading the container images in the installer directory:
bash download_images.sh ~/.element-onprem-config/ -o ~/.element-onprem-config/airgapped/images
Now, in the installer directory, go get the pip artifacts:
mkdir ~/.element-onprem-config/airgapped/pip
pip3 download -r requirements.txt -d ~/.element-onprem-config/airgapped/pip
pip3 download wheel setuptools_scm pyyaml MarkupSafe cryptography simplejson -d ~/.element-onprem-config/airgapped/pip
Now, we need to get the necessary ansible-galaxy collections:
mkdir ~/.element-onprem-config/airgapped/galaxy
pip3 install ansible --upgrade --user
~/.local/bin/ansible-galaxy collection download -r requirements.yml -p ~/.element-onprem-config/airgapped/galaxy
Now, let's download the requiste items from Snapcraft:
mkdir ~/.element-onprem-config/airgapped/snaps
cd ~/.element-onprem-config/airgapped/snaps
snap download microk8s --channel=1.2124
snap download core18
snap download snapd
Now, let's grab helm3:
cd ~/.element-onprem-config/airgapped
wget https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz
tar xvzf helm-v3.5.0-linux-amd64.tar.gz
Now let's tar all of this up:
cd ~
tar cvzf installer-sneaker-net.tar.gz ./.element-onprem-config ./<path to installer>
And now we take installer-sneaker-net.tar.gz into our air-gapped environment.
Running the installer in the air gapped environment
Make sure that the machine is set up per the instructions in our PoC Installer Guide up through the Unpacking the Installer section.
Sneaker net over your tarball and unpack it in your user's home directory.
Patch install.sh, changing this line:
pip3 install -r requirements.txt --upgrade --user
to:
pip3 install -r requirements.txt --upgrade --user --no-index --find-links ~/.element-onprem-config/airgapped/pip
and this line:
ansible-galaxy collection install -r requirements.yml
to:
cd ~/.element-onprem-config/airgapped/galaxy && ansible-galaxy collection install -r requirements.yml && cd -
Make sure you finish setting up parameters.yml and secrets.yml per the PoC Installer Guide as well.
Now we need to install microk8s. To do that, we need to finish configuring snapd:
sudo systemctl start snapd
sudo systemctl enable snapd
cd ~/.element-onprem-config/airgapped/snaps
sudo snap ack core18_2409.assert
sudo snap install core18_2409.snap
sudo snap ack snapd_15904.assert
sudo snap install snapd_15904.snap
sudo snap ack microk8s_3202.assert
sudo ln -sf /var/lib/snapd/snap /snap
sudo snap install microk8s_3202.snap --classic
sudo usermod -a -G microk8s karl1
sudo chown -f -R karl1 ~/.kube
sudo mkdir /var/snap/microk8s/current/bin/
sudo cp ~/.element-onprem-config/airgapped/linux-amd64/helm /var/snap/microk8s/current/bin/helm3
sudo systemctl reboot
You should now be ready to run the installer, which is going to fail at a few key points:
- calico-node tends to hang and needs to be killed for the installer to proceed -- this holds up port 9099.
- The registry doesn't actually start and this causes port 32000 to not be accessible. This happens due to kube-system calico-kube-controllers-f7868dd95-864wz 0/1 ErrImagePull 0 16m
not being available. We are missing all of these images : (re-enabling the internet for a brief moment allowed all these images to pull)
kube-system hostpath-provisioner-566686b959-bpj9l 1/1 Running 0 4m21s
kube-system dashboard-metrics-scraper-78d7698477-jf2wc 1/1 Running 0 4m21s
kube-system coredns-7f9c69c78c-c2rb4 1/1 Running 0 4m21s
kube-system metrics-server-8bbfb4bdb-8twg6 1/1 Running 0 4m21s
ingress nginx-ingress-microk8s-controller-nt9c7 1/1 Running 0 4m21s
container-registry registry-9b57d9df8-7mr5j 1/1 Running 0 4m21s
kube-system calico-kube-controllers-f7868dd95-9fsdc 1/1 Running 0 32s
kube-system kubernetes-dashboard-85fd7f45cb-bsc8r 1/1 Running 0 19s
bits and bobs and random notes
wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.3/mkcert-v1.4.3-linux-amd64
EPEL via satellite on RHEL. Ubuntu via some local repo as well.
EPEL is needed to install pwgen, snapd, snap-confine, and snapd-selinux in RHEL.
If you install snapd manually, you need to then start snapd (and enable snapd) or the installer will puke.
cat /var/lib/snapd/snap/microk8s/current/actions/enable.helm3.sh | awk -F 'HELM_VERSION=' '{print $2}' - | xargs | sort -b | uniq
In my case, this yields:
v3.5.0
With this knowledge, we can now construct a wget command to grab the proper helm3 binary:
wget https://get.helm.sh/helm-<VERSION>-linux-amd64.tar.gz
which with our output from above would be:
wget https://get.helm.sh/helm-v3.5.0-linux-amd64.tar.gz
Now we need to unpack this tarball:
tar xvzf helm-v3.5.0-linux-amd64.tar.gz
and now we will find the proper helm binary for microk8s in the linux-amd64 directory.
Now, we need to take the snap artifacts and the helm binary to our disconnected environment.
Microk8s must be running and the installer will have failed to enable helm3 to do this:
sudo cp /path/to/helm /var/snap/microk8s/current/bin/helm3
microk8s.enable helm3
Once you have helm3 enabled, you can opt to re-run the installer as it will now get past the enabling of helm3.
ContainerD Registry export for non-Element bits: (on connected host)
for i in $(microk8s.ctr images list -q | grep -v sha256); do tarfile=$(echo $i | awk -F '/' '{print $NF}' -); microk8s.ctr images export $tarfile.tar $i && echo "microk8s.ctr images import --base-name $i $tarfile.tar" >> import-script; done
This creates import-script which can be run by moving the script and tar files to disconnected box and running:
chmod +x ./import-script
./import-script
unpacking docker.io/calico/cni:v3.19.1 (sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4)...done
unpacking docker.io/calico/kube-controllers:v3.17.3 (sha256:072e5f661f9c1af9ab49d127743fdbfe5d8ba16bac7c2616b92e64dd304ec3c4)...done
unpacking docker.io/calico/node:v3.19.1 (sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55)...done
unpacking docker.io/calico/pod2daemon-flexvol:v3.19.1 (sha256:bcac7dc4f1301b062d91a177a52d13716907636975c44131fb8350e7f851c944)...done
unpacking docker.io/cdkbot/hostpath-provisioner:1.1.0 (sha256:317ed1b7324dc5f20db5dfdc0647a26a1ac44df426e9caad9cfdd6ba84c47697)...done
unpacking docker.io/coredns/coredns:1.8.0 (sha256:cc8fb77bc2a0541949d1d9320a641b82fd392b0d3d8145469ca4709ae769980e)...done
unpacking docker.io/kubernetesui/dashboard:v2.3.0 (sha256:11e006ccc6399880360f6bafaf8545892fe3c92959074297205937496f8baa86)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.6 (sha256:1f977343873ed0e2efd4916a6b2f3075f310ff6fe42ee098f54fc58aa7a28ab7)...done
unpacking docker.io/library/registry:2.7.1 (sha256:169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375)...done
unpacking k8s.gcr.io/ingress-nginx/controller:v1.1.0 (sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a)...done
unpacking k8s.gcr.io/metrics-server/metrics-server:v0.5.2 (sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10)...done
unpacking k8s.gcr.io/pause:3.1 (sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea)...done