Skip to main content

Using the PoC Installer in an Air-Gapped Environment

Defining Air-Gapped Environments

An air-gapped environment is any environment in which the running hosts will not have access to the greater internet. This proposes a situation in which these hosts are unable to get access to various needed bits of software from Element and also are unable to share telemetry data back with Element.

For some of these environments, they can be connected to the internet from time to time and updated during those connection periods. In other environments, the hosts are never connected to the internet and everything must be moved over sneaker net.

This guide will cover running the microk8s installer when only sneaker net is available as that is the most restrictive of these environments.

Preparing the media to sneaker net into the air gapped environment

First,Run youthe script to run download the images to a local directory. The script will needalso compute your images digest in a file named images_digests.yml. Copy the whole directory containing the images and the images_digests.yml file over the airgapped environment.

It is also required to get the latestairgap On-Premisedependencies. installer.They Weare willmade unpackof thattwo into an installation directory and then you will create a configuration directory like this:

tar xvzf /path/to/installer.tar.gz mkdirfiles -p ~/.element-onprem-config/airgapped/images

Now, let's set up a minimal parameters.yml and secrets.yml. Run these commands in the installer directory::

sh mv secrets.yml ~/
  • element-enterprise-poc-installer-airgapped-deps-<version>.element-onprem-config/ cp config-sample/parameters.yml.sample ~/.element-onprem-config/paramters.yml

Edit ~/.element-onprem-config/secrets.yml and set ems_image_store_username and ems_image_store_token. You should have this username/token pair provided to you by Element. If not, contact your Element representative.

Edit ~/.element-onprem-config/parameters.yml and set the following two parameters:

images_dir: ~/.element-onprem-config/airgapped/images

Start by downloading the container images in the installer directory:

bash ~/.element-onprem-config/ -o ~/.element-onprem-config/airgapped/images

Now, in the installer directory, go get the pip artifacts:

mkdir ~/.element-onprem-config/airgapped/pip
pip3 download -r requirements.txt -d ~/.element-onprem-config/airgapped/pip
pip3 download wheel setuptools_scm pyyaml MarkupSafe cryptography simplejson -d ~/.element-onprem-config/airgapped/pip

Now, we need to get the necessary ansible-galaxy collections:

mkdir ~/.element-onprem-config/airgapped/galaxy
pip3 install ansible --upgrade --user
~/.local/bin/ansible-galaxy collection download -r requirements.yml -p ~/.element-onprem-config/airgapped/galaxy

Now, let's download the requiste items from Snapcraft:

mkdir ~/.element-onprem-config/airgapped/snaps
cd ~/.element-onprem-config/airgapped/snaps
snap download microk8s --channel=1.24
snap download core18
snap download snapd

Now, let's grab helm3:

cd ~/.element-onprem-config/airgapped
tar xvzf helm-v3.5.0-linux-amd64.
  • element-enterprise-poc-installer-airgapped-microk8-<version>.tar.gz
  • Now let's tar all of this up:

    cd ~
    tar cvzf installer-sneaker-net.tar.gz ./.element-onprem-config ./<path to installer>

    And now we take installer-sneaker-net.tar.gz into our air-gapped environment.

    Running the installer in the air gapped environment

    Make sure thatExtract the machineairgapped isdependencies set up perto the instructionsairgapped indirectory our PoC Installer Guide up throughat the Unpackingroot of the Installerinstaller section.folder. You obtain the following directories :

    • airgapped/pip
    • airgapped/galaxy
    • airgapped/snaps
    • airgapped/containerd

    You airgapped machine will still require access to airgapped linux repositories depending on your OS.

    SneakerAdd netthe overfollowing your tarball and unpack itparameters in your user's home directory.

    Patch, changing this line:

    pip3 install -r requirements.txt --upgrade --user


    pip3 install -r requirements.txt --upgrade --user --no-index --find-links ~/.element-onprem-config/airgapped/pip

    and this line:

    ansible-galaxy collection install -r requirements.yml


    cd ~/.element-onprem-config/airgapped/galaxy && ansible-galaxy collection install -r requirements.yml && cd -

    Make sure you finish setting up parameters.yml and:

    • local_registry:
    • images_dir: <absolute path to the PoCcopied Installerimage Guidedirectory> as
    • well.

    Now we need to install microk8s. To do that, we need to finish configuring snapd:

    sudo systemctl start snapd
    sudo systemctl enable snapd
    cd ~/.element-onprem-config/airgapped/snaps
    sudo snap ack core18_2409.assert
    sudo snap install core18_2409.snap
    sudo snap ack snapd_15904.assert
    sudo snap install snapd_15904.snap
    sudo snap ack microk8s_3202.assert
    sudo ln -sf /var/lib/snapd/snap /snap
    sudo snap install microk8s_3202.snap --classic
    sudo usermod -a -G microk8s karl1
    sudo chown -f -R karl1 ~/.kube
    sudo mkdir /var/snap/microk8s/current/bin/
    sudo cp ~/.element-onprem-config/airgapped/linux-amd64/helm /var/snap/microk8s/current/bin/helm3
    sudo systemctl reboot

    You should now be ready to run the installer, which is going to fail at a few key points:

    1. calico-node tends to hang and needs to be killed for the installer to proceed -- this holds up port 9099.
    2. The registry doesn't actually start and this causes port 32000 to not be accessible. This happens due to kube-system calico-kube-controllers-f7868dd95-864wz 0/1 ErrImagePull 0 16m

    not being available. We are missing all of these images : (re-enabling the internet for a brief moment allowed all these images to pull)

    kube-system          hostpath-provisioner-566686b959-bpj9l        1/1     Running   0          4m21s
    kube-system          dashboard-metrics-scraper-78d7698477-jf2wc   1/1     Running   0          4m21s
    kube-system          coredns-7f9c69c78c-c2rb4                     1/1     Running   0          4m21s
    kube-system          metrics-server-8bbfb4bdb-8twg6               1/1     Running   0          4m21s
    ingress              nginx-ingress-microk8s-controller-nt9c7      1/1     Running   0          4m21s
    container-registry   registry-9b57d9df8-7mr5j                     1/1     Running   0          4m21s
    kube-system          calico-kube-controllers-f7868dd95-9fsdc      1/1     Running   0          32s
    kube-system          kubernetes-dashboard-85fd7f45cb-bsc8r        1/1     Running   0          19s

    bits and bobs and random notes


    EPEL via satellite on RHEL. Ubuntu via some local repo as well.

    EPEL is needed to install pwgen, snapd, snap-confine, and snapd-selinux in RHEL.

    If you install snapd manually, you need to then start snapd (and enable snapd) or the installer will puke.

    catthe /var/lib/snapd/snap/microk8s/current/actions/enable.helm3.shimages | awk -F 'HELM_VERSION=' '{print $2}' - | xargs | sort -b | uniq 

    In my case, this yields:


    With this knowledge, we can now construct a wget commandautomatically to grabyour local registry, and use these references to start the proper helm3 binary:workloads.

    When running the install script, add the parameter wget--airgapped<VERSION>-linux-amd64.tar.gzso


    whichit withinstalls ourits outputpip and galaxy dependencies from abovethe wouldairgapped be:folder.


    Now we need to unpack this tarball:

    tar xvzf helm-v3.5.0-linux-amd64.tar.gz

    and now we will find the proper helm binary for microk8s in the linux-amd64 directory.

    Now, we need to take the snap artifacts and the helm binary to our disconnected environment.

    Microk8s must be running and the installer will have failed to enable helm3 to do this:

    sudo cp /path/to/helm /var/snap/microk8s/current/bin/helm3
    microk8s.enable helm3

    Once you have helm3 enabled, you can opt to re-run the installer as it will now get past the enabling of helm3.

    ContainerD Registry export for non-Element bits: (on connected host)

     for i in $(microk8s.ctr images list -q | grep -v sha256); do tarfile=$(echo $i | awk -F '/' '{print $NF}' -); microk8s.ctr images export $tarfile.tar $i && echo "microk8s.ctr images import --base-name $i $tarfile.tar" >> import-script; done

    This creates import-script which can be run by moving the script and tar files to disconnected box and running:

    chmod +x ./import-script
    unpacking (sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4)...done
    unpacking (sha256:072e5f661f9c1af9ab49d127743fdbfe5d8ba16bac7c2616b92e64dd304ec3c4)...done
    unpacking (sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55)...done
    unpacking (sha256:bcac7dc4f1301b062d91a177a52d13716907636975c44131fb8350e7f851c944)...done
    unpacking (sha256:317ed1b7324dc5f20db5dfdc0647a26a1ac44df426e9caad9cfdd6ba84c47697)...done
    unpacking (sha256:cc8fb77bc2a0541949d1d9320a641b82fd392b0d3d8145469ca4709ae769980e)...done
    unpacking (sha256:11e006ccc6399880360f6bafaf8545892fe3c92959074297205937496f8baa86)...done
    unpacking (sha256:1f977343873ed0e2efd4916a6b2f3075f310ff6fe42ee098f54fc58aa7a28ab7)...done
    unpacking (sha256:169211e20e2f2d5d115674681eb79d21a217b296b43374b8e39f97fcf866b375)...done
    unpacking (sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a)...done
    unpacking (sha256:6385aec64bb97040a5e692947107b81e178555c7a5b71caa90d733e4130efc10)...done
    unpacking (sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea)...done
    microk8s.ctr content fetch --all-platforms

    and need pause as well.