Adventures with Bootable Containers
Posted on August 18, 2024
Every so often, I come across some new software that really catches my attention - bootc(https://github.com/containers/bootc) and bootable containers are the most recent examples of this. The cogs in my head starting spinning when I first heard about bootable containers with some of the potential use cases for bootable containers.
What are Bootable Containers?
Bootable containers are a way of deploying and booting from OCI container images. There is a huge amount of tooling around building and publishing container images these days thanks to the cloud native ecosystem. These tools help to make it very easy to define and deploy immutable image based OS installs. I would recommend checking out the "Getting Started with Bootable Containers" (https://docs.fedoraproject.org/en-US/bootc/getting-started) guide in the Fedora Docs which I found very useful as a starting point.
I have been using the "immutable" variants of Fedora on my laptops for the last number of years - started off with Silverblue but I have moved to Sway Atomic in the last couple of years. Overall I have been very impressed with this "immutable" model on my laptops. It helps me to keep my base OS install very clean over long periods of time while I can still use toolbox containers for any development tools that I need which are not included in the base. This has led to a very stable experience and have made upgrade worries a thing of the past for me. The most impressive "immutable" experience that I have had so far is the Steam Deck - you turn it on and it just works every time without any messing about. The immutable OS really suits the appliance like nature of the Steam Deck.
I have a couple of machines that I would like to treat as simple appliances along the same lines of the Steam Deck. The first machine is a Raspberry Pi 4 that I use as a TV set top box with Kodi and Jellyfin. This machine should just boot directly into the Kodi interface and connect to my Jellyfin server so it seemed like a good candidate for bootable containers.
Immutable Kodi Box
I am fairly familiar with building containers so it didn't take me very long to get to a point where I had an image booting on the Pi 4 - there were a couple of extra steps needed due to firmware fun with Arm boards. The Containerfile for this machine is pretty straightforward.
FROM quay.io/fedora/fedora-bootc:latest
RUN dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm && dnf install -y kodi polkit sway unzip && dnf clean all
COPY etc /etc
RUN systemctl set-default graphical.target && systemctl enable kodi.service
As you can see the Containerfile is only a few lines long but a lot of the configuration is contained under the etc folder that is copied into the container image
$ tree etc/
etc/
├── hostname
├── sudoers.d
│ └── wheel-nopasswd
├── sway
│ └── sway-config
└── systemd
└── system
└── kodi.service
The hostname file is used for setting the hostname of the machine - kodi-pi in this case. wheel-nopasswd just removes the need for members of the wheel group to enter a password when using sudo.
$ cat etc/sudoers.d/wheel-nopasswd
# Enable passwordless sudo for the wheel group
%wheel ALL=(ALL) NOPASSWD: ALL
The kodi.service systemd unit starts the sway desktop with the included sway-config. Th is file tells sway to start without the bar and borders and to open directly into Kodi.
$ cat etc/systemd/system/kodi.service
[Unit]
Description=Kodi
After=remote-fs.target systemd-user-sessions.service network-online.target nss-lookup.target sound.target polkit.service lircd.service
Wants=network-online.target polkit.service
Conflicts=getty@tty1.service
[Service]
User=brian
Group=brian
SupplementaryGroups=audio dialout disk input render tty video
PAMName=login
TTYPath=/dev/tty1
ExecStartPre=+-/usr/bin/sh -c 'echo "on 0" | cec-client -s'
ExecStart=/usr/bin/sway -c /etc/sway/sway-config
ExecStop=/usr/bin/killall --user brian --exact --wait kodi.bin
Restart=on-abort
StandardInput=tty
StandardOutput=journal
# Hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
[Install]
WantedBy=graphical.target
$ cat etc/sway/sway-config
# Disable the bar and windows borders
bar {
mode invisible
}
default_border none
# Start Kodi on Sway start and exit Sway on player exit
exec swaymsg -t get_outputs && /usr/bin/kodi-standalone && /usr/bin/swaymsg exit
The combination of these configs allow me to simply power on the Raspberry Pi 4 and jump into Kodi to watch some films.
Ok, so I have the container built but how do I get this booting on a machine? There's a useful tool called bootc-image-builder(https://github.com/osbuild/bootc-image-builder) which takes in a bootable container image and outputs a bootable disk image. There are a number of supported disk image types including qcow2, raw and anaconda-iso. Podman can be used to run the bootc-image-builder - the following command is what I used to build a raw disk image for the Pi 4:
$ sudo podman build --arch arm64 -t kodi-pi:0.1.0 .
$ sudo podman run --rm -it --privileged --security-opt label=type:unconfined_t -v $(pwd)/output:/output -v $(pwd)/config.toml:/config.toml:ro -v /var/lib/containers/storage:/var/lib/containers/storage quay.io/centos-bootc/bootc-image-builder:latest --type raw --rootfs xfs --target-arch aarch64 --local localhost/kodi-pi:0.1.0
There are a few things worth noting in the above command. Firstly the config.toml that is mounted into the bootc-image-builder container is a handy configuration file for making customizations to your disk images like adding users or custom file systems. Here is how you can add a simple user to your image:
$ cat config.toml
[[customizations.user]]
name = "brian"
key = "ssh-rsa AAAA..............................................................................................................................................."
groups = ["wheel"]
You can also see two more directories mounted into the bootc-image-builder container - /var/lib/containers/storage allows the image-builder container to see your previously built bootable container image while the output directory is used for writing the resulting disk image.
The command also specifies which type of disk image to output and the type of file system to use for the root filesystem.
In my case I had to write the resulting disk image to a micro SD card and include the right firmware for the Raspberry Pi 4 to the EFI partition on the micro SD card before trying to boot the Pi. Luckily Fedora CoreOS have good documentation on how to add the required firmware to boot the Pi 4 (https://docs.fedoraproject.org/en-US/fedora-coreos/provisioning-raspberry-pi4/#_installing_fcos_and_booting_via_u_boot) which is the same process as for these bootc images.
Following all this I was ready to pop in the micro SD card into the Pi and power it on where after a few minutes I was greeted by the Kodi UI. When I was trying this out and testing out different ways of booting into Kodi, I found it very handy to run tests by outputting a qcow2 image and just starting it as a simple VM on my local machine - this allowed me to iterate over the configuration files very quickly. I spent some time trying to figure out how to include the Jellyfin Kodi addon in the bootable container image but unfortunately never got it working. To add this addon I had to precreate the ~/.kodi/addon directory and unzip the jellyfin addon there but this caused permission issues when starting Kodi. If anyone has an idea how I could achieve this please let me know!
After the initial install it was very easy to make changes to what packages and configurations were installed to the Pi. All I had to do was update the Containerfile with the additional packages/configs, rebuild the bootable container image, push to my local registry and rebase to the updated image.
$ sudo bootc switch git.careyscloud.ie/brian/kodi-pi:0.1.5Also if I push updated versions of that image tag to the registry - after a period of time bootc will pick up the updated and apply the changes automatically - if there are any issues with the update I can simply rollback to the previous image. OS update worries really become a thing of the past with this model of applying updates.
Forgejo Runner
The second machine that I thought would be a good candidate for bootable containers was a forgejo-runner(https://code.forgejo.org/forgejo/runner) machine. This forgejo-runner will allow me to reintroduce CI automation for my self hosted git repositories which I have found myself missing. I was more likely to add new blog posts and website updates when I had the CI/CD automation in place that would automatically deploy these changes once the pull requests are merged. This useful automation was all built with gitlab-ci so it was lost when I moved to the lighter weight gitea and eventually forgejo for my self hosted repos.
Forgejo(https://forgejo.org/) uses forgejo-runners to replicate a CI system similar to the well known Github Actions. This is a straight forward way for me to reintroduce automation for my self hosted repos. Previously when I provisioned CI machine for my old Gitlab instance, I installed it manually and included any of the dependencies that may have been needed. The machine would then sit there untouched for long periods of time just running CI jobs that were triggered against it. This again seems like a good candidate for bootable containers as I don't want to have to manually manage the machine or have to care about updating it.
There's a little bit more to the Containerfile for the forgejo-runner as it creates a user forgejo-runner that is a allowed to run processes without the user being logged into the machine.
FROM quay.io/fedora/fedora-bootc:latest
RUN groupadd forgejo-runner && adduser forgejo-runner -g forgejo-runner
RUN mkdir -p /var/lib/systemd/linger && touch /var/lib/systemd/linger/forgejo-runner
RUN dnf -y install git nodejs
COPY etc /etc
COPY lets-encrypt-r11.pem /usr/share/pki/ca-trust-source/anchors/
ADD https://code.forgejo.org/forgejo/runner/releases/download/v3.5.0/forgejo-runner-3.5.0-linux-amd64 /usr/bin/forgejo-runner
RUN update-ca-trust && \
chmod +x /usr/bin/forgejo-runner && \
systemctl enable podman.socket && \
chown forgejo-runner:forgejo-runner -R /etc/forgejo-runner && \
systemctl enable forgejo-runner.service
The podman socket has to be enabled in order for the forgejo-runner to use container images for different CI jobs in the workflows and the forgejo-runner service starts the forge-runner process at boot so that it can poll for any new CI jobs to run.
$ tree etc
etc/
├── forgejo-runner
│ └── config.yaml
├── hostname
├── sudoers.d
│ └── wheel-nopasswd
└── systemd
└── system
├── forgejo-runner.service
└── podman.socket.d
└── 10-socketgroup.conf
$ cat etc/systemd/system/forgejo-runner.service
[Unit]
Description=forgejo-runner
After=network-online.target
Wants=network-online.target
[Service]
User=forgejo-runner
Group=forgejo-runner
ExecStart=/usr/bin/forgejo-runner -c /etc/forgejo-runner/config.yaml daemon
ExecStop=/usr/bin/killall --user forgejo-runner
Restart=on-abort
StandardOutput=journal
[Install]
WantedBy=multi-user.target
The forgejo-runner needs to be registered with you forgejo server in order for it to pick up CI jobs. As I just need a single runner for my repos I was able to preregister the runner and add the .runner file to the bootable container image. The means that the runner machine is available as soon as it boots.
$ cat etc/forgejo-runner/.runner
{
"WARNING": "This file is automatically generated by act-runner. Do not edit it manually unless you know what you are doing. Removing this file will cause act runner to re-register as a new runner.",
"id": 4,
"uuid": "8973ae00-2f7a-4b15-95b1-922ffw4267",
"name": "podman-runner",
"token": "c54aa7950c991f391d424f980bc17infw325",
"address": "https://git.careyscloud.ie",
"labels": [
"podman"
]
}
This forgejo-runner will be handy for pushing updates to this website automatically but also for publishing updated images for itself and my Kodi Pi 4 machine. I have been trying to think of other machines that may be suited to bootable containers - my Gen10 Microserver which is my main home server could be another candidate as all of the services running on this machine are running inside containers using podman which should be easily included in a bootable container. I have also been playing with the idea of migrating my home router from pfsense to a fedora based bootable container image. I should be able to define all of the required network interfaces with NetworkManager config files and firewall rules using firewalld.conf. A pihole container could be used as the DHCP and DNS servers for the networks. The plan is to wait and see how this image based flow works for these two initial machines first before attempting to migrate the home router or server.
I'd be very interested to hear if anybody else has tried building a home router using bootable containers?!