An ephemeral KubeVirt development environment on Fedora Silverblue

Framework laptop

Posted on March 6, 2023

I've been using Fedora Silverblue on my personal laptop for the last couple of years and it's been fantastic - it provides a very stable desktop experience which can be upgraded very easily. For those that might not have heard of Silverblue - it is basically an immutable version of Fedora Workstation. Desktop applications are installed with flatpak and for everything else there is toolbx. Toolbx is a great tool that uses podman to create containerized environments where you can install any packages that you may be missing in the base Silverblue install. If you really need a package installed on the host, you can layer the package using rpm-ostree install but I generally treat this as a last resort. Currently I only have two packages layered on the base install so I do like to keep it pretty clean. This is one of the main reasons why I created this ephemeral environment built from the scripts here - silverblue-kv


Screenshot of About Laptop

As I am one of the CI maintainers for the KubeVirt project, occassionally I do need to test some things quickly on my laptop. Rather than ruining my lovely Silverblue install by layering a tonne of dependencies, I decided to create these scripts to provide a very useful ephemeral development environment that I can spin up in no time. There are a couple of reasons why I decided against using toolbox for this. Firstly a lot of the development flows in KubeVirt are containerized and rely heavily on podman/docker. I wanted to keep the podman in podman instance separate so that my machine doesn't get clogged up with a large number of container images etc. As far as I can see, to get podman running in a toolbox container you have to pass the host's podman socket to the toolbox container which is not what I was looking for. I also wanted to make this environment docker compatiable so that I could use this container for a number of different projects that depend on docker or podman for building/testing. Unfortunately docker requires quite a few more privileges than podman which are not available when running under toolbox. Another reason is that previously I have been guilty of treating some of my toolbox containers as "pet" environments - as in spending time to set up toolboxes for different projects only to realise that they've disappeared after I have run a podman system prune without thinking...so having everything built into the container image avoids this.

This environment was mainly built to target KubeVirt but it should be useful for any golang repos or any projects that use podman/docker to containerize their development flows. These scripts do not require Fedora Silverblue - they should work on any machine with podman installed. The great thing about this evnironment for me is that it allows me to replicate the actual CI environment very closely which means that I can troubleshoot CI issues locally rather than having to jump into the CI environment. These scripts do require sudo privileges so if you are going to try this out please do give them a read first. If your projects use large container images (like some of the KubeVirt CI images), you may want to persist the storage for the container images rather than constantly redownloading the same images so I added an option to create a volume for the podman in podman data.

./up.sh -p [path to working directory]

For those that may not be aware, the KubeVirt repo actually comes with a built-in virtual Kubernetes cluster provider that can be spun up in a couple minutes and this ephemeral environment has everything needed to create the cluster. This cluster is perfect for carrying out quick test deployments but can also be used by someone wanting to try out KubeVirt. You will need a machine with 16GB of RAM if you want to use this test cluster - even with the single node cluster, my RAM usage goes up to about 9GB. The container images for the Kubernetes nodes are quite large so it can take a bit to pull the images depending on your internet connection.

To spin up the test cluster:

  • Clone the KubeVirt repo
    • git clone https://github.com/kubevirt/kubevirt
  • If you want to try the ephemeral dev environment
    • git clone https://github.com/brianmcarey/silverblue-kv
    cd silverblue-kv
    ./up.sh [path to kubevirt repo]
  • To start the virtual Kubernetes cluster:
    • cd ./kubevirt
    make cluster-up
    export KUBECONFIG=$(./cluster-up/kubeconfig.sh)

This will give you a single node cluster installed on a CentOS stream base. If you want a multinode cluster, you can set an environment variable before bringing up the cluster.

export KUBEVIRT_NUM_NODE=3
make cluster-up

Once you have a cluster, its pretty easy to get started with KubeVirt - you just have to run a make cluster-sync from the kubevirt directory - this will install the latest KubeVirt from the main branch. At this point you should be able to create VMs using KubeVirt. These VMs will be running with nested virtualization so the performance won't be as good as running on bare metal but it is very useful for getting an idea of how to use KubeVirt.


Listing running pods

Again while this environment is mainly aimed at KubeVirt, it should still be very useful for any projects that are go based or that use podman/docker in their workflows.