Homelab Update 2024
Posted on November 14, 2024
This homelab update has become an (almost) annual blog post for me. Over the years I have probably gotten more enjoyment out of these posts than I imagine anybody else has. They are great for me to see how my homelab has evolved and changed over time while also reminding me of some of the reasons why I made certain changes. I was really expecting a quiet year for my homelab in 2024... but it turned out be quite an eventful year for my self-hosted setup.
The Unexpected
Things always seem to break when I am gone away for a few days. Back in September I arrived home after a long weekend to find that I couldn't access any of my machines. I had an inkling that there could be some issue with my network router but I was not willing to face it yet so I just avoided it for a couple of weeks. Eventually I had to bite the bullet and check it out.
The eMMC drive had died on my old netgate sg-1100 which I found quite surprising as I hadn't any disk intensive tasks setup out on the router apart from whatever tasks came as default from pfSense. The router had been running constantly for a number of years to be fair and had been very reliable up until this point. Netgate had recently switched to using ZFS for the root filesystem on the SG-1100 which I thought was a bit overkill for this small router so that might have caused more activity on the drive. I had put off investigating the issue partly because I wasn't sure what I was going to replace the SG-1100 with. I didn't really want to go for another Netgate device as they are quite expensive and difficult to get here in Ireland. Also I had gotten tired of being locked into using pfSense as the OS on the router. The 4 port x86 routers/mini pcs that you can get on Amazon seem to be a bit dodgy and some of them come with much older CPUs. Not really interested in UniFi either for a lot of the same reasons as the Netgate devices. Ideally what I am looking for is a low power x86 Mini PC with a least 4 2.5G ethernet ports, good Linux support and that comes from a reasonably reputable brand. The plan would be to build a Fedora based network router using bootc and bootable containers. The immutability and easy rollbacks would be really nice in a home router. If anybody knows of any devices that fit my needs, please let me know! For now, I am back using my ISPs router and my homelab is on a flat network with everything else including a number of what I would consider as untrusted devices which I am not a fan of.
For the last year or so I have been running all of my homelab services on a dual core Opteron HP Microserver Gen 10 which was released back in 2017. While it could run everything ok, the performance and the experience were not great especially for services like Immich and Jellyfin. I had been on the look out for a compute node to move these services over to so that the old Microserver could focus on just handling storage which it does brilliantly. There was one machine that was released during the year that ticked all my boxes and seemed too good to be true - the Minisforum MS-01 (https://store.minisforum.com/products/minisforum-ms-01). I am sure the majority of ye have at least heard of this machine or come across it on YouTube but it still deserves at least a brief overview of the hardware specs. I went for the barebones option with the Intel i9-13900H which is a 14 core 20 thread CPU.
The three main selling points for me were:
- The Size: Its only a tiny bit bigger than those one litre PCs from Dell & Lenovo and it fits perfectly stacked on top of the Microserver Gen10
- The Dual 10Gbps SFP+ ports: I haven't come across any other machines this size that offer 10Gbps network connectivity. This allows me to connect both my NAS (Microserver) and the MS-01 to my main network switch(QNAP QSW-M2108-2C) with 10Gb. The services running on the MS-01 can have fast network access to the bulk storage on the Microserver which means I don't have to worry about installing large storage drives in the compact MS-01.
- The Intel CPU: Intel Quick Sync is well known for its fantastic video encode and decode performance which I was sorely missing in the Microserver. Also since the i9-13900H is a mobile CPU, it provides high performance and good energy efficiency too.
The barebones option suited me as I had a couple of small NVME drives from previous laptop upgrades that I could use for boot drives and I was able to wait for a good deal on a 64Gb kit of RAM. I have moved all of my homelab services to the MS-01 and added a couple of VMs including a development VM and a couple of CI VMs that run workflows for my selfhosted forgejo instance. I have struggled to even stress the system with all of this running while doing development work including running some KubeVirt builds in parallel. So far the MS-01 has proven to be a great addition to my homelab!
Coming into 2024, free space in my main storage pool was getting a bit tight and the disks themselves were starting to get close to 4 years of power on time. I tried cleaning and pruning as much of the data as I could but the total used storage kept creeping up with every batch of photos from my Sony A7. A storage upgrade was going to have to happen eventually. Luckily I was able to find a buyer for my four old 4TB Seagate Ironwolf drives and I was able to replace them with four 16TB Seagate Ironwolf drives. These drives are configured in a BTRFS RAID 10 which gives me around 32TB of usable space. This is a huge upgrade compared to the 8TB of usable space I had previously so hopefully this should last me for many years to come.
Software Changes
Looking back on my homelab 2023 update from early last year, there are some notable changes that have happened since then when it comes to software. Firstly I have moved away from Gitea for my self hosted git repos to a fork of Gitea called Forgejo - the two forges are still very similar but I have been enjoying using Forgejo so far. As I had some increased compute capacity thanks to the MS-01 I added two forgejo-runner virtual machines which provides Github Actions like CI functionality for my self-hosted git repos.
I dropped both Mastodon and Podgrab from my running services. I fell back into using Apple Podcasts for listening which was just much handier than Jellyfin for podcasts which meant that Podgrab was no longer being needed. Also the last time I checked, there hadn't been any activity in the Podgrab repo for a couple of years so I am not sure if it is being maintained anymore. I moved my Mastodon account back to Fosstodon because I was using it less and the performance issues that I had seen previously on mastodon.ie were no longer an issue as interest and usage of Mastodon seemed to drop after the initial surge due to the problems with Twitter. I had heard great things about Immich as a Google Photos replacement so I decided to give it a go. It quickly replaced Photoprism as the native IOS app made it very easy to backup photos from my phone to my NAS. Immich has really impressed me so far and its a great solution for anyone who doesn't want to be relying on iCloud or Google Photos
For the first couple of months of this year, I was running a self-hosted LLM chat bot called Serge Chat but the novelty wore off pretty quickly and I didn't really have the hardware required to get decent performance from it. You really need some kind of dedicated GPU for these local LLMs to be anyway useful at the moment. One recent addition to my homelab that has managed to stick around is TubeArchivist. TubeArchivist is very handy especially when paired with Jellyin. You can pick up some of your favourite YouTube videos with TubeArchivist and watch them whenever you like through Jellyfin.
Potential Upgrades for 2025:
- Add some larger NVME drives to the MS-01: The 500GB drive that is currently installed is getting pretty tight on free space already and as the MS-01 has two more free M.2 slots there is plenty of room for more drives.
- Build a new network router potentially running with a Fedora based bootable container
- Move away from podman-compose for deploying my services to Quadlets