In the previous part of this series I created a base Linux distribution from a running LFS system. This version only ran as a container which has several benefits that makes building the distribution a lot easier. For a simple container I didn't have to have:
- A service manager (systemd)
- Something to make it bootable on x86_64 systems (grub, syslinux, systemd-boot)
- A kernel
- An initramfs to get my filesystem mounted
- File-system utilities since there's a folder instead of a filesystem.
A few of these are pretty easy to get running. I already have all the dependencies to build a kernel so I generated a kernel from the linux-lts package in Alpine Linux.
To make things easier for myself I just limited the distribution to run on UEFI x86_64 systems for now. This means I don't have to mess with grub ever again and I can just dump systemd-boot into my /boot folder to get a functional system. I had to build this anyway since I had to build systemd to have an init system for my distribution.
The Initramfs
The thing that took by far the longest is messing with the initramfs to make my test system boot. The initramfs generator is certainly one of the parts that have the most distribution-specific "flavor". Everyone invents it's own solution for it like mkinitcpio for ArchLinux and mkinitfs for Alpine and initramfs-tools for Debian as a few examples.
I did the only logical thing and reinvented the wheel here. I'm even planning to reinvent it even further! Like the above solutions my current initramfs generator is a collection of shell scripts. The initramfs is a pretty simple system after all: it has to load some kernel modules, find the rootfs, mount it and then execute the init in the real system.
For a very minimal system the only required thing is the busybox binary, it provides the shell script interpreter required to run the messy shell script that brings up the system and also provides all the base utilities. Due to my previous experiences with BusyBox modprobe in postmarketOS I decided to also move the real modprobe binary in the initramfs to have things loading correctly. To complete it I also added blkid instead of relying on the BusyBox implementation here to have support for partition labels in udev so no custom partition-label-searching code is required.
Getting binaries in the initramfs is super easy. The process for generating an initramfs is:
- Create an empty working directory
- Move in the files you need into the working directory from the regular rootfs like
/usr/bin/busybox>/tmp/initfs-build/usr/bin/busybox - Add in a script that functions as pid 1 in the initramfs and starts execution of the whole system
- Run the
cpiocommand against the/tmp/initfs-builddirectory to create an archive of this temporary rootfs and run that throughgzipto generateinitramfs.gz
Step 2 is fairly simple since I just need to copy the binaries from the host system, but those binaries also have dependencies that need to be copied to make the executable actually work. Normally this is handled by the lddtree utility but I didn't feel like packaging that. It is a shell script that does a complicated task which is never a good thing and it depends on python and calling various ELF binary debugging utilities.
Instead of using lddtree I brought up Hare on my distribution and wrote a replacement utility for it called bindeps. This is just a single binary that loads the ELF file(s) and spits out the dependencies without calling any other tools. This is significantly faster than the performance overhead of lddtree which was always the slowest part of generating the initramfs for postmarketOS.
The output format is also optimized to be easily parse-able in the mkinitfs shellscript.
$ lddtree /usr/sbin/blkid /usr/sbin/modprobe
/usr/sbin/blkid (interpreter => /lib64/ld-linux-x86-64.so.2)
libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
/usr/sbin/modprobe (interpreter => /lib64/ld-linux-x86-64.so.2)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5
libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
$ bindeps /usr/bin/blkid /usr/bin/modprobe
/usr/lib/ld-linux-x86-64.so.2
/usr/lib/libblkid.so.1.1.0
/usr/lib/libc.so.6
/usr/lib/libzstd.so.1.5.6
/usr/lib/liblzma.so.5.6.3
/usr/lib/libz.so.1.3.1
/usr/lib/libcrypto.so.3
The bindeps utility seems to be roughly 100x faster in the few testcases I've used it in and it outputs in a format that needs no further string-mangling to be used in a shell script. In BodgeOS mkinitfs it's used like this:
binaries="/bin/modprobe /bin/busybox /bin/blkid"
for bin in $binaries ; do
install -Dm0755 $bin $workdir/$bin
done
bindeps $binaries | while read lib ; do
install -Dm0755 $lib $workdir/$lib
done
The next part is the kernel modules. Kernel modules are also ELF binaries just like the binaries I just copied over but they sadly don't contain any dependency metadata. This metadata is stored in a seperate file called modules.dep that has to be parsed seperately. I did not bother with this and copied the solution from the initramfs generator example from LFS and just copy hardcoded folders of modules into the initramfs and hope it works.
The file format for modules.dep is trivial so I really want to just integrate support for that into bindeps in the future.
Debugging the boot
It's suprisingly painful to debug a non-booting Linux system that fails in the initramfs. I wasted several hours figuring out why the kernel threw errors at the spot the initramfs should start executing which ended up being an issue with the /sbin/init file had the wrong shebang line at the start so it was not loadable. The kernel has no proper error message that conveys any of this.
After I got the initramfs to actually load and start a lot of time was wasted on executables missing the interperter module. In the example above this is the /lib64/ld-linux-x86-64.so.2 line. The issue here ended up that I was just missing the /lib64 symlink in my initramfs. This was very hard to debug in a system without debug utilities because nothing could execute.
After all that I spend even more time figuring out why I had no kernel log lines on my screen. After much annoyance this turned out to be missing options in the kernel config for the linux-lts kernel config I took from Alpine Linux. So instead of fixing that I took the kernel config from ArchLinux and rebuild the linux-lts package. This fixed my kernel log output issue but also added a new one... The keyboard in my laptop wasn't working in the initramfs.
I never did figure out which module I was missing for that because I fixed the rest of the initramfs script instead so it just continues on to the real rootfs where all the modules are available.
After all that I did manage to get to a login prompt though!
Cleaning up
After booting up this I realized it would be really handy if I actually had a /etc/passwd file in my system and some more of the bare essentials so I could actually log in and use the system.
This mainly involved adding missing dependencies in packages and packaging a few more files in /etc to make it a functional system. After the first boot the journal had a neat list of system binaries from util-linux that systemd depends on but not explicitly, so I added those dependencies to my systemd packaging.
I also had to fix the issue that my newly-installed system did not trust the BodgeOS repository, I was missing the keys package that installs the repository key in /etc/apk/keys for me. In this process I noticed that the key I built the system with was called `-randomdigits.pub` instead of being prefixed with a name. This is pretty annoying because this name is embedded in all the compiled packages and I didn't want to ship a file with that name in my keys package.
There seemed to be a nifty solution though: the abuild-sign tool appends a key to a tar archive, which is normally used to sign the APKINDEX.tar.gz file that contains the package list in the repository. I decided to run abuild-sign *.apk in my main repository after adjusting the abuild signing settings with a correct key name.
Apparently this breaks .apk files and after inspection they now had two keys in them and neither my development LFS install and my test BodgeOS install wanted to have anything to do with the packages anymore.
In the end I had to throw away my built packages and rebuild everything again from the APKBUILD files I had. Luckily this distribution is not that big yet so a full rebuild only took about 2.5x the duration of Dark Side of the Moon.
Next steps
Now I have a basic system that can boot I continued with packaging more libraries and utilites you'd expect in a regular Linux distribution. One of the things I thought would be very neat is having curl available, but that has a suprising amount of dependencies. Some tools like git will be useful too before I can call this system self-hosting.
I also want to remove all the shell scripts from the initramfs generation. None of the tasks in the initramfs are really BodgeOS specific and most of the complications and bugs in this initramfs implementation (and the one in postmarketOS) is because the utilities it depends on are not really intended to do this stuff and system bootup just has a lot of race conditions shell scripts are just not great at handling.
My current plan to fix that is to just replace the entire initramfs with a single statically linked binary. All this logic is way neater to implement in a good programming language.