BodgeOS - BrixIT Blog

BodgeOS pt.4: A working browser

Martijn Braam — Sat, 25 Jan 2025 11:35:26 -0000

After the previous post about bringing up Sway I spend a bit of time packaging random system components you'd expect for a desktop system. Mainly building GTK so I can have some actual applications running. This is mainly pretty relaxing work. Find a component to build, figure out the dependencies, package them one by one by figuring out the required commands and making a tiny build script.

Occasionally the zen of packaging gets disturbed by having to figure out why some build system does something weird, like really wanting to put files in /usr/lib64 instead of /usr/lib while in the previous build it did not do this.

It turns out this is because some build systems perform auto detection of the host system to decide things like "what folder is a good place for the libraries". In this case it was cmake that has autoconfig behavior for this if you don't explicitly define that libs go in the lib folder. At some point installing a broken rebuild of glibc on the host system had changed lib64 back from a symlink to a regular folder again and all builds after that were broken.

The path to Firefox is paved with many dependencies

For my goal of running Firefox I have to deal with several dependencies:

GTK 3 and the components to make that work like Pango and Cairo.
A whole assortment of audio, video and image codecs, these can probably be skipped by manipulating the Firefox buildsystem, but it is probably easier to package all of these instead.
Nodejs is also a dependency for building Firefox since the javascript engine in Firefox itself can't be used to run the javascript components in the build system.
WASI for having WebAssembly support.
Rust for large chunks of the Firefox codebase.
Extra libraries like the Netscape Portable Runtime (nspr) and the libraries for other system features like Alsa and Pulse.

Luckily parts of this have already been packaged as part of getting my Sway desktop working. Many dependencies for GTK 3, WASI and Rust are already packaged for either Mesa or Swaybar, but I had to finally figure out how to bring up Rust in my distro.

This is quite hard in theory because rustc is written in Rust so I need older versions of Rust to build the current version of Rust, and older versions for that again and again until you reach the point where rustc is written in C again. Instead of doing all that I opted to go for the easy road. I curl | sh'ed a functional Rust compiler into my host system with the instructions for rustup.rs and then used that compiler to build my packaged Rust. This was all surprisingly smooth and easy.

Getting NodeJS functional was a bit more painful. This is mainly because it takes forever to build which means it takes forever to fix things while packaging. The same issue with the Firefox build itself actually. The worst thing I've encountered while packaging things is things that take a lot of time to build and don't have the configure script check all the dependencies it needs. I might have been annoyed with autoconf wasting a lot of time at the start of every build of every small package with checks that seem to be completely useless, but the time wasted there does not compare at all to the time wasted in building Firefox and having it crash an hour into the build complaining it's missing a dependency.

Desktop audio

One of the dependencies for Firefox is also the various audio libraries, which in turn means I have to bring up audio on my distro. I spend a moment to figure out wether it would be a PulseAudio or PipeWire system but ended up just picking PipeWire. Most of the issues I've seen with PipeWire seem to stem from bodged migrations from Pulse systems anyway and that is not an issue I would have to deal with. Unfortunately it turns out that for my nice, clean and modern PipeWire system I would still have to build PulseAudio first. While PipeWire provides the compatability layer with applications that use PulseAudio as audio backend (like Firefox) I still need to provide those applications with libpulse first at build time.

My package of PulseAudio only provides the library and not the daemon so I have the least amount of conflicting sound systems as possible in the distribution. Of course Alsa is also packaged but that is required anyway by PipeWire to access the hardware. For the Jack parts of PipeWire I have simply decided to not have the Jack parts.

Building Firefox

So once I had all the dependencies out of the way I started the painful process of building Firefox. This part contained a lot of failed builds and many many many hours of wasted build time due to the build failing at the finish line.

I spend a lot of time trying to figure out why I had syntax errors in the Rust files for the CSS engine. For some reason the CSS engine for Firefox (The component is called "Style" so it's absolutely impossible to find anything for it in a search engine) has massive .rs files generated by Jinja2 templates and somewhere it's not providing syntaxically valid Rust files anymore. I spend weeks swapping out various dependency versions of various parts to figure out why these files are invalid until I gave up.

Then a week later I decided to try again but this time Firefox 134.0 was released so I changed the package to use the newer release, this instantly fixed all the Rust related build issues... Of course I still had a few more build issues to figure out but in the end I was presented with this nice message from the Firefox build system:

This is probably referring to the 68 minute build time, but I'd like to think the Firefox build system is self aware and knows of my days of debugging.

Increasing the difficulty and building for ARM

So at this point I felt like things were going too easy, so one day I decided it would be funny to rebuild BodgeOS for ARMv8. Due to the way BodgeOS is built this means building the distribution from LFS again since I have no support for cross-building in this system, simply because that's a lot of complication to maintain in the build system for something that is only done very rarely.

To build the ARM version of BodgeOS I wanted to build it natively on an ARM system and it turns out that for some reason I have a lot of random ARM systems around :D, I decided to grab the LX2K board I have since it's probably the fastest ARM64 system I have here (I haven't really checked the benchmarks how it compares with the RK3588 systems) but it most certainly is the most sane one of all of them. This is the only ARM SystemReady certified hardware I have, I don't think I can explain it any better than the marketing blurb on the manufacturers website:

The SystemReady ES description from solid-run.com

This means I could just unpack the generic Alpine Linux ARM64 rootfs on a random 2.5" SSD I had around, plug it in and boot it. Even easier is that I already had done that before and the SSD I used last time was still in there so I continued on directly with building from the Alpine 3.15 system I had on there.

The LX2K board in a 2U rack case powered by a picoPSU

From here on out the steps are incredibly similar to bringing up the x86_64 version of BodgeOS. I used the automated JHALFS system to get my clean LFS system to start the whole system from, surprisingly building this on ARM worked practically perfectly the first time while nothing in JHALFS suggests it even is tested on ARM systems.

From there on I copied my temporary helper scripts from my LFS installation on my Thinkpad that I used to build the first iteration of x86_64 BodgeOS and hacked together abuild inside LFS to run the build scripts. The rest was actually a lot easier since I didn't have to figure out any of the packaging again, 99% of the package builds scripts simply worked on ARM64 or were simply missing some build dependency declarations.

What also suprised me is that this ARM board actually outperformed my relatively modern Thinkpad in build speed. The Thinkpad I've been using for this is an X280 with the i5-8250U CPU in it. What also helps for a few builds is that the LX2K board has twice amount of RAM in it currently than the Thinkpad has (16GB vs 8GB) which means I didn't have to pass in -j4 for a few builds that were very memory heavy. Running with only 4 threads is quite painful on this board anyway because it's speed comes from having 16 cores in it, and the cooling to actually run that at full speed.

The initial LFS build on the LX2K board

Continuing on

I have gotten to a minimal working installation on the LX2K for the ARM64 build and I've been packaging random tools I need on the x86_64 version now. To continue I should probably make an actual build system that does automated builds from the git repositories on both platforms instead of me just manually triggering abuild for every package and rsync'ing the file over to the mirror.

One of my goals other than Firefox was building Kicad but this is more painful than getting Firefox running due to the list of dependencies it requires. The main one being that I actually going to have to build Xorg to build Kicad.

BodgeOS pt.3: Graphical desktop

Martijn Braam — Sun, 29 Dec 2024 20:11:18 -0000

In the previous post I figured out all the internal weirdness of Linux booting to get BodgeOS running on actual hardware. The next goal was very clear: getting to a graphical environment. At the start of this month I had the goal set to running a web browser before 2024 ends but I've now slightly adjusted my goals down to being able to type this blog post in a terminal on my new OS.

So what does it take to get graphics in Linux? well the first component is very clear from experience: Mesa. This is the component that provides all the userspace components for the graphics drivers. I started with checking out both the LFS mesa build instructions and the Alpine and ArchLinux mesa packages. This is not a very nice package to build due to the large dependencies it has. This one project contains all the graphical hardware related code for any GPU Linux will run on and due to that it depends on several programming languages and compilers.

I have tried stripping down this package as much as possible: no X11 support, only intel graphics, only EGL for 3D acceleration, no extra components, no software rasterisation. This makes Mesa relatively easy to build. With only the i915 gallium driver for intel graphics I don't have to bring up any of the Vulkan, rust or llvm dependencies to get basic graphics.

The desktop

So I had to pick a graphical environment as goal to run. There are many choices for this and even more opinions on what the best one. I picked Sway here since it's a Wayland based environment so I don't have to go figure out all the X11 stuff. It's also very simple to build compared to something like Gnome or KDE Plasma. I guess there will be someone that has figured out that some random ancient window manager can be built with even less dependencies but this is the smallest one from the desktops I've actually used before :)

The dependency chain for Sway is pretty simple: wlroots abstracts away all the Wayland stuffs and makes it actually communicate with Mesa. Then it has some extra dependencies to render text and simple graphics to draw the bars and window decorations.

So I started figuring out the minimum dependencies for every component in the dependencies of wlroots and Sway and package all the things are that are needed. This included fixing up the Python udev bindings from my systemd package, packaging the Wayland protocols, a bunch of Xorg keyboard stuff because it seems like keymaps are still used from x* packages and finally seatd to provide a way to get a session for the desktop.

Sway

For Sway the dependencies got a lot more annoying to compile since it depends on Pango and Cairo for rendering and those build systems were just a massive pain to deal with. It seems like the higher you get in the stack of a Linux system the more bullshit is added to build systems to make things "easier". My particiular painpoint in the Sway dependencies is glib and gobject-introspection which is not sufficiently documented and seems to work on magic.

Font rendering

Along with Pango I also had to bring up the whole font system in Linux. This involves Pango, Cairo, HarfBuzz, Glib and several obscure libraries for font processing. These packages are fun because they contain circular dependencies so I had to build them a few extra times.

First attempt at booting

After getting the 41 packages built that are required to get to a very minimal Sway experience I generated a new rootfs and tried booting it on a laptop.

This started the hunt for optional dependencies that were not optional for my usecase. The first one was that seatd could not actually make a session for me because I had zero backends compiled in. This was a relatively simple fix of just enabling the builtin backend in seatd to get at least some session.

Next came the graphical stack issues...

It first took a bit of time figuring out which of these errors was the fatal one I should be looking at, it was not one of the red ones in this case. The important error here was the "iris: driver missing" line which is from Mesa. I had initially assumed that i915 was the hardware backend I needed for my laptops since it's the name I've always come across. Apparently my laptops are old but not old enough to require i915 graphics, instead I needed to enable the iris backend.

Enabling the flag for iris in Mesa is very simple, but the hard part is the additional dependencies this adds to Mesa. Iris requires libclc which is the library for OpenCL. This depends on SPIR-V and LLVM which means packaging another massive project.

LLVM was by far the biggest time sink for a single package I've had so far. This package takes absolutely forever to compile and I was building this on an x280 with 8GB ram. Since this laptop has 8 cores I have built everything with -j8 so far which works fine except for LLVM where I had to drop to -j4 to not run out of memory while building. I had the same issues with Clang as well and together I've spend 3 days waiting on either one of them to build to hit the next issue that needed slight adjustments in the flags.

With LLVM working I managed to build all the packages required for SPIR-V and libclc so I could finally build the iris backend in Mesa. Since I now had a few extra dependencies packaged I also could enable llvmpipe as software rasterizer and osmesa, the off-screen mesa renderer.

Sway starts

With my graphics drivers fixed I finally got Sway to run. This was a very unexciting start though since the only thing it actually rendered was a black screen with my cursor. To make this more annoying to debug it also did not allow me to switch back to a TTY with ctrl+alt+F{1,2,3,4} anymore to see any of the debug logs. This forced me to build the thing I had been postponing: openSSH.

By launching Sway through an SSH session I noticed that the first thing I was missing was the swaybg binary which apparently is a seperate package, that explains why by background was completely black. This was packaged and built in a few minutes which fixed 90% of the screen area. The next mystery was the missing bars.

Suprisingly with all the logging turned to max in Sway I still got no error message whatsoever about the bars not showing up. Even more suprising is that if I reloaded the config a few times there occiasionally were some graphical artifacts where the bars were supposed to be.

After trying a few things and guessing even more things I figured out why it did not show up: I have the entire font rendering pipeline working but I haven't packaged a single font.... So that was an easy fix.

To complete the minimal working environment I also built the foot package to have a terminal available that did not have too many extra dependencies to work.

BodgeOS Sway!

There's also several more low-level things I had to figure out on the way, like my installed system not having any locales available. This was quickly fixed by importing the locale-gen script from ArchLinux to generate the locales I need and fixing up my glibc package to put the locale files in the right location.

Branding

So now I have the bare minimum I could focus on more cleanup work and small features. One of those is making the default wallpaper for my distro. I ended up doing the same thing I always do when I get annoyed with Inkscape not doing what I want: Rendering the graphics directly using Python instead.

I made a small python script that uses Pillow to render a wallpaper at the requested resolution.

This was inspired by the KiCad PCB editor I had open. I thought it was topical since the distro name was also inspired by my electronics projects :D

Continuing on

So this most likely completes my BodgeOS project for this year. I'm now up to 238 APKBUILD scripts in the repository which build ~900 packages for the distribution.

I'll have to package a lot more probably for my next goal, which is getting Firefox to run. This includes some things I've been avoiding like figuring out how to bring up the rust ecosystem and packaging GTK. While the current packages might've been hard to figure out, the rust ecosystem seem to actively resist packaging efforts to make it even harder. Maybe I should get more of Python packaged first so I can use my own utilities for working with APKBUILD files.

Since this is also the last blog post of the year, happy new year everyone!

BodgeOS pt.2: Running on real hardware

Martijn Braam — Tue, 17 Dec 2024 00:30:04 -0000

In the previous part of this series I created a base Linux distribution from a running LFS system. This version only ran as a container which has several benefits that makes building the distribution a lot easier. For a simple container I didn't have to have:

A service manager (systemd)
Something to make it bootable on x86_64 systems (grub, syslinux, systemd-boot)
A kernel
An initramfs to get my filesystem mounted
File-system utilities since there's a folder instead of a filesystem.

A few of these are pretty easy to get running. I already have all the dependencies to build a kernel so I generated a kernel from the linux-lts package in Alpine Linux.

To make things easier for myself I just limited the distribution to run on UEFI x86_64 systems for now. This means I don't have to mess with grub ever again and I can just dump systemd-boot into my /boot folder to get a functional system. I had to build this anyway since I had to build systemd to have an init system for my distribution.

The Initramfs

The thing that took by far the longest is messing with the initramfs to make my test system boot. The initramfs generator is certainly one of the parts that have the most distribution-specific "flavor". Everyone invents it's own solution for it like mkinitcpio for ArchLinux and mkinitfs for Alpine and initramfs-tools for Debian as a few examples.

I did the only logical thing and reinvented the wheel here. I'm even planning to reinvent it even further! Like the above solutions my current initramfs generator is a collection of shell scripts. The initramfs is a pretty simple system after all: it has to load some kernel modules, find the rootfs, mount it and then execute the init in the real system.

For a very minimal system the only required thing is the busybox binary, it provides the shell script interpreter required to run the messy shell script that brings up the system and also provides all the base utilities. Due to my previous experiences with BusyBox modprobe in postmarketOS I decided to also move the real modprobe binary in the initramfs to have things loading correctly. To complete it I also added blkid instead of relying on the BusyBox implementation here to have support for partition labels in udev so no custom partition-label-searching code is required.

Getting binaries in the initramfs is super easy. The process for generating an initramfs is:

Create an empty working directory
Move in the files you need into the working directory from the regular rootfs like /usr/bin/busybox > /tmp/initfs-build/usr/bin/busybox
Add in a script that functions as pid 1 in the initramfs and starts execution of the whole system
Run the cpio command against the /tmp/initfs-build directory to create an archive of this temporary rootfs and run that through gzip to generate initramfs.gz

Step 2 is fairly simple since I just need to copy the binaries from the host system, but those binaries also have dependencies that need to be copied to make the executable actually work. Normally this is handled by the lddtree utility but I didn't feel like packaging that. It is a shell script that does a complicated task which is never a good thing and it depends on python and calling various ELF binary debugging utilities.

Instead of using lddtree I brought up Hare on my distribution and wrote a replacement utility for it called bindeps. This is just a single binary that loads the ELF file(s) and spits out the dependencies without calling any other tools. This is significantly faster than the performance overhead of lddtree which was always the slowest part of generating the initramfs for postmarketOS.

The output format is also optimized to be easily parse-able in the mkinitfs shellscript.

$ lddtree /usr/sbin/blkid /usr/sbin/modprobe 
/usr/sbin/blkid (interpreter => /lib64/ld-linux-x86-64.so.2)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
/usr/sbin/modprobe (interpreter => /lib64/ld-linux-x86-64.so.2)
    libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5
    libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6

$ bindeps /usr/bin/blkid /usr/bin/modprobe 
/usr/lib/ld-linux-x86-64.so.2
/usr/lib/libblkid.so.1.1.0
/usr/lib/libc.so.6
/usr/lib/libzstd.so.1.5.6
/usr/lib/liblzma.so.5.6.3
/usr/lib/libz.so.1.3.1
/usr/lib/libcrypto.so.3

The bindeps utility seems to be roughly 100x faster in the few testcases I've used it in and it outputs in a format that needs no further string-mangling to be used in a shell script. In BodgeOS mkinitfs it's used like this:

binaries="/bin/modprobe /bin/busybox /bin/blkid"
for bin in $binaries ; do
    install -Dm0755 $bin $workdir/$bin
done

bindeps $binaries | while read lib ; do
    install -Dm0755 $lib $workdir/$lib
done

The next part is the kernel modules. Kernel modules are also ELF binaries just like the binaries I just copied over but they sadly don't contain any dependency metadata. This metadata is stored in a seperate file called modules.dep that has to be parsed seperately. I did not bother with this and copied the solution from the initramfs generator example from LFS and just copy hardcoded folders of modules into the initramfs and hope it works.

The file format for modules.dep is trivial so I really want to just integrate support for that into bindeps in the future.

Debugging the boot

It's suprisingly painful to debug a non-booting Linux system that fails in the initramfs. I wasted several hours figuring out why the kernel threw errors at the spot the initramfs should start executing which ended up being an issue with the /sbin/init file had the wrong shebang line at the start so it was not loadable. The kernel has no proper error message that conveys any of this.

After I got the initramfs to actually load and start a lot of time was wasted on executables missing the interperter module. In the example above this is the /lib64/ld-linux-x86-64.so.2 line. The issue here ended up that I was just missing the /lib64 symlink in my initramfs. This was very hard to debug in a system without debug utilities because nothing could execute.

After all that I spend even more time figuring out why I had no kernel log lines on my screen. After much annoyance this turned out to be missing options in the kernel config for the linux-lts kernel config I took from Alpine Linux. So instead of fixing that I took the kernel config from ArchLinux and rebuild the linux-lts package. This fixed my kernel log output issue but also added a new one... The keyboard in my laptop wasn't working in the initramfs.

I never did figure out which module I was missing for that because I fixed the rest of the initramfs script instead so it just continues on to the real rootfs where all the modules are available.

After all that I did manage to get to a login prompt though!

Cleaning up

After booting up this I realized it would be really handy if I actually had a /etc/passwd file in my system and some more of the bare essentials so I could actually log in and use the system.

This mainly involved adding missing dependencies in packages and packaging a few more files in /etc to make it a functional system. After the first boot the journal had a neat list of system binaries from util-linux that systemd depends on but not explicitly, so I added those dependencies to my systemd packaging.

I also had to fix the issue that my newly-installed system did not trust the BodgeOS repository, I was missing the keys package that installs the repository key in /etc/apk/keys for me. In this process I noticed that the key I built the system with was called `-randomdigits.pub` instead of being prefixed with a name. This is pretty annoying because this name is embedded in all the compiled packages and I didn't want to ship a file with that name in my keys package.

There seemed to be a nifty solution though: the abuild-sign tool appends a key to a tar archive, which is normally used to sign the APKINDEX.tar.gz file that contains the package list in the repository. I decided to run abuild-sign *.apk in my main repository after adjusting the abuild signing settings with a correct key name.

Apparently this breaks .apk files and after inspection they now had two keys in them and neither my development LFS install and my test BodgeOS install wanted to have anything to do with the packages anymore.

In the end I had to throw away my built packages and rebuild everything again from the APKBUILD files I had. Luckily this distribution is not that big yet so a full rebuild only took about 2.5x the duration of Dark Side of the Moon.

Next steps

Now I have a basic system that can boot I continued with packaging more libraries and utilites you'd expect in a regular Linux distribution. One of the things I thought would be very neat is having curl available, but that has a suprising amount of dependencies. Some tools like git will be useful too before I can call this system self-hosting.

I also want to remove all the shell scripts from the initramfs generation. None of the tasks in the initramfs are really BodgeOS specific and most of the complications and bugs in this initramfs implementation (and the one in postmarketOS) is because the utilities it depends on are not really intended to do this stuff and system bootup just has a lot of race conditions shell scripts are just not great at handling.

My current plan to fix that is to just replace the entire initramfs with a single statically linked binary. All this logic is way neater to implement in a good programming language.

Conjuring a Linux distribution out of thin air

Martijn Braam — Sat, 07 Dec 2024 23:08:27 -0000

I decided I had to get something with slightly more CPU power than my Thinkpad x230 for a few tasks so I got a refurbished x280, aside from the worse keyboard the laptop is pretty nice and light. It shipped with Windows of course so the first thing I did is to install Ubuntu on the thing to wipe that off and verify all the hardware is working decently.

I was wondering if I should leave Ubuntu on the thing, it works pretty well and it's still possible to get rid of all the Snap stuff, it's not my main machine anyway. The issue I ran into quickly though is some software is pretty outdated, like I don't want to use Kicad 7 anymore...

Picking distros once again

You'd think after using Linux for decades I would know what distro I'd put on a new machine. All the options I could think off though had annoying trade-offs I didn't want to deal with once again.

The three main distributions I have running on hardware I manage is Alpine Linux, Archlinux and Debian. I like Alpine a lot but it is quite annoying when you deal with closed-source software. Since this is my go-to laptop to take with me to outages and repairs then I need it to handle random software thrown on it relatively easily.

ArchLinux satisfies that requirement pretty well but my main issue with it is pacman. If you don't religiously run upgrades every hour on the thing it will just break because for some reason key management is not figured out yet there. The installation is also quite big usually due to packages not being split.

Debian fixes the stability issues but comes with the trade off that software is usually much much older, this also leaks into Ubuntu that's running on the laptop now. It is also internally a lot more complicated due to the way it automatically sets up stuff while installing which I don't usually need.

There is another solution though. Just build my own!

Artisanal home-grown Linux

Creating a new Linux distribution is one of those things that sounds much harder than it actually is. I just haven't done it before. I did build a small Debian derivative distro before just to avoid re-doing all the config for all machines but that's just adding an extra repository to an existing distribution. Of course I've also worked extensively on postmarketOS and while the scope of that is a lot larger it still is only a repository with additions on Alpine Linux.

Some of you might be familiar with this graphic of Linux distributions:

There are many many many derivative distributions here, way less distributions that are built up from nothing. And that's exactly the part that I want to figure out. How do I make a distribution from scratch?

I have once, a long time ago, build a working Linux machine from sources using the great Linux From Scratch project. I would recommend that anyone that's really into Linux internals do that at least once. Just like Archlinux learns you how the distribution installer works (before they added an installer) the LFS book will learn you how you bootstrap a separate userspace from another Linux distribution and doing the gcc/glibc dependency loop build.

So my plan for the distribution is: create a super barebones system using the LFS book, package up that installation using abuild and apk from Alpine Linux. This way I can basically make my own systemd/glibc distribution that is mostly like Archlinux but uses the packaging tools and methodology from Alpine Linux.

The bootstrap build

So to make my distribution I first have to build the Linux installation that will build the packages for the distribution. To get through this part relatively quickly I used the automated LFS installer called ALFS. This basically does all the steps from the LFS book but very quickly. My intention is to replace all of these packages anyway so this part was not super important. It does all the required setup though to validate the GCC I'm using to build my distribution is sane and tested.

In the ALFS installer I picked the systemd option since I didn't want to deal with openrc again and ended up with a nice functional rootfs to work in. I immediately encountered a few things that were critical and missing though. There was no wget or curl. I fixed this by grabbing the Busybox binary from my host Ubuntu distribution and putting that inside with only a wget symlink to it.

This wget was not functional in my chroot though since it could not connect to https servers. Annoyingly all solutions you'd find for these errors is passsing --no-check-certificates to wget to get your files which is unacceptable when building a distribution. After a lot of debugging with openssl configs I ended up copying over all the certificates from the host Ubuntu system again and pointed wget to the certificate bundle with the wgetrc file.

The very next thing I needed is abuild. This is the tool from Alpine Linux that's used to build the packages. Luckily it's a few small C programs and a large shell script so it's very easy to install in the temporary system. I also added apk.static to the system to be able to install the built packages.

Building packages

So now I have my temporary system running I could start writing APKBUILD files for all the packages in my LFS system. I started with the very simplest one of course that only provides /etc/services and /etc/protocols. No compiling and dependencies needed.

For this package made the script that built the current APKBUILD using the abuild subcommands and generated a neat local repository so apk could install the files. So I ran that and now /etc/services and /etc/protocols are now exactly the same files but managed by apk and in a package.

The reason I had to use the subcommands for abuild to run seperate stages instead of just running abuild itself to do the whole things is because one of the first steps is installing the build dependencies. In this case I'm in the weird setup where I have all the build dependencies installed through LFS but apk doesn't know about that so I simply skip that step.

And that's how I re-build a lot of the LFS packages once again, but this time through my half broken abuild installation. One of the things that abuild was not happy about is the lack of the scanelf utility which it uses after building a package to check which .so files the binaries in the package depend on. Due to this a lot of dependencies between packages are simply missing. The scanelf utility has enough build dependencies though that I could not have that as the first few packages so most of the packages are broken in this stage.

When I finally built and installed scanelf I ran into another issue. The packages I build after this failed at the last step because scanelf found the .so files required for the package but the package metadata for all the packages I made before it lack this information about the included .so files apparently. At this point I had to build all those packages for a fourth time (twice in LFS and now twice in abuild) to make dependencies here work.

After getting through practically everything in the base system I ended up with around 333 packages in my local repository.

Most of these APKBUILD files are a combination of the metadata header copied from the Alpine Linux ABUILD so I have a neat description and the correct license data. And then the build steps and flags from LFS and sometimes the install step from Archlinux.

This means that for example the xz build steps in LFS are:

$ ./configure --prefix=/usr    \
              --disable-static \
              --docdir=/usr/share/doc/xz-5.6.3
$ make
$ make check
$ make install

And that combined with the Alpine Linux metadata headers and adjustments for packaging becomes:

pkgname=xz
pkgver=5.6.3
pkgrel=0
pkgdesc="Library and CLI tools for XZ and LZMA compressed files"
url="https://tukaani.org/xz/"
arch="all"
license="GPL-2.0-or-later AND 0BSD AND Public-Domain AND LGPL-2.1-or-later"
subpackages="$pkgname-doc $pkgname-libs $pkgname-dev"
source="https://github.com//tukaani-project/xz/releases/download/v$pkgver/xz-$pkgver.tar.xz"

build() {
        ./configure \
                --prefix=/usr \
                --disable-static \
                --docdir=/usr/share/doc/xz-$pkgver
        make
}

check() {
        make check
}

package() {
        make DESTDIR="$pkgdir" install
}

Getting the first install to run

Now the hard part, finding all the issues that prevent this new installation from starting. The first thing I tried to use the apk.static on my host system to generate a new chroot from the repository I created, just like you'd install an Alpine chroot.

Unfortunately this did not work and I had to fork apk-tools and make my own adjusted version. This is mainly because apk-tools hardcodes paths in it which conflict with my usrmerge setup. So I now have an apk.static build from my fork that does not try to create /var/ for the database before the baselayout package can create the actual filesystem hierarchy with a symlink at that spot.

With that fixed apk.static would be able to finish creating an installation from my repository, but I could not chroot into it for some reason. All the binaries are broken and return "Not found" when trying to execute them. I managed to actually enter the chroot by throwing my trusty busybox binary in there but did not get any more information out of that installation.

After a bunch of testing, debugging, and more testing, I found out the reason was that I don't have /lib64 in my installation. It seems like it's required specifically because x86_64 binaries specify /lib64/ld-linux-x86-64.so.2 as loader. The fix for that is quite easy by just having the glibc package place a symlink at that spot to the real ld-linux.so.

Beyond the first run

There's a lot of things that need to be fixed up to be a good distribution. All the packages will need to be rebuild again from the distribution installled from these first generation packages to leave behind the last parts of LFS in there. There's also the system setup that needs to happen to make it bootable and maintainable. For example things like the keys package that install and update the repository keys and adding the logo to neofetch (after packaging neofetch).

I've also rsync'd the repository for the distribution to a webserver so it can actually be added to installations. I've been using this now to create test chroots using my locally build patched apk-tools.

The repository itself also needs quite a bit of work, gcc shouldn't've been pulled in here for these packages and a bunch of the large packages need to be split up to remove the uncommon parts. Currently the glibc package is already 10x larger than a base Alpine Linux installation, luckily apk is a very fast package manager and everything still installs super quickly.

So why?

It doesn't really make sense to do this. I wanted to have done it anyway because that's how you learn. This took about 3 days of fiddling with build scripts between other work since it's mostly waiting on builds to finish.

At the very least this distribution created this blog post, which is surprisingly one of the very few pieces of information available for bootstrapping a distribution.

Even with this tiny base of packages this is already a quite usable OS since I have a kernel, a service manager and python. All you need to build some embedded stuff if this had an ARM64 port bootstrapped or something. For desktop work there's still a mountain of work to be done to package everything required to launch Firefox, getting the basic graphics stack up and running should be relatively straightforward with bringing up Sway with its dependencies.

In the end it probably would've been easier to just add a ppa for Kicad to my Ubuntu installation :)