BrixIT Blog

Building a browser game based on KiCad

Martijn Braam — Thu, 03 Apr 2025 16:56:41 -0000

I've been making boards in KiCad for a while now. I really enjoy figuring out how to route all the components in the PCB editor, especially the weird "hard" things like differential high speed signals. I'm probably not very good at it but so far the margins for the stuff I'm designing is wide enough that it works anyway.

Every time I'm working on this I feel like this would make a neat puzzle game, but I don't feel like building an entire PCB editor in a game engine to make this concept work. So I took a page from the Minecraft book and turned it into cubes. Making the problem discrete into a grid makes it way easier to reason about and a more logical game.

To make the whole thing easy to build I decided to go with a simple browser game. No need to deal with much platform support there and I don't have to either use one of the big game engines or write one myself. I debated for a few seconds whether I should go with TypeScript or just vanilla JS. In the end I wrote the entire thing in plain javascript.

The game

The whole game area is drawn as a element and the javascript code generates the DOM elements for a few of the user interface elements.

The goal of the game is to connect all the source blocks to the sink blocks of the same color. The source blocks are the arrows pointing up and the sink blocks are the arrows pointing down. They probably need to have less similar icons but that's where my creativity ended.

The connection for every color needs to be exactly the same length. In the solution above I've already failed this since the orange connection reached the end already at 11 ticks before the rest of the signals are even halfway. Reaching the sink at the same time is the bare minimum to have a valid solution, to get a higher score though the signal integrity needs to be as high as possible.

Signal integrity is highest when the signals are traveling together like in the green/blue line. Every tick every signal is checked for neighbors and if it has a neigboring signal it will increase the quality, if it has two neighboring signals it will increase the quality even more. This is balanced in a way that having no neighbors at all for the signal will quickly drop all the points of the connection so you can only do that for small segments.

This is a solution to the initial level I made, basically the bare minimum to have a valid solution and it generates 2482 points. This is also the initial version I released on mastodon some days ago so a few people have tried to beat that score, and it has been done easily. The current high score for this level is now 4000 points by @viciouss.

The current dev version of the game can be played at https://brixitcdn.net/game/

Internals

The internals for the game are not terribly exciting. It's mostly full of hooking javascript events to gamecode like this:

const self = this;
this.canvas.addEventListener('mouseenter', function () {
    self.hover = true;
    self.dirty = true;
});
this.canvas.addEventListener('mouseleave', function () {
    self.hover = false;
    self.dirty = true;
});
this.canvas.addEventListener('mousemove', function (event) {
    const bounds = self.canvas.getBoundingClientRect();
    self.hover = true;
    self.mouse.x = event.clientX - bounds.left;
    self.mouse.y = event.clientY - bounds.top;
    self.dirty = true;
    if (self.mouseHeld) {
        let mouse = [Math.floor(self.mouse.x / 16), Math.floor(self.mouse.y / 16)];
        if (mouse[0] !== self.lastMouse[0] || mouse[1] !== self.lastMouse[1]) {
            self.lastMouse = mouse;
            self.click(mouse[0], mouse[1]);
        }
    }
});

The game uses a dirty flag to repaint the game only when there's actual changes, because on the naive requestAnimationFrame() approach the thing still pins an entire core just drawing a basic grid for some reason.

The entire state of the game is stored in a simple two dimensional array where the contents are either null or an TCBlock() object that stores the color, type and state of that pixel. Most the code in the game deals with implementing drawing lines inside this array.

When the verify button is pressed a loop is started where every 200ms the entire playfield is simulated. On the initial step all the source blocks will get a "charge" and then every step that charge gets propagated in all cardinal directions if a block of the same color exists in that location. The game tracks every pixel that has had a charge before so it is not possible to create an infinite loop with this mechanic. The simulation terminates when either all charges have been absorbed by a sink block or a simulation step doesn't change the state of the game anymore.

The most interesting part is the loading/saving system. I wanted to have the entire level stored in the URL so it's easy to share levels made in the game. This way I also don't need any server side code and the entire thing can just be static HTML, CSS and javascript.

My initial implementation of saving the game was exporting the state array to JSON and then base64 encoding that to an URL fragment. It turns out that this already consumes 400 characters for just the initial level with 8 blocks on it set. The size explodes when there's actually lines drawn in the level.

My first fix for this was adding brotli.js into the codebase and compressing the json blob before encoding it as base64. This helps a lot and brings the size down to 80 characters for the initial state, but it's still way too large. The solution is obviously to use some actual engineering instead of just throwing things into JSON.stringify here.

There are only four numbers that need to be saved for every block that is set:

The X and Y coordinate
The color, every block has a hex string as color.
The block type: "source", "sink", "wire", "wall".

For the color I added a lookup table to the game and defined a series of colors so I can just store an index into that array. For the block type I simply stored a single character for the types: "S", "s", "w" and "o" for the four block types. The coordinates are also never larger than 256 in any direction so this means I can pack every block into 4 bytes. With this improvement my initial state was down to 32 bytes and with my test solution it was around 200 bytes.

I decided to throw brotli against this problem again to fix this. The initial state of 32 bytes gets compressed with brotli to... 36 bytes... so that doesn't help at all. But the 200 byte field gets compressed to ~100 bytes which is a nice improvement. It can still be better though...

I don't actually need that many colors, I probably don't need more than 4 types in total and the coordinates are nowhere near 256. I slightly shrunk the play area from 640x480 to 512x480 so the largest coorinate in my 16x16 grid becomes 32. This means I can pack the X and Y coorinate together in just 10 bits. If I limit the other values to 8 maximum I can store that in 6 bits together giving me the state for a single block packed into 2 bytes instead of 4, halving the size of the savegames again.

This is what's now actually in use in the demo to save the initial states and what makes the "export" button work that puts your entire solution in the URL.

const buf = new ArrayBuffer(result.length * 2);
const bufView = new Uint8Array(buf);
result.forEach((item, i) => {
    bufView[(i * 2)] = (item[0] & 0x1F) | (item[2] << 5);
    bufView[(i * 2) + 1] = (item[1] & 0x1F) | (item[3] << 5);
});
console.log("Packed:", buf.byteLength);
const compressed = brotli.compressArray(bufView, 11);
console.log("Compressed:", compressed.length);
window.location.hash = btoa(String.fromCharCode.apply(null, compressed));

The result array already contains the data compressed into the [[x,y,colorIdx,typeIdx]] format so that's run through the byte packing code, the compressor and finally the base64 encoder to produce the savegame.

Conclusion

It was suprisingly easy to make all of this, I have of course already used all the seperate components I've used here before so that saves a lot of research. In total I probably spend less than a day to implement all of this, including writing this blog post.

The game itself seems reasonably fun. It can probably be a bit more difficult and the level editing can certainly be a lot better.

The mechanic that is stolen from PCB design in KiCad seems to work pretty well and the scoring mechanism certainly points you into a solution that closely mirrors the advice you get for routing differential signals: Length match the pairs, add timing wiggles close to the source of the features that change the timing to make the differential signalling work the best.

You can play the game at https://brixitcdn.net/game/ and the level editor is available at https://brixitcdn.net/game/#editor

Source code for this is at https://git.sr.ht/~martijnbraam/TimeCode

Megapixels 2.0: Small fixes and GTK breakage

Martijn Braam — Fri, 28 Mar 2025 20:52:26 -0000

Looking back it seems like making an alpha release of Megapixels 2.0 was a great choice. The various components that make up Megapixels have been through the packaging steps at Mobian for example which brought some light to issues that would improve the packaging. A lot more eyes have hit the code in this release than the random stuff thrown to git and I'm really happy I've received a lot of improvements from a bunch of developers.

When I started working on Megapixels it was my first C codebase, when I started working on libmegapixels/libdng those were the first times figuring out how do do C libraries. In the years I've been working on this codebase I've leared a whole lot, mostly around cleaner code and seeing merge requests that fix up minor issues in the code are a great reference for figuring out what the idiomatic way is of writing specific pieces of code.

Of course quite a few of these are minor memory safety violations and there has been a few un-free()'d resources around within Megapixels, but in the end running free on a few bytes before quitting the app or letting Linux release that memory doesn't make that much of a difference, most of the effort has been going through making sure the main image processing loop doesn't leak memory anywhere. If that part leaks memory then it will starts adding up really fast :)

GTK throwing a wrench into the development process

Of course Linux development can't ever run smoothly... there's always something new and exciting to break everything.

In the case of Megapixels it has been the NGL backend for GTK4. The 4.17 release made in Februari dropped the GL backend in favor of the NGL and Vulkan renderers. Which is great if you're on the latest and greatest Macbooks.

The issue is that GTK now also dropped for GLES 2.0 which means that a lot of older devices are no longer GPU accelerated in GTK4. For Megapixels it's an even bigger issue since the debayering depends on GPU acceleration so it won't run at all if GTK4 doesn't have an OpenGL context anymore.

This hardware doesn't even have to be terribly old. For example here's some of the hardware supported by Megapixels 2.0:

Device	GPU	OpenGL	OpenGL ES
PinePhone	Mali 400	OpenGL 2.0	OpenGL ES 2.1
Samsung Galaxy SIII	Mali 400	OpenGL 2.0	OpenGL ES 2.1
PinePhone Pro	Mali T860	OpenGL 3.1	OpenGL ES 3.1
Librem 5	Vivante GC7000Lite	OpenGL 2.0	OpenGL ES 2.x

What they all have in common is that they don't really support the latest and greatest OpenGL versions. It's not very easy to get any hard docs on what OpenGL requirements GTK has now, but it seems like it's at least OpenGL 3.3 and there's still references to OpenGL ES 3.0 in the codebase. Which means that for the devices I've been targetting for Megapixels the support simply isn't there anymore.

So far there's been a workaround for this by putting Megapixels in a flatpak with a runtime that doesn't have the latest version of GTK4 in it. This is obviously not a long term solution but at least there's some workaround for now. Many thanks to Andrey Skvortsov for creating a flatpak package for Megapixels.

There's a build available now on my flatpak repository, you should be able to get your favorite graphical packagemanager frontend to install it with this link: https://flatpak.brixit.nl/megapixels2.flatpakref

The future

I'm not entirely sure what a good solution for this mess is. My current feeling is that it's best to not rely on GTK4 anymore because even if somehow a workaround is figured out to make this work, there's always the next GTK issue coming up.

Switching to another framework also isn't great, especially since Megapixels just has been through rewrite hell already, it'd be 2026 before we'd have a working camera app again. In theory it should be slightly easier now to make megapixels-qt now with libmegapixels but that only abstracts the device usage, the main magic of Megapixels is the threading mess and OpenGL debayer code that gives it the realtime performance for photography and that would have to be recreated on another platform.

Maybe someone has a great idea or a solution for this, I'd love to hear it.

After-FOSDEM videobox updates

Martijn Braam — Mon, 10 Mar 2025 02:30:20 -0000

The FOSDEM video capture box is a custom device for doing all the in-room stuff for the live-streams of the event. It contains a Radxa x4 SBC, a digital audio mixer, an HDMI capture card, some USB chargers and a network switch. Every room that is livestreamed has two of these boxes, one in front that captures the audio and the HDMI signal for the projector. The other box is in the back of the room hooked up to the camera.

Wiring diagram for the box internals

The network switch is the managed network switch I've written about before, but instead of being hooked up to an ARM computer and using the DSA system in Linux to make it appear native it's mostly unmanaged here. There is connectivity between the switch and the radxa only for some basic monitoring.

The audio mixer here is also the digital audio mixer design I've written about before. It's handling the audio input from the wireless microphones used at FOSDEM and it sends out the audio to the XLR inputs on the camera used for recording.

The power board is another custom designed board by Dexter, it takes 12V from the wall and provides the power for all the internal components, it also provides 4 USB charging ports in the front which are electrically isolated from the rest of the system to have the wireless receivers plugged into the chargers without creating ground loops. This boards also has a few auxilary functions like controlling the fans in the box.

A talk has also been given about this hardware at FOSDEM itself (at the end of the event after we were sure everything was working smoothly) and is watchable at https://fosdem.org/2025/schedule/event/fosdem-2025-6832-fosdem-videobox-2025/.

To add a bit of irony to the situation this talk about fixing audio quality issues at FOSDEM has bad audio because we decided at the last moment to plug in a spare wireless mic into the system to have mic handoffs and had the gain of the receiver set higher than the box volume could be adjusted.

Next up after FOSDEM

The hardware worked great during the event and now the pressure of having it work at all is gone some extra time has been spend on fixing up issues in the design.

One of the more annoying things about the box is that you can hear it in the recordings and in the rooms because the fans are too loud. This is a combination of various design tradeoffs. The first is that the fans are hooked up to an EMC2305 fan controller chip. This chip is also used in a few computers as the internal fan controller. Sadly the automatic control functionality of this chip simply doesn't work (and we checked the Linux kernel driver for it, it also doesn't use the automatic functions). In this design it's only used to read the tachometer signals of the fans and output the PWM signal for the fans.

The second issue is that the fans don't have a PWM input so the fan control is done by applying the PWM signal to the power input. Due to this control mechanic and the limitation that the PWM of the fan controller chip has only 8 bit resolution the usable range of control for the fan is reduced to ~6 levels where there's measureable difference in the fan speed, and the only real levels there are off, slow and 4 levels of way too loud.

The firmware that ran at FOSDEM had a control loop on the fans that measured the fan RPM and adjusted the PWM signal to keep the fan speed at a preset PWM which due to the limited control resolution just meant the system occasionally started oscillating and the fan speed was ramped up a lot due to the anti-stall mechanism.

This has now been replaced by a far simpler control loop that only allows setting "off, low and fast" so it matches reality and not use any tachometer feedback to control this but instead only use that for stall detection.

Audio firmware changes

The audio mixer at fosdem did not use the USB audio path at all, the audio signals were mixed and then the analog XLR outputs were used to send the audio to the speakers in the room if available, and to the audio input of the camera so the audio is embedded into the HDMI signal that is captured. This is mainly to maintain sync between audio and video through the system. The firmware running at FOSDEM had stability issues when using the USB soundcard functionality of the box which luckily wasn't needed.

But these boxes are not only used at FOSDEM, it would be a shame to make custom hardware that is only used 2 days per year. The boxes are loaned out to other conferences to try it out and one of those conferences is FOSSASIA in Bankok. On this conference there is no HDMI camera used for the capture in the rear of the room but instead it's using a webcam, and webcams don't have inputs for audio input. After figuring out that it was the patches to run the Teensy audio board at 48KHz sampling rate instead of 44.1KHz it was decided that those patches should just be ripped out. So far the firmware has been stable with these changes.

VLAN support for the network switch

The network switch in the box has performed perfectly during the FOSDEM event but none of the fancy features of the switch chip were used. The main thing that was done was reading out the port status so it can be shown on the display on the front of the box.

For more flexibility it would be nice to give this more managed switch functions, one of the main things being VLAN support. The basic functionality for the switch is very easy since you just supply the chip with power and it starts switching, figuring out how to control the chip itself is way harder. There's not nearly enough public information available on programming the RTL8367S, the datasheet only covers the registers for port status that I've been using.

Using a lot of puzzling and reverse engineering I've figured out how to load a VLAN config into the chip. This is mainly based around bits of info available from other realtek switch chips and some bits of leaked SDK code for similar chips. This means there's now some extra APIs available in the rp2040 code that controls the switch in the box for setting up a static vlan configuration on boot:

// Set the MemberConfig for all ports to 0 and enable tag handling on the ports
nsw_vlan_init();

// Enable the VLAN processing in the switch
nsw_config_vlans(true);

// Create a VLAN table entry for VLAN 10 with port 1, 2 and 3 as members
// Port 2 and 3 will be untagged on egress
nsw_vlan_cfg_t vlan10 = {
		.vid = 10,
		.mbr = BIT(0) | BIT(1) | BIT(2),
		.untag = BIT(1) | BIT(2),
};
nsw_vlan_set(&vlan10);

// Create a MemberConfig row to ingress untagged traffic on port 2 and 3
// Use MemberConfig index 1 and vlan 10
nsw_mc_set(1, 10, BIT(1) | BIT(2));

// Point the two untagged ports to MemberConfig index 1. This needs to
// happen _after_ setting the ports as member in the MemberConfig itself
nsw_port_set_mc(1, 1);
nsw_port_set_mc(1, 2);

// Set the untagged ports to drop traffic with a vlan tag set
nsw_port_vlan_filtering(1, PORT_ACCEPT_UNTAGGED_ONLY);
nsw_port_vlan_filtering(2, PORT_ACCEPT_UNTAGGED_ONLY);

// Drop untagged traffic from the trunk port
nsw_port_vlan_filtering(2, PORT_ACCEPT_TAGGED_ONLY);

There's also half an implementation for reading port statistics. The switch provides the standard port statistics MIB block, I just need to figure out the final bits for mapping the MIB numbers to the right address in memory.

The current port status information that's read out of the PHY registers of the switch are equivalent of the data in the ethtool eth0 command, the MIB registers contain the information presented in the ethtool -S eth0 command. This would great to produce some pretty Grafana charts.

Hardware improvements

The audio board performed decently given the constraints of the FOSDEM environment and the hardware plugged into it, but I'd like it to be a bit more flexible. The XLR inputs right now are designed to accept -10dBu consumer line level signals. Plugging in a cheap microphone doesn't really work since the board does not have enough gain available to crank up the levels in hardware and is also too noisy to just boost the levels in software. It is also possible to clip the inputs using the AVX wireless systems used at FOSDEM which also isn't ideal.

The reason for the small dynamic range of the inputs is for the FOSDEM constraints the simplest and most reliable solution was to make an analog frontend that works from the 5V usb power directly and is tweaked line level signals. The initial version of the board had switching regulators on it to produce +12V and -12V power rails for the analog inputs and the filtering for these inputs was not enough. To make some more HiFi inputs I'm now working on some prototypes of better power supplies and have some new designs around better ADCs and better analog frontends.

Power supply 2.0 test board

This design of the power supply reverts back to the original switching regulator but now uses the filtering circuitry used with the 5V powered analog inputs of the FOSDEM hardware to hopefully produce a less noisy signal. The redesign for audio input also replaces the integrated TI codec which has a lot of issues with a dedicated ADC and DAC chip, one of the major improvements here is that the inputs of these ADC chips is already balanced so the board doesn't have any unbalanced audio signals on it anymore.

One of the main things that needs figuring out is how to have cheap software control of the input gain, for using cheap wired microphones it would be needed to boost the signal levels a lot, somewhere between 40 and 60 dB of extra gain without too much noise added. There are easy solutions for this, but the easy solutions are never cheap, especially considering you need a copy of the circuit for every input channel.

Having some better input circuitry would make the FOSS digital audio mixer a lot more widely usable, there's many situations where I really would've used a small audio mixer to do some simple routing.

BodgeOS pt.4: A working browser

Martijn Braam — Sat, 25 Jan 2025 11:35:26 -0000

After the previous post about bringing up Sway I spend a bit of time packaging random system components you'd expect for a desktop system. Mainly building GTK so I can have some actual applications running. This is mainly pretty relaxing work. Find a component to build, figure out the dependencies, package them one by one by figuring out the required commands and making a tiny build script.

Occasionally the zen of packaging gets disturbed by having to figure out why some build system does something weird, like really wanting to put files in /usr/lib64 instead of /usr/lib while in the previous build it did not do this.

It turns out this is because some build systems perform auto detection of the host system to decide things like "what folder is a good place for the libraries". In this case it was cmake that has autoconfig behavior for this if you don't explicitly define that libs go in the lib folder. At some point installing a broken rebuild of glibc on the host system had changed lib64 back from a symlink to a regular folder again and all builds after that were broken.

The path to Firefox is paved with many dependencies

For my goal of running Firefox I have to deal with several dependencies:

GTK 3 and the components to make that work like Pango and Cairo.
A whole assortment of audio, video and image codecs, these can probably be skipped by manipulating the Firefox buildsystem, but it is probably easier to package all of these instead.
Nodejs is also a dependency for building Firefox since the javascript engine in Firefox itself can't be used to run the javascript components in the build system.
WASI for having WebAssembly support.
Rust for large chunks of the Firefox codebase.
Extra libraries like the Netscape Portable Runtime (nspr) and the libraries for other system features like Alsa and Pulse.

Luckily parts of this have already been packaged as part of getting my Sway desktop working. Many dependencies for GTK 3, WASI and Rust are already packaged for either Mesa or Swaybar, but I had to finally figure out how to bring up Rust in my distro.

This is quite hard in theory because rustc is written in Rust so I need older versions of Rust to build the current version of Rust, and older versions for that again and again until you reach the point where rustc is written in C again. Instead of doing all that I opted to go for the easy road. I curl | sh'ed a functional Rust compiler into my host system with the instructions for rustup.rs and then used that compiler to build my packaged Rust. This was all surprisingly smooth and easy.

Getting NodeJS functional was a bit more painful. This is mainly because it takes forever to build which means it takes forever to fix things while packaging. The same issue with the Firefox build itself actually. The worst thing I've encountered while packaging things is things that take a lot of time to build and don't have the configure script check all the dependencies it needs. I might have been annoyed with autoconf wasting a lot of time at the start of every build of every small package with checks that seem to be completely useless, but the time wasted there does not compare at all to the time wasted in building Firefox and having it crash an hour into the build complaining it's missing a dependency.

Desktop audio

One of the dependencies for Firefox is also the various audio libraries, which in turn means I have to bring up audio on my distro. I spend a moment to figure out wether it would be a PulseAudio or PipeWire system but ended up just picking PipeWire. Most of the issues I've seen with PipeWire seem to stem from bodged migrations from Pulse systems anyway and that is not an issue I would have to deal with. Unfortunately it turns out that for my nice, clean and modern PipeWire system I would still have to build PulseAudio first. While PipeWire provides the compatability layer with applications that use PulseAudio as audio backend (like Firefox) I still need to provide those applications with libpulse first at build time.

My package of PulseAudio only provides the library and not the daemon so I have the least amount of conflicting sound systems as possible in the distribution. Of course Alsa is also packaged but that is required anyway by PipeWire to access the hardware. For the Jack parts of PipeWire I have simply decided to not have the Jack parts.

Building Firefox

So once I had all the dependencies out of the way I started the painful process of building Firefox. This part contained a lot of failed builds and many many many hours of wasted build time due to the build failing at the finish line.

I spend a lot of time trying to figure out why I had syntax errors in the Rust files for the CSS engine. For some reason the CSS engine for Firefox (The component is called "Style" so it's absolutely impossible to find anything for it in a search engine) has massive .rs files generated by Jinja2 templates and somewhere it's not providing syntaxically valid Rust files anymore. I spend weeks swapping out various dependency versions of various parts to figure out why these files are invalid until I gave up.

Then a week later I decided to try again but this time Firefox 134.0 was released so I changed the package to use the newer release, this instantly fixed all the Rust related build issues... Of course I still had a few more build issues to figure out but in the end I was presented with this nice message from the Firefox build system:

This is probably referring to the 68 minute build time, but I'd like to think the Firefox build system is self aware and knows of my days of debugging.

Increasing the difficulty and building for ARM

So at this point I felt like things were going too easy, so one day I decided it would be funny to rebuild BodgeOS for ARMv8. Due to the way BodgeOS is built this means building the distribution from LFS again since I have no support for cross-building in this system, simply because that's a lot of complication to maintain in the build system for something that is only done very rarely.

To build the ARM version of BodgeOS I wanted to build it natively on an ARM system and it turns out that for some reason I have a lot of random ARM systems around :D, I decided to grab the LX2K board I have since it's probably the fastest ARM64 system I have here (I haven't really checked the benchmarks how it compares with the RK3588 systems) but it most certainly is the most sane one of all of them. This is the only ARM SystemReady certified hardware I have, I don't think I can explain it any better than the marketing blurb on the manufacturers website:

The SystemReady ES description from solid-run.com

This means I could just unpack the generic Alpine Linux ARM64 rootfs on a random 2.5" SSD I had around, plug it in and boot it. Even easier is that I already had done that before and the SSD I used last time was still in there so I continued on directly with building from the Alpine 3.15 system I had on there.

The LX2K board in a 2U rack case powered by a picoPSU

From here on out the steps are incredibly similar to bringing up the x86_64 version of BodgeOS. I used the automated JHALFS system to get my clean LFS system to start the whole system from, surprisingly building this on ARM worked practically perfectly the first time while nothing in JHALFS suggests it even is tested on ARM systems.

From there on I copied my temporary helper scripts from my LFS installation on my Thinkpad that I used to build the first iteration of x86_64 BodgeOS and hacked together abuild inside LFS to run the build scripts. The rest was actually a lot easier since I didn't have to figure out any of the packaging again, 99% of the package builds scripts simply worked on ARM64 or were simply missing some build dependency declarations.

What also suprised me is that this ARM board actually outperformed my relatively modern Thinkpad in build speed. The Thinkpad I've been using for this is an X280 with the i5-8250U CPU in it. What also helps for a few builds is that the LX2K board has twice amount of RAM in it currently than the Thinkpad has (16GB vs 8GB) which means I didn't have to pass in -j4 for a few builds that were very memory heavy. Running with only 4 threads is quite painful on this board anyway because it's speed comes from having 16 cores in it, and the cooling to actually run that at full speed.

The initial LFS build on the LX2K board

Continuing on

I have gotten to a minimal working installation on the LX2K for the ARM64 build and I've been packaging random tools I need on the x86_64 version now. To continue I should probably make an actual build system that does automated builds from the git repositories on both platforms instead of me just manually triggering abuild for every package and rsync'ing the file over to the mirror.

One of my goals other than Firefox was building Kicad but this is more painful than getting Firefox running due to the list of dependencies it requires. The main one being that I actually going to have to build Xorg to build Kicad.

BodgeOS pt.3: Graphical desktop

Martijn Braam — Sun, 29 Dec 2024 20:11:18 -0000

In the previous post I figured out all the internal weirdness of Linux booting to get BodgeOS running on actual hardware. The next goal was very clear: getting to a graphical environment. At the start of this month I had the goal set to running a web browser before 2024 ends but I've now slightly adjusted my goals down to being able to type this blog post in a terminal on my new OS.

So what does it take to get graphics in Linux? well the first component is very clear from experience: Mesa. This is the component that provides all the userspace components for the graphics drivers. I started with checking out both the LFS mesa build instructions and the Alpine and ArchLinux mesa packages. This is not a very nice package to build due to the large dependencies it has. This one project contains all the graphical hardware related code for any GPU Linux will run on and due to that it depends on several programming languages and compilers.

I have tried stripping down this package as much as possible: no X11 support, only intel graphics, only EGL for 3D acceleration, no extra components, no software rasterisation. This makes Mesa relatively easy to build. With only the i915 gallium driver for intel graphics I don't have to bring up any of the Vulkan, rust or llvm dependencies to get basic graphics.

The desktop

So I had to pick a graphical environment as goal to run. There are many choices for this and even more opinions on what the best one. I picked Sway here since it's a Wayland based environment so I don't have to go figure out all the X11 stuff. It's also very simple to build compared to something like Gnome or KDE Plasma. I guess there will be someone that has figured out that some random ancient window manager can be built with even less dependencies but this is the smallest one from the desktops I've actually used before :)

The dependency chain for Sway is pretty simple: wlroots abstracts away all the Wayland stuffs and makes it actually communicate with Mesa. Then it has some extra dependencies to render text and simple graphics to draw the bars and window decorations.

So I started figuring out the minimum dependencies for every component in the dependencies of wlroots and Sway and package all the things are that are needed. This included fixing up the Python udev bindings from my systemd package, packaging the Wayland protocols, a bunch of Xorg keyboard stuff because it seems like keymaps are still used from x* packages and finally seatd to provide a way to get a session for the desktop.

Sway

For Sway the dependencies got a lot more annoying to compile since it depends on Pango and Cairo for rendering and those build systems were just a massive pain to deal with. It seems like the higher you get in the stack of a Linux system the more bullshit is added to build systems to make things "easier". My particiular painpoint in the Sway dependencies is glib and gobject-introspection which is not sufficiently documented and seems to work on magic.

Font rendering

Along with Pango I also had to bring up the whole font system in Linux. This involves Pango, Cairo, HarfBuzz, Glib and several obscure libraries for font processing. These packages are fun because they contain circular dependencies so I had to build them a few extra times.

First attempt at booting

After getting the 41 packages built that are required to get to a very minimal Sway experience I generated a new rootfs and tried booting it on a laptop.

This started the hunt for optional dependencies that were not optional for my usecase. The first one was that seatd could not actually make a session for me because I had zero backends compiled in. This was a relatively simple fix of just enabling the builtin backend in seatd to get at least some session.

Next came the graphical stack issues...

It first took a bit of time figuring out which of these errors was the fatal one I should be looking at, it was not one of the red ones in this case. The important error here was the "iris: driver missing" line which is from Mesa. I had initially assumed that i915 was the hardware backend I needed for my laptops since it's the name I've always come across. Apparently my laptops are old but not old enough to require i915 graphics, instead I needed to enable the iris backend.

Enabling the flag for iris in Mesa is very simple, but the hard part is the additional dependencies this adds to Mesa. Iris requires libclc which is the library for OpenCL. This depends on SPIR-V and LLVM which means packaging another massive project.

LLVM was by far the biggest time sink for a single package I've had so far. This package takes absolutely forever to compile and I was building this on an x280 with 8GB ram. Since this laptop has 8 cores I have built everything with -j8 so far which works fine except for LLVM where I had to drop to -j4 to not run out of memory while building. I had the same issues with Clang as well and together I've spend 3 days waiting on either one of them to build to hit the next issue that needed slight adjustments in the flags.

With LLVM working I managed to build all the packages required for SPIR-V and libclc so I could finally build the iris backend in Mesa. Since I now had a few extra dependencies packaged I also could enable llvmpipe as software rasterizer and osmesa, the off-screen mesa renderer.

Sway starts

With my graphics drivers fixed I finally got Sway to run. This was a very unexciting start though since the only thing it actually rendered was a black screen with my cursor. To make this more annoying to debug it also did not allow me to switch back to a TTY with ctrl+alt+F{1,2,3,4} anymore to see any of the debug logs. This forced me to build the thing I had been postponing: openSSH.

By launching Sway through an SSH session I noticed that the first thing I was missing was the swaybg binary which apparently is a seperate package, that explains why by background was completely black. This was packaged and built in a few minutes which fixed 90% of the screen area. The next mystery was the missing bars.

Suprisingly with all the logging turned to max in Sway I still got no error message whatsoever about the bars not showing up. Even more suprising is that if I reloaded the config a few times there occiasionally were some graphical artifacts where the bars were supposed to be.

After trying a few things and guessing even more things I figured out why it did not show up: I have the entire font rendering pipeline working but I haven't packaged a single font.... So that was an easy fix.

To complete the minimal working environment I also built the foot package to have a terminal available that did not have too many extra dependencies to work.

BodgeOS Sway!

There's also several more low-level things I had to figure out on the way, like my installed system not having any locales available. This was quickly fixed by importing the locale-gen script from ArchLinux to generate the locales I need and fixing up my glibc package to put the locale files in the right location.

Branding

So now I have the bare minimum I could focus on more cleanup work and small features. One of those is making the default wallpaper for my distro. I ended up doing the same thing I always do when I get annoyed with Inkscape not doing what I want: Rendering the graphics directly using Python instead.

I made a small python script that uses Pillow to render a wallpaper at the requested resolution.

This was inspired by the KiCad PCB editor I had open. I thought it was topical since the distro name was also inspired by my electronics projects :D

Continuing on

So this most likely completes my BodgeOS project for this year. I'm now up to 238 APKBUILD scripts in the repository which build ~900 packages for the distribution.

I'll have to package a lot more probably for my next goal, which is getting Firefox to run. This includes some things I've been avoiding like figuring out how to bring up the rust ecosystem and packaging GTK. While the current packages might've been hard to figure out, the rust ecosystem seem to actively resist packaging efforts to make it even harder. Maybe I should get more of Python packaged first so I can use my own utilities for working with APKBUILD files.

Since this is also the last blog post of the year, happy new year everyone!

Megapixels 2.0 alpha release

Martijn Braam — Tue, 24 Dec 2024 13:16:57 -0000

It's been quite a while since I wrote a Megapixels update post. Since my last post libmegapixels has had a lot more testing on hardware other than the PINE64 devices and the Librem 5 which I originally wrote it for. This obviously found a few flaws in my library code for edge cases I hadn't had to deal with before but overall the fundamental ideas behind the library seem to work.

I have now removed the last device-specific workaround from the libmegapixels code and the device support is now purely config files with a few flags to turn on quirks present in a few drivers like not having ioctls implemented correctly.

I once again stood before the software release dilemma: should I push a release that's not perfect or keep waiting and waiting to release until every last bug has been ironed out. Currently when running Megapixels 2.0 on the original PinePhone it's not a perfect drop-in replacement with all the features which is why I wanted to hold off on a release. But there's a few other devices that now already have 100% camera functionality on the development branch and for those devices a release would be great.

Megapixels 2.0.0-alpha1

As a compromise I have tagged an alpha release now from the development branch. This was issues can be ironed out that will happen when running Megapixels on one of the many combinations of distributions and devices. Since Megapixels now is also split up in the megapixels, libdng and libmegapixels projects the packaging can also now be figured out. The two libraries also have a new 0.2.0 tag now that marks the minimum version required for running the alpha release.

With this release it also means that all the library apis are now somewhat stable, but more importantly I'm now pretty confident that the config file format won't need any intrusive changes anymore so files for other devices can now be created without risking a lot of breakage down the line.

This format now also finally has some proper documentation over at https://libme.gapixels.me/config.html because "copy another file and hope for the best" is simply not a great developer experience.

Megapixels 2.0 running on the Samsung Galaxy SIII (ported by @jack_kekzoz)

Many thanks

I've not build this release alone ofcourse. I'd like to thank @k.vos, @pavelm, @pastalian, @jack_kekzoz, @barni2000 and @Luigi311 for their contributions to all the various parts of this codebase. I'd also like to thank the people that have supported my patreon/liberapay to sponsor me working on this :)

The release

The most important link ofcourse, the Megapixels tag:

https://gitlab.com/megapixels-org/Megapixels/-/tags/2.0.0-alpha1

The libraries releases are:

Documentation available at:

BodgeOS pt.2: Running on real hardware

Martijn Braam — Tue, 17 Dec 2024 00:30:04 -0000

In the previous part of this series I created a base Linux distribution from a running LFS system. This version only ran as a container which has several benefits that makes building the distribution a lot easier. For a simple container I didn't have to have:

A service manager (systemd)
Something to make it bootable on x86_64 systems (grub, syslinux, systemd-boot)
A kernel
An initramfs to get my filesystem mounted
File-system utilities since there's a folder instead of a filesystem.

A few of these are pretty easy to get running. I already have all the dependencies to build a kernel so I generated a kernel from the linux-lts package in Alpine Linux.

To make things easier for myself I just limited the distribution to run on UEFI x86_64 systems for now. This means I don't have to mess with grub ever again and I can just dump systemd-boot into my /boot folder to get a functional system. I had to build this anyway since I had to build systemd to have an init system for my distribution.

The Initramfs

The thing that took by far the longest is messing with the initramfs to make my test system boot. The initramfs generator is certainly one of the parts that have the most distribution-specific "flavor". Everyone invents it's own solution for it like mkinitcpio for ArchLinux and mkinitfs for Alpine and initramfs-tools for Debian as a few examples.

I did the only logical thing and reinvented the wheel here. I'm even planning to reinvent it even further! Like the above solutions my current initramfs generator is a collection of shell scripts. The initramfs is a pretty simple system after all: it has to load some kernel modules, find the rootfs, mount it and then execute the init in the real system.

For a very minimal system the only required thing is the busybox binary, it provides the shell script interpreter required to run the messy shell script that brings up the system and also provides all the base utilities. Due to my previous experiences with BusyBox modprobe in postmarketOS I decided to also move the real modprobe binary in the initramfs to have things loading correctly. To complete it I also added blkid instead of relying on the BusyBox implementation here to have support for partition labels in udev so no custom partition-label-searching code is required.

Getting binaries in the initramfs is super easy. The process for generating an initramfs is:

Create an empty working directory
Move in the files you need into the working directory from the regular rootfs like /usr/bin/busybox > /tmp/initfs-build/usr/bin/busybox
Add in a script that functions as pid 1 in the initramfs and starts execution of the whole system
Run the cpio command against the /tmp/initfs-build directory to create an archive of this temporary rootfs and run that through gzip to generate initramfs.gz

Step 2 is fairly simple since I just need to copy the binaries from the host system, but those binaries also have dependencies that need to be copied to make the executable actually work. Normally this is handled by the lddtree utility but I didn't feel like packaging that. It is a shell script that does a complicated task which is never a good thing and it depends on python and calling various ELF binary debugging utilities.

Instead of using lddtree I brought up Hare on my distribution and wrote a replacement utility for it called bindeps. This is just a single binary that loads the ELF file(s) and spits out the dependencies without calling any other tools. This is significantly faster than the performance overhead of lddtree which was always the slowest part of generating the initramfs for postmarketOS.

The output format is also optimized to be easily parse-able in the mkinitfs shellscript.

$ lddtree /usr/sbin/blkid /usr/sbin/modprobe 
/usr/sbin/blkid (interpreter => /lib64/ld-linux-x86-64.so.2)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
/usr/sbin/modprobe (interpreter => /lib64/ld-linux-x86-64.so.2)
    libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1
    liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5
    libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6

$ bindeps /usr/bin/blkid /usr/bin/modprobe 
/usr/lib/ld-linux-x86-64.so.2
/usr/lib/libblkid.so.1.1.0
/usr/lib/libc.so.6
/usr/lib/libzstd.so.1.5.6
/usr/lib/liblzma.so.5.6.3
/usr/lib/libz.so.1.3.1
/usr/lib/libcrypto.so.3

The bindeps utility seems to be roughly 100x faster in the few testcases I've used it in and it outputs in a format that needs no further string-mangling to be used in a shell script. In BodgeOS mkinitfs it's used like this:

binaries="/bin/modprobe /bin/busybox /bin/blkid"
for bin in $binaries ; do
    install -Dm0755 $bin $workdir/$bin
done

bindeps $binaries | while read lib ; do
    install -Dm0755 $lib $workdir/$lib
done

The next part is the kernel modules. Kernel modules are also ELF binaries just like the binaries I just copied over but they sadly don't contain any dependency metadata. This metadata is stored in a seperate file called modules.dep that has to be parsed seperately. I did not bother with this and copied the solution from the initramfs generator example from LFS and just copy hardcoded folders of modules into the initramfs and hope it works.

The file format for modules.dep is trivial so I really want to just integrate support for that into bindeps in the future.

Debugging the boot

It's suprisingly painful to debug a non-booting Linux system that fails in the initramfs. I wasted several hours figuring out why the kernel threw errors at the spot the initramfs should start executing which ended up being an issue with the /sbin/init file had the wrong shebang line at the start so it was not loadable. The kernel has no proper error message that conveys any of this.

After I got the initramfs to actually load and start a lot of time was wasted on executables missing the interperter module. In the example above this is the /lib64/ld-linux-x86-64.so.2 line. The issue here ended up that I was just missing the /lib64 symlink in my initramfs. This was very hard to debug in a system without debug utilities because nothing could execute.

After all that I spend even more time figuring out why I had no kernel log lines on my screen. After much annoyance this turned out to be missing options in the kernel config for the linux-lts kernel config I took from Alpine Linux. So instead of fixing that I took the kernel config from ArchLinux and rebuild the linux-lts package. This fixed my kernel log output issue but also added a new one... The keyboard in my laptop wasn't working in the initramfs.

I never did figure out which module I was missing for that because I fixed the rest of the initramfs script instead so it just continues on to the real rootfs where all the modules are available.

After all that I did manage to get to a login prompt though!

Cleaning up

After booting up this I realized it would be really handy if I actually had a /etc/passwd file in my system and some more of the bare essentials so I could actually log in and use the system.

This mainly involved adding missing dependencies in packages and packaging a few more files in /etc to make it a functional system. After the first boot the journal had a neat list of system binaries from util-linux that systemd depends on but not explicitly, so I added those dependencies to my systemd packaging.

I also had to fix the issue that my newly-installed system did not trust the BodgeOS repository, I was missing the keys package that installs the repository key in /etc/apk/keys for me. In this process I noticed that the key I built the system with was called `-randomdigits.pub` instead of being prefixed with a name. This is pretty annoying because this name is embedded in all the compiled packages and I didn't want to ship a file with that name in my keys package.

There seemed to be a nifty solution though: the abuild-sign tool appends a key to a tar archive, which is normally used to sign the APKINDEX.tar.gz file that contains the package list in the repository. I decided to run abuild-sign *.apk in my main repository after adjusting the abuild signing settings with a correct key name.

Apparently this breaks .apk files and after inspection they now had two keys in them and neither my development LFS install and my test BodgeOS install wanted to have anything to do with the packages anymore.

In the end I had to throw away my built packages and rebuild everything again from the APKBUILD files I had. Luckily this distribution is not that big yet so a full rebuild only took about 2.5x the duration of Dark Side of the Moon.

Next steps

Now I have a basic system that can boot I continued with packaging more libraries and utilites you'd expect in a regular Linux distribution. One of the things I thought would be very neat is having curl available, but that has a suprising amount of dependencies. Some tools like git will be useful too before I can call this system self-hosting.

I also want to remove all the shell scripts from the initramfs generation. None of the tasks in the initramfs are really BodgeOS specific and most of the complications and bugs in this initramfs implementation (and the one in postmarketOS) is because the utilities it depends on are not really intended to do this stuff and system bootup just has a lot of race conditions shell scripts are just not great at handling.

My current plan to fix that is to just replace the entire initramfs with a single statically linked binary. All this logic is way neater to implement in a good programming language.

Conjuring a Linux distribution out of thin air

Martijn Braam — Sat, 07 Dec 2024 23:08:27 -0000

I decided I had to get something with slightly more CPU power than my Thinkpad x230 for a few tasks so I got a refurbished x280, aside from the worse keyboard the laptop is pretty nice and light. It shipped with Windows of course so the first thing I did is to install Ubuntu on the thing to wipe that off and verify all the hardware is working decently.

I was wondering if I should leave Ubuntu on the thing, it works pretty well and it's still possible to get rid of all the Snap stuff, it's not my main machine anyway. The issue I ran into quickly though is some software is pretty outdated, like I don't want to use Kicad 7 anymore...

Picking distros once again

You'd think after using Linux for decades I would know what distro I'd put on a new machine. All the options I could think off though had annoying trade-offs I didn't want to deal with once again.

The three main distributions I have running on hardware I manage is Alpine Linux, Archlinux and Debian. I like Alpine a lot but it is quite annoying when you deal with closed-source software. Since this is my go-to laptop to take with me to outages and repairs then I need it to handle random software thrown on it relatively easily.

ArchLinux satisfies that requirement pretty well but my main issue with it is pacman. If you don't religiously run upgrades every hour on the thing it will just break because for some reason key management is not figured out yet there. The installation is also quite big usually due to packages not being split.

Debian fixes the stability issues but comes with the trade off that software is usually much much older, this also leaks into Ubuntu that's running on the laptop now. It is also internally a lot more complicated due to the way it automatically sets up stuff while installing which I don't usually need.

There is another solution though. Just build my own!

Artisanal home-grown Linux

Creating a new Linux distribution is one of those things that sounds much harder than it actually is. I just haven't done it before. I did build a small Debian derivative distro before just to avoid re-doing all the config for all machines but that's just adding an extra repository to an existing distribution. Of course I've also worked extensively on postmarketOS and while the scope of that is a lot larger it still is only a repository with additions on Alpine Linux.

Some of you might be familiar with this graphic of Linux distributions:

There are many many many derivative distributions here, way less distributions that are built up from nothing. And that's exactly the part that I want to figure out. How do I make a distribution from scratch?

I have once, a long time ago, build a working Linux machine from sources using the great Linux From Scratch project. I would recommend that anyone that's really into Linux internals do that at least once. Just like Archlinux learns you how the distribution installer works (before they added an installer) the LFS book will learn you how you bootstrap a separate userspace from another Linux distribution and doing the gcc/glibc dependency loop build.

So my plan for the distribution is: create a super barebones system using the LFS book, package up that installation using abuild and apk from Alpine Linux. This way I can basically make my own systemd/glibc distribution that is mostly like Archlinux but uses the packaging tools and methodology from Alpine Linux.

The bootstrap build

So to make my distribution I first have to build the Linux installation that will build the packages for the distribution. To get through this part relatively quickly I used the automated LFS installer called ALFS. This basically does all the steps from the LFS book but very quickly. My intention is to replace all of these packages anyway so this part was not super important. It does all the required setup though to validate the GCC I'm using to build my distribution is sane and tested.

In the ALFS installer I picked the systemd option since I didn't want to deal with openrc again and ended up with a nice functional rootfs to work in. I immediately encountered a few things that were critical and missing though. There was no wget or curl. I fixed this by grabbing the Busybox binary from my host Ubuntu distribution and putting that inside with only a wget symlink to it.

This wget was not functional in my chroot though since it could not connect to https servers. Annoyingly all solutions you'd find for these errors is passsing --no-check-certificates to wget to get your files which is unacceptable when building a distribution. After a lot of debugging with openssl configs I ended up copying over all the certificates from the host Ubuntu system again and pointed wget to the certificate bundle with the wgetrc file.

The very next thing I needed is abuild. This is the tool from Alpine Linux that's used to build the packages. Luckily it's a few small C programs and a large shell script so it's very easy to install in the temporary system. I also added apk.static to the system to be able to install the built packages.

Building packages

So now I have my temporary system running I could start writing APKBUILD files for all the packages in my LFS system. I started with the very simplest one of course that only provides /etc/services and /etc/protocols. No compiling and dependencies needed.

For this package made the script that built the current APKBUILD using the abuild subcommands and generated a neat local repository so apk could install the files. So I ran that and now /etc/services and /etc/protocols are now exactly the same files but managed by apk and in a package.

The reason I had to use the subcommands for abuild to run seperate stages instead of just running abuild itself to do the whole things is because one of the first steps is installing the build dependencies. In this case I'm in the weird setup where I have all the build dependencies installed through LFS but apk doesn't know about that so I simply skip that step.

And that's how I re-build a lot of the LFS packages once again, but this time through my half broken abuild installation. One of the things that abuild was not happy about is the lack of the scanelf utility which it uses after building a package to check which .so files the binaries in the package depend on. Due to this a lot of dependencies between packages are simply missing. The scanelf utility has enough build dependencies though that I could not have that as the first few packages so most of the packages are broken in this stage.

When I finally built and installed scanelf I ran into another issue. The packages I build after this failed at the last step because scanelf found the .so files required for the package but the package metadata for all the packages I made before it lack this information about the included .so files apparently. At this point I had to build all those packages for a fourth time (twice in LFS and now twice in abuild) to make dependencies here work.

After getting through practically everything in the base system I ended up with around 333 packages in my local repository.

Most of these APKBUILD files are a combination of the metadata header copied from the Alpine Linux ABUILD so I have a neat description and the correct license data. And then the build steps and flags from LFS and sometimes the install step from Archlinux.

This means that for example the xz build steps in LFS are:

$ ./configure --prefix=/usr    \
              --disable-static \
              --docdir=/usr/share/doc/xz-5.6.3
$ make
$ make check
$ make install

And that combined with the Alpine Linux metadata headers and adjustments for packaging becomes:

pkgname=xz
pkgver=5.6.3
pkgrel=0
pkgdesc="Library and CLI tools for XZ and LZMA compressed files"
url="https://tukaani.org/xz/"
arch="all"
license="GPL-2.0-or-later AND 0BSD AND Public-Domain AND LGPL-2.1-or-later"
subpackages="$pkgname-doc $pkgname-libs $pkgname-dev"
source="https://github.com//tukaani-project/xz/releases/download/v$pkgver/xz-$pkgver.tar.xz"

build() {
        ./configure \
                --prefix=/usr \
                --disable-static \
                --docdir=/usr/share/doc/xz-$pkgver
        make
}

check() {
        make check
}

package() {
        make DESTDIR="$pkgdir" install
}

Getting the first install to run

Now the hard part, finding all the issues that prevent this new installation from starting. The first thing I tried to use the apk.static on my host system to generate a new chroot from the repository I created, just like you'd install an Alpine chroot.

Unfortunately this did not work and I had to fork apk-tools and make my own adjusted version. This is mainly because apk-tools hardcodes paths in it which conflict with my usrmerge setup. So I now have an apk.static build from my fork that does not try to create /var/ for the database before the baselayout package can create the actual filesystem hierarchy with a symlink at that spot.

With that fixed apk.static would be able to finish creating an installation from my repository, but I could not chroot into it for some reason. All the binaries are broken and return "Not found" when trying to execute them. I managed to actually enter the chroot by throwing my trusty busybox binary in there but did not get any more information out of that installation.

After a bunch of testing, debugging, and more testing, I found out the reason was that I don't have /lib64 in my installation. It seems like it's required specifically because x86_64 binaries specify /lib64/ld-linux-x86-64.so.2 as loader. The fix for that is quite easy by just having the glibc package place a symlink at that spot to the real ld-linux.so.

Beyond the first run

There's a lot of things that need to be fixed up to be a good distribution. All the packages will need to be rebuild again from the distribution installled from these first generation packages to leave behind the last parts of LFS in there. There's also the system setup that needs to happen to make it bootable and maintainable. For example things like the keys package that install and update the repository keys and adding the logo to neofetch (after packaging neofetch).

I've also rsync'd the repository for the distribution to a webserver so it can actually be added to installations. I've been using this now to create test chroots using my locally build patched apk-tools.

The repository itself also needs quite a bit of work, gcc shouldn't've been pulled in here for these packages and a bunch of the large packages need to be split up to remove the uncommon parts. Currently the glibc package is already 10x larger than a base Alpine Linux installation, luckily apk is a very fast package manager and everything still installs super quickly.

So why?

It doesn't really make sense to do this. I wanted to have done it anyway because that's how you learn. This took about 3 days of fiddling with build scripts between other work since it's mostly waiting on builds to finish.

At the very least this distribution created this blog post, which is surprisingly one of the very few pieces of information available for bootstrapping a distribution.

Even with this tiny base of packages this is already a quite usable OS since I have a kernel, a service manager and python. All you need to build some embedded stuff if this had an ARM64 port bootstrapped or something. For desktop work there's still a mountain of work to be done to package everything required to launch Firefox, getting the basic graphics stack up and running should be relatively straightforward with bringing up Sway with its dependencies.

In the end it probably would've been easier to just add a ppa for Kicad to my Ubuntu installation :)

Building a timeseries database for fun

Martijn Braam — Mon, 28 Oct 2024 22:50:55 -0000

Everyone that has tried to make some nice charts in Grafana has probably come across timeseries databases, for example InfluxDB or Prometheus. I've deployed a few instances of these things for various tasks and I've always been annoyed by how they work. But this is a consequence of having great performance right?

The thing is... most the dataseries I'm working with don't need that level of performance. If you're just logging the power delivered by a solar inverter to a raspberry pi then you don't need a datastore for 1000 datapoints per second. My experience with timeseries is not that performance is my issue but the queries I want to do which seem very simple are practically impossible, especially when combinated with Grafana.

Something like having a daily total of a measurement as a bar graph to have some long-term history with keeping the bars aligned to the day boundary instead of 24 hour offsets based on my current time. Or being able to actually query the data from a single month to get a total instead of querying chunks of 30.5 days.

But most importantly, writing software is fun and I want to write something that does this for me. Not everything has to scale to every usecase from a single raspberry pi to a list of fortune 500 company logos on your homepage.

A prototype

Since I don't care about high performance and I want to prototype quickly I started with a Python Flask application. This is mainly because I already wrote a bunch of Python glue before to pump the data from my MQTT broker into InfluxDB or Prometheus so I can just directly integrate that.

I decided that as storage backend just using a SQLite database will be fine and to integrate with Grafana I'll just implement the relevant parts of the Prometheus API and query language.

To complete it I made a small web UI for configuring and monitoring the whole thing. Mainly to make it easy to adjust the MQTT topic mapping without editing config files and restarting the server.

I've honestly probably spend way too much time writing random javascript for the MQTT configuration window. I had already written a MQTT library for Flask that allows using the Flask route syntax to extract data from the topic so I reused that backend. To make that work nicely I also wrote a simple parser for the syntax in Javascript to visualize the parsing while you type and give you dropdowns for selecting the values.

This is not at all related to the dataseries part but at least it allows me to easily get a bunch of data into my test environment while writing the rest of the code.

The database

For storing the data I'm using the sqlite3 module in Python. I dynamically generate a schema based on the data that's coming in with one table per measurement.

There's two kinds of devices on my MQTT broker, some send the data as a JSON blob and some just send single values to various topics.

Example data from the MQTT broker

The JSON blobs are still considered a single measurement and all the top-level values get stored in seperate columns. Later in the querying stage the specific column is selected.

My worst case is a bunch of ESP devices that measure various unrelated things and output JSON to the topic shown above with JSON. I have a single ingestion rule in my database that grabs devices/hoofdweg/ and dumps it in a table that has the columns for the various sensors, which ends up with a schema like this:

A timestamp is stored, no consideration is made for timezones since in practically all cases a house isn't located right on a timezone boundary. The tags are stored in seperate columns with a tag_ prefix and the fields are stored in column with a field_ prefix. The maximum granularity of data is also a single second since I don't store the timestamp as a float.

A lot of the queries I do don't need every single datapoint though but instead I just need hourly, daily or monthly data. For that a second table is created with the same structure but with aggregated data:

This contains a row for every hour with the min(), max() and avg() of every field, it also contains a row for every day and one for every month. This makes it possible to after a preconfigured amount of time just throw away the data that has single-second granularity and keep the aggregated data way longer. For querying you explicitly select which table you want the data from.

The querying

To make the Grafana UI not complain too much I kept the language syntax the same as Prometheus but simply implemented less of the features because I don't use most of them. The supported features right now are:

Simple math queries like 1+1, this can only do addition queries and is only here to satisfy the Grafana connection tester.
Selecting a single measurement from the database and filtering on tags using the braces like my_sensors{sensor="solar"}
Selecting a time granularity with brackets like example_sensor[1h]. This only supports 1h, 1d and 1M and selects which rows are queried
The aggregate functions like max(my_sensors[1h]) which makes it select the columns from the reduced table with the max_ prefix for querying when using the reduced table. For selecting the realtime data it will use the SQLite max() function.

This is also just about enough to make the graphical query builder in Grafana work for most cases. The other setting used for the queries is the step value that Grafana calculates and passes to the Prometheus API. For the reduced table this is completely ignored and for the realtime table this is converted to SQL to do aggregation across rows.

As an example the query avg(sensors{sensor="solar", _col="voltage"}) gets translated to:

SELECT
  instant,
  tag_sensor,
  avg(field_voltage) as field_voltage
FROM series_sensors
WHERE instant BETWEEN ? AND ? -- Grafana time range
  AND tag_sensor = ? -- solar
GROUP BY instant/30 -- 30 is the step value from Grafana

To get nice aligned hourly data for a bar chart the query simply changes to avg(sensors{sensor="solar", _col="voltage"}[1h]) which generates this query:

SELECT
  instant,
  date,
  hour,
  tag_sensor,
  avg_voltage
FROM reduced_sensors
WHERE instant BETWEEN ? AND ? -- Grafana time range
  AND tag_sensor = ? -- solar
  AND scale = 0 -- hourly

This reduced data is generated as background task in the server and makes sure that the row with the aggregate of a single hour selects the datapoints that fit exactly in that hour, not shifted by the local time when querying like I now have issues with in Grafana:

The query running against the old Prometheus database

The bars in this chart don't align with the dates because this screenshot wasn't made at midnight. The data in the bars is also only technically correct when viewing the Grafana dashboard at midnight since on other hours it selects data from other days as well. If I view this at 13:00 then I get the data from 13:00 the day before to today which is a bit annoying in most cases and useless in the case of this chart because the daily_total metric in my solar inverter is reset at night and I pick the highest value.

For monthly bars this issue gets worse because it's apparently impossible to accurately get monthly data from the timeseries databases I've used. Because I'm pregenerating this data instead of using magic intervals this also Just Works(tm) in my implementation.

The same sort of query on the Miniseries backend, hourly instead because I don't have enough demo data yet.

Is this better?

It is certainly in the prototype stage and has not had enough testing to find weird edgecases. It does provide all the features though I need to recreate by existing home automation dashboard and performance is absolutely fine. The next step here is to implement a feature to lie to Grafana about the date of the data to actually use the heatmap chart to show data from multiple days as multiple rows.

Once the kinks are worked out in this prototype it's probably a good idea to rewrite it into something like Go for example because while a lot of the data processing is done in SQLite the first bottleneck will probably be the single-threaded nature of the webserver and the MQTT ingestion code.

The source code is online at https://git.sr.ht/~martijnbraam/miniseries

Making a Linux-managed network switch

Martijn Braam — Wed, 03 Jul 2024 14:10:04 -0000

Network switches are simple devices, packets go in, packets go out. Luckily people have figured out how to make it complicated instead and invented managed switches.

Usually this is done by adding a web-interface for configuring the settings and see things like port status. If you have more expensive switches then you'd even get access to some alternate interfaces like telnet and serial console ports.

There is a whole second category of managed switches though that people don't initially think of. These are the network switches that are inside consumer routers. These routers are little Linux devices that have a switch chip inside of them, one or more ports are internally connected to the CPU and the rest are on the outside as physical ports.

Mikrotik RB2011 block diagram from mikrotik.com

Here is an example of such a device that actually has this documented. I always thought that the configuration of these switch connected ports was just a nice abstraction by the webinterface but I was suprised to learn that with the DSA and switchdev subsystem in Linux these ports are actually fully functioning "local" network ports. Due to this practically only being available inside integrated routers It's pretty hard to play around with unfortunately.

What is shown as a single line on this diagram is actually the connection of the SoC of the router and the switch over the SGMII bus (or maybe RGMII in this case) and a management bus that's either SMI or MDIO. Network switches have a lot of these fun acronyms that even with the full name written out make little sense unless you know how all of this fits together.

Controlling your standard off-the-shelf switch using this system simply isn't possible because the required connections of the switch chip aren't exposed for this. So there's only one option left...

Making my own gigabit network switch

Making my own network switch can't be that hard right? Those things are available for the price of a cup of coffee and are most likely highly integrated to reach that price point. Since I don't see any homemade switches around on the internet I guess the chips for those must be pretty hard to get...

Nope, very easy to get. There's even a datasheet available for these. So I created a new KiCad project and started creating some footprints and symbols.

I'm glad there's any amount of datasheet available for this chip since that's not usually the case for Realtek devices, but it's still pretty minimal. I resorted to finding any devices that has schematics available for similar Realtek chips to find out how to integrate it and looking at a lot of documentation for how to implement ethernet in a design at all.

The implementation for the chip initially looked very complicated, there's about 7 different power nets it requires and there are several pretty badly documented communication interfaces. After going through other implementations it seem like the easiest way to power it is just connect all the nets with overlapping voltage ranges together and you're left with only needing a 3.3V and 1.1V regulator.

The extra communication busses are for all the extra ports I don't seem to need. The switch chip I selected is the RTL8367S which is a very widely used 5-port gigabit switch chip, but it's actually not a 5-port chip. It's a 7 port switch chip where 5 ports have an integrated PHY and two are for CPU connections.

CPU connection block diagram from the RTL8367S datasheet

My plan is different though, while there are these CPU ports available there is actually nothing in the Linux switchdev subsystem that requires the CPU connection to be to those ports. Instead I'll be connecting to port 0 on the switch with a network cable and as far as the switchdev driver knows there's no ethernet PHY in between.

The next hurdle is the configuration of the switch chip, there's several configuration systems available and the datasheet does not really describe what is the minimum required setup to actually get it to function as a regular dumb switch. To sum up the configuration options of the chip:

There's 8 pins on the chip that are read when it's starting up. These pins are shared with the led pins for the ports so that makes designing pretty annoying. Switching the setting from pull-up to pull-down also requires the led to be connected in the different orientation.
There's an i2c bus that can be connected to an eeprom chip. The pins for this are shared with the SMI bus that I require to make this chip talk to Linux though. There is pin configuration to select from one of two eeprom size ranges but does not further specify what this setting actually changes.
There's a SPI bus that supports connecting a NOR flash chip to it. This can store either configuration registers or firmware for the embedded 8051 core depending on the configuration of the bootup pins. The SPI bus pins are also shared with one of the CPU network ports.
There is a serial port available but from what I guess it probably does nothing at all unless there's firmware loaded in the 8051.

My solution to figuring out is to just order a board and solder connections differently until it works. I've added a footprint for a flash chip that I ended up not needing and for all the configuration pins I added solder jumpers. I left out all the leds since making that configurable would be pretty hard.

The next step is figuring out how to do ethernet properly. There has been a lot of documentation written about this and they all make it sound like gigabit ethernet requires perfect precision engineering, impedance managed boards and a blessing from the ethernet gods themselves to work. This does not seem to match up with the reality that these switches are very very cheaply constructed and seem to work just fine. So I decided to open up a switch to check how many of these coupling capacitors and impedance matching planes are actually used in a real design. The answer seems to be that it doesn't matter that much.

This is the design I have ended up with now but it is not what is on my test PCB. I got it almost right the first time though :D

The important parts seem to be matching the pair skew but matching the length of the 4 network pairs is completely useless, this is mainly because network cables don't have the same twisting rate for the 4 pairs and so the length of these are already significantly different inside the cable.

The pairs between the transformer and the RJ45 jack has it's own ground plane that's coupled to the main ground through a capacitor. The pairs after the transformer are just on the main board ground fill.

What I did wrong on my initial board revision was forgetting the capacitor that connects the center taps of the transformer on the switch side to ground making the signals on that side referenced to board ground. This makes ethernet very much not work anymore so I had to manually cut tiny traces on the board to disconnect that short to ground. In my test setup the capacitor just doesn't exist and all the center taps float. This seems to work just fine but the final design does have that capacitor added.

Cut ground traces on the ethernet transformer

The end result is this slightly weird gigabit switch. It has 4 ports facing one direction and one facing backwards and it is powered over a 2.54mm pinheader. I have also added a footprint for a USB Type-C connector to have an easy way to power it without bringing out the DuPont wires.

Connecting it to Linux

For my test setup I've picked the PINE64 A64-lts board since it has the connectors roughly in the spots where I want them. It not being an x86 platform is also pretty important because configuration requires a device tree change, can't do that on a platform that doesn't use device trees.

The first required thing was rebuilding the kernel for the board since most kernels simply don't have these kernel modules enabled. For this I enabled these options:

CONFIG_NET_DSA for the Distributed Switch Architecture system
CONFIG_NET_DSA_TAG_RTL8_4 for having port tagging for this Realtek switch chip
CONFIG_NET_SWITCHDEV the driver system for network switches
CONFIG_NET_DSA_REALTEK, CONFIG_NET_DSA_REALTEK_SMI, CONFIG_NET_DSA_REALTEK_RTL8365MB for the actual switch chip driver

Then the more complicated part was figuring out how to actually get this all loaded. In theory it is possible to create a device tree overlay for this and get it loaded by U-Boot. I decided to not do that and patch the device tree for the A64-lts board instead since I'm rebuilding the kernel anyway. The device tree change I ended up with is this:

diff --git a/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts b/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts
index 596a25907..10c1a5187 100644
--- a/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts
+++ b/arch/arm64/boot/dts/allwinner/sun50i-a64-pine64-lts.dts
@@ -18,8 +18,78 @@ led {
 			gpios = <&r_pio 0 7 GPIO_ACTIVE_LOW>; /* PL7 */
 		};
 	};
+
+switch {
+	compatible = "realtek,rtl8365rb";
+	mdc-gpios = <&pio 2 5 GPIO_ACTIVE_HIGH>; // PC5
+	mdio-gpios = <&pio 2 7 GPIO_ACTIVE_HIGH>; // PC7
+	reset-gpios = <&pio 8 5 GPIO_ACTIVE_LOW>; // PH5
+	realtek,disable-leds;
+
+	mdio {
+		compatible = "realtek,smi-mdio";
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		ethphy0: ethernet-phy@0 {
+			reg = <0>;
+		};
+
+		ethphy1: ethernet-phy@1 {
+			reg = <1>;
+		};
+
+		ethphy2: ethernet-phy@2 {
+			reg = <2>;
+		};
+
+		ethphy3: ethernet-phy@3 {
+			reg = <3>;
+		};
+
+		ethphy4: ethernet-phy@4 {
+			reg = <4>;
+		};
+	};
+
+	ports {
+		#address-cells = <1>;
+		#size-cells = <0>;
+
+		port@0 {
+			reg = <0>;
+			label = "cpu";
+			ethernet = <&emac>;
+		};
+
+		port@1 {
+			reg = <1>;
+			label = "lan1";
+			phy-handle = <&ethphy1>;
+		};
+
+		port@2 {
+			reg = <2>;
+			label = "lan2";
+			phy-handle = <&ethphy2>;
+		};
+
+		port@3 {
+			reg = <3>;
+			label = "lan3";
+			phy-handle = <&ethphy3>;
+		};
+
+		port@4 {
+			reg = <4>;
+			label = "lan4";
+			phy-handle = <&ethphy4>;
+		};
+	};
+};
 };

It loads the driver for the switch with the realtek,rtl8365rb, this driver supports a whole range of Realtek switch chips including the RTL8367S I've used in this design. I've removed the CPU ports from the documentation example and just added the definitions of the 5 regular switch ports.

The important part is in port@0, this is the port that is facing backwards on my switch and is connected to the A64-lts, I've linked it up to &emac which is a reference to the ethernet port of the computer. The rest of the ports are linked up to their respective PHYs in the switch chip.

In the top of the code there's also 3 GPIOs defined, these link up to SDA/SCL and Reset on the switch PCB to make the communication work. After booting up the system the result is this:

1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1508 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
3 lan1@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
4 lan2@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
5 lan3@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
6 lan4@eth0: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff

I have the eth0 device here like normal and then I have the 4 interfaces for the ports on the switch I defined in the device tree. To make it actually do something the interfaces actually need to be brought online first:

$ ip link set eth0 up
$ ip link set lan1 up
$ ip link set lan2 up
$ ip link set lan3 up
$ ip link set lan4 up
$ ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1508 qdisc mq state UP qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
3: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
4: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
5: lan3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff
6: lan4@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN qlen 1000
    link/ether 02:ba:6f:0c:21:c4 brd ff:ff:ff:ff:ff:ff

Now the switch is up you can see I have a cable plugged into the third port. This system hooks into a lot of the Linux networking so it Just Works(tm) with a lot of tooling. Some examples:

Add a few of the lan ports into a standard Linux bridge and the switchdev system will bridge those ports together in the switch chip so Linux doesn't have to forward that traffic.
Thinks like ethtool lan3 just work to get information about the link. and with ethtool -S lan3 all the standard status return info which includes packets that have been fully handled by the switch.

Limitations

There's a few things that makes this not very nice to work with. First of all the requirement of either building a custom network switch or tearing open an existing one and finding the right connections.

It's not really possible to use this system on regular computers/servers since you need device trees to configure the kernel for this and most computers don't have kernel-controlled GPIO pins available to hook up a switch.

As far as I can find there's also no way to use this with a network port on the computer side that's not fixed, USB network interfaces don't have a device tree node handle to refer to to set the conduit port.

There is a chance some of these limitations are possible to work around, maybe there's some weird USB device that exposes pins on the GPIO subsystem, maybe there's a way to load switchdev without being on an ARM device but that would certainly take a bit more documentation...

Building a DSMR reading board

Martijn Braam — Mon, 27 May 2024 10:08:00 -0000

Quite a while back I designed a small PCB for hooking up sensors to an ESP8266 module to gather data and have a nice Grafana dashboard with temperature readings. While building this setup I grabbed one of my spare ESP8266 dev boards and soldered that to the P1 port on my smart energy meter to log the power usage of the whole house.

For those unfamiliar with P1 and DSMR since it's quite regional: DSMR is the Dutch Smart Meter Requirements specification. It's a document that describes the connectivity of various ports on energy meters and it's used in the Netherlands and a few countries around it. One of the specifications within DSMR is P1 which is the document for the RJ12 connector for plugging third party monitoring tools.

Electrical design

So the first part in the project is figuring out how to make the hardware itself work. I've decided to slightly modernize my design so this is the first module I build that uses an ESP32 chip instead of the older ESP8266 I have in use everywhere.

I specifically designed the board around the ESP32-S3-WROOM-1 module. This makes things significantly simpler than my older designs since there's no programming circuitry required.

The design contraints I've used for this board are:

Make the design handsolderable instead of using JLCPCB assembly service for the boards. I enjoy designing boards and I probably should get some more actual soldering experience with SMD parts.
Connect to the smart energy meter using off the shelf RJ12 cables instead of soldering wires on the board. Also include a passive P1 splitter on the board since it's becoming more and more popular to have charging points for electric cars which can also be hooked up to the P1 port.
USB-C for programming and power. I have a back-up programming header footprint on the board but it shouldn't be necessary . Just like putting USB-B Micro ports on devices shouldn't be allowed anymore in 2024.

The schematic is basically the absolute minimum required to get the ESP32 module up and running. Since the ESP32-S3 has native USB support it means that I can drop a whole bunch of parts from the schematic that is normally required for programming.

Another neat feature is that this chip has a fully connected I/O mux inside which means I can basically pick any random GPIO pin and later configure in software which hardware block in the chip it hooks up to. This feature is also available in other chips like the RP2040 but there it's way more limited. There is only a few valid choices for pins for UART1 for example and there's no way to swap RX and TX on a board without using a soldering iron.

In the DSMR design I'll only be using a single GPIO pin configured to be the RX of one of the hardware UARTS so it can receive the data.

The full schematic for the DSMR module

The board has a jumper on it to pick how the data requests are handled. In one position the line is connected straight to 5V so the energy meter will just continuously send the data over the P1 port, which should be correct in most situations. The other mode connects the data request line to the pass-through port so the device connected after the DSMR module can select when the data should be sent, in this case the module will just be passively sniffing the traffic whenever it's sent.

There is also a solder jumper on the board to select whether it's powered from the USB connection or from the P1 port itself. According to the P1 specifications the power supplied by most energy meters won't be enough to reliably run the ESP32 module so there is always to option to power it from the USB-C port. In most cases there will be random device like a router nearby that has a powered USB connection to run the module.

The PCB for the DSMR module

I managed to fit everything on a small 40x40mm PCB by having the the two RJ12 connectors hang off the board. This also makes those neatly line up with the edges of the case for this board. The ESP32 module also hangs off the edge of the board, pictured at the top here. This is because the antenna part of the ESP module has a keepout area where there shouldn't be any copper on the board anyway to not disturb the WiFi signal.

To make it fit I've also moved the power regulator and a few passives to the bottom side of the board. Normally it would be pretty expensive to do this but since I'm hand soldering these boards using both sides of the board is free.

Another upside with handsoldering the boards is that I'm not limited to the JLCPCB parts for once and I can just whatever random parts I can source. I've decided to get these boards made by Aisler this time so the boards are made in Germany instead of China and the boards just look great. I normally use the KiCad Aisler Push plugin in my workflow anyway just to run some sanity checks on the board design in addition to the checks ran in KiCad.

Some PCB design notes generated by Aisler

In this screenshot it's just warning me about the holes for the plastic pins in the footprint of the USB-C connector. It doesn't matter whether those are copper plated or not so I just ignore this warning. After receiving the boards I've also checked this and the holes did not get copper plated at all.

I also noticed there was a discount for using components from Würth Elektronik on the board so naturally I took that as a challenge to see how for I can push that. It turns out that in this design the answer is pretty far: The easy thing is to use WE part numbers for the passives on the board.

The second thing I replaced is the connectors on the board. I was already pretty happy I did not have to make custom footprints for once because that's just a time consuming job. After comparing a lot of footprints it turns out that WE makes an RJ12 connectors with the exact footprint I had already made. The USB-C connector is a bit more difficult since I'm used to picking the one available at JLCPCB that also happens to have a footprint already in KiCad. It turns out that the Würth Elektronik USB-C connector fits on the USB_C_Receptacle_GCT_USB4105-xx-A_16P_TopMnt_Horizontal footprint in KiCad. I guess there's simply not too many different ways to make spec compliant USB-C connectors.

The soldering

So the board is relatively straightforward to solder, it doesn't have that many parts and I decided to not pick the tiniest SMT parts I could find. If you're not familiar with SMT parts, they get delivered on reels and if you order small quantities you get a smaller cut-off portion of a reel. Here's some capacitors for example:

These are the 22uF 0805 capacitors I've ordered for the ESP32 power rail. The rest of the passives are the slightly smaller 0603 packages. On the right of the board you can see the pads for the 0805 capacitor and the 0603 capacitor side by side. In this case the Würth capacitors came in a neat transparent piece of tube. If you order a full reel of cheap resistors they usually come on paper tape which looks like this:

This is 5000 resistors in 0603 size

This is probably one of few times I've ever ordered 5000 of something. Even the bigger parts like the ESP32 modules come on a reel:

This strip holds 20 cores of pure computing power :D

This soldering itself was quite time consuming but not that hard. The secret is to just use a lot of flux and not use plastic tweezers that melt while soldering.

Maybe after some experience I can even get them on straight. Supposedly it's a lot easier and neater to use solder paste and stencils to do this so I also ordered the stencils for this PCB. I don't have the soldering paste and extra equipment for it though so that experiment will have to wait a bit longer.

The case

To have this look somewhat neat when it's finished I also need a case for the board. The need for cases has annoyed me several times already when designing PCBs. The options are either grabbing an off the shelf project box which is usually expensive and never fits exactly with the design or just 3D printing a case which involves me angrily staring at CAD software to figure out how to make things fit and match up to my PCB.

I've already written a blog post about my solution here which is TurboCase. It's a tool that automatically generates a case based on the PCB file from KiCad. I wrote the software specifically because I needed it for this PCB so naturally the case generated by it works perfectly for this project :)

TurboCase generated case for this board

Since writing the blog post about the details on how TurboCase works I've added support for TurboCase-specific footprints. I now have a global KiCad library that holds footprints that are designed to go on the User.6 layer to add prefab features to the final OpenSCAD file like screw holes in the case and the hole for by USB-C connectors.

The contents of the User.6 layer of my board design

Here you can see the drawing on the User.6 layer of the DSMR module. It contains several features:

The outline of the case drawn using the graphic elements in KiCad. This functions the same way as the board outline normally works on the Edge.Cuts layer except it describes the outline of the inside of the case.
Three mounting holes that fit countersunk M3 sized screws to mount the case on the wall.
A M3-sized keyhole screw-mounting thingy so the board can be mounted in a less permanent way.
The cube with the line in it marks a spot where turbocase makes an USB-C shaped hole matched up perfectly to the USB-C connector on the board.
The RJ12 connectors already cross the border of the case in KiCad so there will be holes generated for them automatically in the case based on the connector outline on the fabrication layer.
The 3 MountingHole footprints on the PCB will be used to generate mounting posts for the PCB inside the case, the main reason why I started making this tool to make sure those things align perfectly every time.

I did a few iterations on the footprint library and did a series of test prints of the case to make sure everything is correct and I can happily report that everything just matches up perfectly.

The few days I spend building and testing this tool now narrows down the whole process from kicad to physical product down to:

Draw outline in KiCad
Run TurboCase on the .kicad_pcb file to get my OpenSCAD file.
Export the OpenSCAD file to .STL and 3D print it.

I hope this utility is useful for more people and I would love to see how it performs with other board designs!

The software side

The more annoying part of the whole process was dealing with the software. For the ESP8266 I've always used the Arduino IDE. This is mainly because I did not want to deal with the xtensa toolchain required to build software for these chips. I don't do complicated things with these modules and with the Arduino IDE they simply always just worked.

Switching over to the ESP32 did mean that stuff stopped working automatically for me sadly, the base software worked great but I had some troubly getting the MQTT library to work. The dependency resolution in the Arduino IDE is very simple and it does not account for having different dependencies with the same include name so I tried PlatformIO for this project.

The whole setup process for PlatformIO was pretty smooth until I got to platform selection. There's a few different variations of the ESP32-S3 module available and even a few more specific variations of the WROOM-1 module I put on my board. I dit however commit the sin of picking the 4MB flash version of the board. This is the only one that doesn't have a specific config in PlatformIO and the merge request for it was denied based on that it would be easy to override the platform config with a json file. I did not find how to do that easily so I picked one of the random dev boards that used the wroom-1 module with 4MB flash instead to use whatever changes they have made to the JSON file for the board to make this work.

Now for actually implementing the P1 protocol: This is deceptively hard. The way the protocol works is that you pull the data request line high on the connector and then the energy meter will start spewing out datagrams over the serial port on either a 1 second or 10 second interval depending on the protocol version. These datagrams look something like this:

/XMX5LGBBFG10

1-3:0.2.8(42)
0-0:1.0.0(170108161107W)
0-0:96.1.1(4530303331303033303031363939353135)
1-0:1.8.1(002074.842*kWh)
1-0:1.8.2(000881.383*kWh)
1-0:2.8.1(000010.981*kWh)
1-0:2.8.2(000028.031*kWh)
0-0:96.14.0(0001)
1-0:1.7.0(00.494*kW)
1-0:2.7.0(00.000*kW)
0-0:96.7.21(00004)
0-0:96.7.9(00003)
1-0:32.32.0(00000)
1-0:32.36.0(00000)
0-0:96.13.1()
0-0:96.13.0()
1-0:31.7.0(003*A)
1-0:21.7.0(00.494*kW)
1-0:22.7.0(00.000*kW)
0-1:24.1.0(003)
0-1:96.1.0(4730303139333430323231313938343135)
0-1:24.2.1(170108160000W)(01234.000*m3)
!D3B0

This looks to be a relatively simple line based protocol. The fields are identified by a numeric code called the "OBIS" code. This stands for OBject Identification System. Then there's values added in parenthesis after it. The difficulty in parsing this is that there can be multiple values in parentheses added after each OBIS field. This would not be terrifically hard to parse if the DSMR spec had nailed down the encoding a bit more but it merly specifies that the field ends in a newline.

In my case some of the values themselves also contain newlines so this breaks the assumption you can parse this based simply based on lines and that combined by the variable (but unspecified) amount of values means that the parser code for this becomes quite nasty.

Luckily I got to use one of the features I gained by switching to PlatformIO: Unit testing. The whole DSMR parser is moved to it's own class in the codebase and only gets a Stream reference to parse the DSMR datagrams from. This means I don't have to sit with my laptop in the hallway debugging the parser and I can check if the parser works with dumps from various smart energy meters.

This all glued together means that the DSMR module will get the data from the P1 port and then sent it out over WiFi to my MQTT server with a separate topic for every field. And the result:

Grafana showing the power consumption and production as measured by the smart energy meter

The source for the hardware and firmware is available at https://git.sr.ht/~martijnbraam/dsmr-module and the board itself is also available on https://aisler.net/p/IWCEGPWL

Automatic case design for KiCad

Martijn Braam — Wed, 15 May 2024 19:48:27 -0000

I don't generally get along great with CAD software with the exception of KiCad. I guess the UX for designing things is just a lot simpler when you only have 2 dimensions to worry about. After enjoying making a PCB in KiCad the annoying for me is always getting a case designed to fit the board.

If I'm lucky I don't need many external holes to fit buttons or connectors and if I'm really lucky the mounting holes for the board are even in sensible locations. I wondered if there was a quick way to get the positions of the mounting holes into some 3D CAD software to make the mounting posts in the right position without doing math or measuring.

But what's even better than importing mounting hole locations? Not having to build the case at all!

Turbocase

So the solution is just several hundred lines of Python code. I've evaluated a few ways of getting data out of KiCad to mess with it and initially the kicad-cli tool looked really promising since it allows exporting the PCB design to several vector formats without launching KiCad. After exporting a few formats and seeing how easy it would be to get the data into Python I remembered that the PCB design files are just s-expressions, so the easiest way is just reading the .kicad_pcb file directly.

A snippet of the .kicad_pcb source file

So with this there's several pieces of data the tool extracts for the case design:

The footprints with MountingHole in the name to place posts for threaded metal inserts in my 3d prints
A case outline from a user layer. This has the same semantics as the Edge.Cuts layer except that it defines the shape of the inner edge of the case.
Information about connectors to make holes in the case and to have placeholders in the final OpenSCAD file for easier modifications.

The locations of the mounting holes is pretty easy to get up and running. By iterating over the PCB file the tool saves all the footprints that are mounting holes and for each of those footprints it locates the pad with the largest hole in it and saves that to make a mounting post in the case.

In my test PCB I already had some nice edge-cases to deal with since the specific footprint I used has vias in it which also counts as pads. The outer side of the pad is used as the diameter of the mounting post that will be generated and inside that a hole will be created that fits the bag of threaded metal inserts I happen to have here.

Case outlines

To make the actual case I initially planned to just grab the PCB outline and slightly enlarge it as a template. This turns out to have a few obvious flaws, for example the ESP32 module that I have hanging over the edge of the PCB. Grabbing the bounding box of the entire design would also be a quick fix but that would mean the case would be way too large again due to the keepout area of the ESP32 footprint. The solution is manually defining the shape of the case since this gives the most flexibility.

I picked the User.6 layer as the case outline layer since it has a neat blue color in this KiCad theme. The semantics for the case outline is the same as what you'd normally do on the Edge.Cuts layer except that this defines the inner wall of the case. The turbocase utility will then add a wall thickness around it to make the 3D model of the case.

Connector holes

What use is a case without any holes for connections? This turned out to be a more difficult issue. For the connectors I would actually need some height information and KiCAD is still very much 2D. It is possible to link a 3D model to the footprint which is great to see how the final board will look like:

Sadly using this 3D model data would be quite difficult for turbocase since it would require an importer for all the various supported 3D model formats and then a way to grab the outline of a slice of the model.

So I picked the uglier but simpler solution. Just give the connector a height in some metadata and treat them as cubes. For the footprint I use the bounding box of the F.Fab layer which should correspond relatively closely to the size of the connector. To store the 3rd dimension I simply added a property to the connectors I wanted to be relevant to the case design:

Exporting the case

I decided to export the cases as OpenSCAD files. This is mostly because these are simple text files I can generate and I already have some experience with OpenSCAD design.

A large part of the generated file is boilerplate code for doing the basic case components. After that it will export the case outline as a polygon and do the regular OpenSCAD things to it to make a 3D object.

standoff_height = 5;
floor_height = 1.2;
pcb_thickness = 1.6;
inner_height = standoff_height + pcb_thickness + 11.5;
pcb_top = floor_height + standoff_height + pcb_thickness;

box(1.2, 1.2, inner_height) {
	polygon(points = [[102,145], [140,145], .... ]);
}

The pcb_thickness is also one of the variables exported from the KiCad PCB file and the box(wall, bottom, height) module creates the actual basic case.

For the connector a series of cubes is generated and those are substracted from the generated box:

// J1 Connector_USB:USB_C_Receptacle_GCT_USB4105-xx-A_16P_TopMnt_Horizontal USB 2.0-only 16P Type-C Receptacle connector
translate([104.15, 119, pcb_top])
  rotate([0, 0, 90])
    #connector(-4.47,-3.675,4.47,3.675,3.5100000000000002);

// J2 Connector_RJ:RJ12_Amphenol_54601-x06_Horizontal 
translate([115, 131.9, pcb_top])
  rotate([0, 0, 90])
    #connector(-3.42,-1.23,9.78,16.77,11.7);

// J3 Connector_RJ:RJ12_Amphenol_54601-x06_Horizontal 
translate([127.11, 138.26, pcb_top])
  rotate([0, 0, 270])
    #connector(-3.42,-1.23,9.78,16.77,11.7);

This also has some extraced metadata from the PCB to figure out what's what when editing the .scad file. The connector module is a simple helper for generating a cube from a bounding box that accounts for the origin of the connector not being in the center.

Finally the mounting posts are added to the case as simple cylinders:

// H1
translate([105, 108, 1.2])
	mount(3.4000000000000004, 6.4, 5);

// H3
translate([121, 140, 1.2])
	mount(3.4000000000000004, 6.4, 5);

// H2
translate([137, 108, 1.2])
	mount(3.4000000000000004, 6.4, 5);

The extracted information from the mounting hole is the 3.2mm drill diameter and the 6.2mm pad diameter. The inner hole is expanded by 0.2mm in this case to make the holes work with the metal inserts.

Further improvements

There's a lot of neat things that could be added to this. The major one being a lid for the case. This also would need a bunch more configurability to deal with mounting mechanisms for the lid like screw holes or some clips.

The system could also be extended by producing a footprint library specifically for turbocase to signify where to add specific features to the case. This could be things like cooling holes, led holes. Maybe some fancier connector integration.

The output from turbocase also suffers from the same issue a lot of OpenSCAD designs suffer from: it's very hard to add chamfers and fillets to make a less rectangular case. That would require someone with more OpenSCAD knowledge to improve the generated output.

The source code for the turbocase tool is available at https://sr.ht/~martijnbraam/turbocase/ and the utility is available on pypi under the turbocase name.

Megapixels contributions

Martijn Braam — Sat, 11 May 2024 14:45:17 -0000

I've been working on the code that has become libmegapixels for a bit more as a year now. It has taken several thrown-away codebases to come to a general architecture I was happy with and it it has been quite a task to split off media pipeline tasks from the original Megapixels codebase.

After staring at this code for many months I thought I've made libmegapixels a nearly perfect little library. That's the problem with working on a codebase without anyone else looking at it.

About two weeks ago libmegapixels and the general Megapixels 2.x codebase had it's first contact with external contributors and that has put a spotlight on all the low hanging fruit in documentation and codebase issues. A great example of that is this commit:

diff --git a/src/parse.c b/src/parse.c
index bfea3ec..93072d0 100644
--- a/src/parse.c
+++ b/src/parse.c
@@ -403,6 +403,8 @@ libmegapixels_load_file(libmegapixels_devconfig *config, const char *file)
 	config_init(&cfg);
 	if (!config_read_file(&cfg, file)) {
 		fprintf(stderr, "Could not read %s\n", file);
+		fprintf(stderr, "%s:%d - %s\n",
+			config_error_file(&cfg), config_error_line(&cfg), config_error_text(&cfg));
 		config_destroy(&cfg);
 		return 0;
 	}

A simple patch that massively improves the usablility for people writing libmegapixels config files: Actually printing the parsing errors from libconfig when a file could not be read. Because I generally run libmegapixels through the IDE and have all the syntax highlighting etc set up for the files I simply haven't triggered this codepath enough to actually implement this part.

These last two weeks there have also been some significantly more complicated fixes like tracing segfault issues in Megapixels 2.x which helps a lot with getting the new codebase ready for daily use. Figuring out some API issues in libmegapixels like not correctly setting camera indexes in the returned data. Also the config files have now been updated to work with the latest versions of the PinePhone Pro kernel instead of the year old build I've been developing against.

Video recording

I've been saying for a long time that video recording on the PinePhone won't be possible, especially not to the level of support on Android and iOS due to hardware limitations. The only real hope for proper video recording would be that someone gets H.264 hardware encoding to work on the A64 processor.

I can happily report that I was wrong. Pavel Machek has made significant progress in PinePhone video recording with a few large contributions that implement the UI bits to add video recording. A new second postprocessing pipeline for running external video encoding scripts just like Megapixels already lets you write your own custom scripts for processing the raw pictures into JPEGs.

Video recording is a complicated issue though, mainly due to the sheer amount of data that needs to be processed to make it work smoothly. On the maximum resultion of the sensor in the PinePhone the framerate isn't high enough for recording normal videos (unless you enjoy 15fps video files) but on lower resolutions the pipeline can run at normal video framerates. The maximum framerates from the sensor for this are 1080p at 30fps and 720p at 60fps.

For 720p60 the bandwidth of the raw sensor data is 442 Mbps and for 1080P30 this is 497 Mbps. This is a third of the expected bandwidth because the raw sensor data is essentially a greyscale image where every pixels has a different color filter in front. This is too much data to write out to the eMMC or SD card to process later and the PinePhone also struggles already to encode 720p30 video live without even running a desktop environment.

There are two implementations of video recording right now. One that saves the raw DNG frames to a tmpfs since RAM is the only thing that can keep up with the data rate. This should give you roughly 30 seconds of video recording capabilities and after that recording time it will take a while to actually encode the video.

Pavel has posted an example of this video recording on his mastodon.

The second way is putting the sensor in a YUV mode instead of raw data. This gives worse picture quality on the sensor in the PinePhone but the data format matches more closely to the way frames are stored in video files so the expensive debayer step can be skipped while video recording. This together with encoding H.264 video with the ultrafast preset should make it just about possible to record real-time encoded video on the PinePhone.

Many thanks

It's great to see contributions to Megapixels 2 and libmegapixels. It's a big step towards getting the Megapixels 2.x codebase production ready and it's simply a lot more fun to work on a project together with other people.

It's great to have contributors working on the UI code, the camera support fixes for devices and the many bugfixes to the internals. It's also very helpful to actually have issues created by people building and testing the code on other distributions. This already ironed out a few issues in the build system.

There also has been some nice contributions to the Megapixels 1.x codebase, all of those should by now already have been merged into your favorite PinePhone distribution :)

The last few Megapixels update blogposts have all been around Megapixels 2.x and the supporting libraries so none of the improvements are immediately usable by actual PinePhone{,Pro} and Librem 5 users until there is an actual release. It will take a bunch more polish until feature parity with Megapixels 1.x is reached.

Moving to a RTOS on the RP2040

Martijn Braam — Mon, 06 May 2024 15:58:55 -0000

I've been working on a bunch of small projects involving microcontrollers. Currently a lot of them are based around the Raspberry Pi Pico boards because I like the development experience of those a lot. They have a decent SDK and cheap hardware to get started and the debugger works with gdb/openocd so it just integrates in all IDEs that support that.

One of my current projects is making a fancy hardware controller for a bunch of video equipment I use. The main things that will be controlled are two PTZ cameras (those are cameras that have motors to move them). One stationary camera and the video switching equipment that that's hooked up to.

Currently the control of the control of the PTZ cameras is done with an unbranded panel that looks suspiciously like the Marshall VS-PTC-200:

(Image from marshall-usa.com)

The performance of this controller is simply not very great, especially for the price. It was a €650 device several years ago and for that money it has very annoying squishy buttons and the cheapest analog joystick you could find. Most of the buttons are also not functional with the cameras in use since this seems to be optimized for security cameras. This connects to the cameras over an RS-485 bus.

The second thing I want my panel to do is very basic ATEM video switcher control. Currently that's fully done using the software panel on the computer because the panels from Blackmagic Design are very expensive.

There's a tiny cheaper one now though. (from blackmagicdesign.com)

After a bit of designing I figured the most minimal design I can get away with is 9 buttons, the joystick and a display for the user interface. The hardware design has gone through several iterations over the last year but I now have some PCBs with the 9 RGB buttons on it, the $10 joystick that was also in the Marshall-clone panel and to interface with the outside world it has the TP8485E to communicate with the cameras over RS-485 and a Wiznet W5500 module to communicate with the video switcher over ethernet.

This includes a bunch of "oops-the-wrong-pinout" fixes...

After a lot of fixing of the board I had made I now have all the hardware parts functional, but the difficult part of this project is the software.

Initial software

I first started creating the software like I do all the RP2040 based projects. A cmake project that pulls in the pico-sdk. To make anything work at all I dedicated the second core of the pico to dealing with the Wiznet module and the first core then handles all the user interface I/O. This worked fine to blink some leds and I did implement a DHCP client that ran on the second core. It did make implementing the rest of the system a lot more complicated. There's simply a lot of things that need to happen at once:

Draw an user interface on the display that's somewhat smooth
Send out VISCA commands over the RS-485 interface
Respond to button presses
Keep the entire network stack alive with multiple connections

There's a bunch of things that need to happen on the network, the first of which is some actually standards complicant DHCP support. This would require keeping track of the expire times and occasionally talk to the DHCP server to keep the lease active. The second background task is making mDNS work. The ATEM video switcher IP can be autodiscovered using DNS-SD and it would be great to also announce the existence of the control panel.

The ATEM protocol itself is also one of the harder parts to get right, the protocol itself is pretty simple but it does involve sometimes receiving a lot of data that exceeds the buffer size of the Wiznet module and the protocol has a very low timeout for disconnection for when you stop sending UDP datagrams to the ATEM.

This all made me decide that it's probably better to switch to an RTOS for this project.

FreeRTOS

The first project I've looked into is FreeRTOS. This is technically already bundled inside the pico-sdk but all tutorials I've found for this download a fresh copy anyway so that's what I did. FreeRTOS seems to be the simplest RTOS I've looked at from this list, the main thing it provides is the RTOS scheduler and some communication between tasks. The simplest way I can show it is with some code:

#include "FreeRTOS.h"

TaskHandle_t button_task = NULL;
TaskHandle_t led_task = NULL;
QueueHandle_t led_queue = NULL;

void buttonTask(void *param) {
    while (1) {
      bool state = get_button_pressed();
      xQueueSend(led_queue, &state, 0);
    }
}

void ledTask(void *param) {
    while (1) {
        bool state;
        if(xQueueReceive(led_queue, &state, portMAX_DELAY)) {
            gpio_put(LED_PIN, state);
        }
    }
}

int main() {
    xTaskCreate(buttonTask, "Button", 128, NULL, 2, &button_task);
    xTaskCreate(ledTask, "Led", 128, NULL, 2, &led_task);
    vTaskStartScheduler();
    // Code will never reach here
    return 0;
}

Both the buttonTask and the ledTask function will seem to run in parallel and there's a few IPC systems to move data between the various tasks. The code above is not functional but I stripped it down to get the general usage across.

I've used this for a few days to make an enormous mess of my codebase. I have created several tasks in my test project:

The buttonsTask that polls the i2c gpio expander to check if buttons have been pressed and then put a message on the button queue.
The ledTask that sets the right RGB color on the right button by putting a message on the ledQueue;
The mainTask that runs the main loop of the project that updates the state based on the button presses.
The networkTask that communicates with the Wiznet module.
The dhcpTask that is spawned by the networkTask when a network cable is plugged in.
The mdnsTask that is spawned by the dhcpTask once an ip address is aquired.
the atemTask that is spawned by the mdnsTask when it gets a response from an ATEM device.
the viscaTask that does nothing but should send data out the RS-485 port.

This is a lot of tasks and the hardware doesn't even do anything yet except appear on the network.

I ran into a few issues with FreeRTOS. The main annoying one is that printf simply caused things to hang every single time which makes debugging very hard. Sure the gdb debugger works but it's not neat for dumping out DHCP traffic for example.

The FreeRTOS also doesn't seem to provide any hardware abstraction at all which means all the code I wrote to communicate with the various chips is not easily re-used.

After a few days I created a new clean FreeRTOS project and started porting the various functionalities from the previous version over to try to get a cleaner and more manageable codebase but ended up giving up because blind debugging because there's no serial output is quite annoying. I decided to look what the alternatives have to offer.

Apache NuttX

Another seemingly popular RTOS is NuttX. This project seems a lot closer to what you'd expect from a regular operating system. It makes your microcontroller look like an unix system.

First thing the tutorial tells me to do is fetching the pico-sdk and set the environment variable. No problem, I already have the sdk in /usr/share and that environment variable already exists on my system. Suprisingly this made the build fail because NuttX decides that it really needs to overwrite the version.h file in my pico-sdk for which it doesn't have permissions... why...

After doing the initial setup of building a minimal NuttX firmware for my board I connected to the serial port and was greeted by an actual shell.

nsh> uptime
00:01:34 up  0:01, load average: 0.00, 0.00, 0.00
nsh> uname
NuttX
nsh> uname -a
NuttX 12.5.1 9d6e2b97fb May  6 2024 15:18:54 arm raspberrypi-pico

It looks like I'd just be able to write an app for this operating system and have it auto-launch on boot. Since this tries to do the Unix thing it also has a filesystem of course so the hardware has FS abstractions like /dev/i2c0 and /dev/adc0.

One thing I liked a lot was that it's build around menuconfig/Kconfig which I'm already used to for Linux development. This also means there's an actual hardware driver system and the GPIO expander chip I've used for the buttons already had a driver. The menuconfig system also allows me to configure the pin muxing of the rp2040 chip so I don't have to keep constants around with pin numbers and do a bunch of hardware setup to make my i2c bus work. I can just go into the menuconfig and tell it that i2c0 of the pico is used and that it's on two specific pins. I've also enabled the i2c testing utility as one of the apps that will be build into the firmware.

nsh> i2c dev 0 79
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- -- -- --                   
nsh>

Well uuuuh... yup the basics aren't working. I've spend a bit of time going through the rp2040 setup code and the various i2c related menuconfig options but it seems like this just doesn't really work...

Update: This actually works fine but I managed to make a config mistake that broke the i2c bus, I have gotten the board to work more now on NuttX and will write a more detailed update post in the future for this. Consider the rest of this section as most likely invalid.

I also have not figured out yet how I can tell NuttX that my gpio buttons are behind the gpio extender, or how to actually link the gpio extender to my non-functional i2c bus.

Another thing that annoyed me is that I had to re-clone the nuttx repository multiple times simply because sometimes one of the configure.sh commands would fail which would leave the repository in an inconsistent state and the distclean command wouldn't work because the repository was in an inconsistent state. Really the classic "configure.sh: you are already configured; distclean: you are not configured yet"

Unix-like seems great at first glance, but I don't really want to deal with filesystem paths on a microcontroller for a pretend filesystem. I also don't need a shell in my production system, it should just run my code.

Zephyr

So next on the list is Zephyr. This provides a python utility to set up a project which should make things a bit easier, or it's a sign something is terribly overcomplicated.

The very first thing this project does is pull in 5GB of git repositories which includes the entire HAL library for every chip under the sun. The second thing it does is for some reason mess with my user-wide cmake stuff on my system.

After that the tutorial told me to install the Zephyr SDK:

The Zephyr Software Development Kit (SDK) contains toolchains for each of Zephyr’s supported architectures, which include a compiler, assembler, linker and other programs required to build Zephyr applications.

It also contains additional host tools, such as custom QEMU and OpenOCD builds that are used to emulate, flash and debug Zephyr applications.

Yeah no thanks, I have already several perfectly fine ARM toolchains and I don't really want to either build or fetch precompiled compilers for every architecture Zephyr supports, lets see if I can get away with not installing this.

After some messing around I figured out how to get away with it. There need to be two command line options set for cross compiling:

$ export ZEPHYR_TOOLCHAIN_VARIANT=cross-compile
$ export CROSS_COMPILE=/usr/bin/arm-none-eabi- 
$ west build -p always -b sparkfun_pro_micro_rp2040 samples/basic/blinky

One thing I also found out is that the Raspberry Pi Pico is not actually supported, only other boards that have the same SoC. No worries, these boards are practically the same. The very second issue I hit is that the blinky demo doesn't build because it requires led0 to be defined to have something to blink.

It turns out the Sparkfun pro Micro RP2040 does not actually have a simple gpio led to blink but a ws2812B adressable led.

So I started following the custom board manual which told me to copy a random other board because that's how it always goes. Maybe if you already have a meta tool to set-up a project make it create this scaffolding.

In the end I did not manage to build for my board because it simply wouldn't start to exist after fixing all the errors and warnings in the build.

Conclusion

Well at least with FreeRTOS I managed to building some of my own application. I guess I have to follow the online instructions of replacing printf with another printf implementation and make sure to call the different function everywhere.

I'll probably continue on trying to get FreeRTOS to do the things I want since it's the only one that can be simply integrated in your own environment instead of the other way around.

Bootstrapping Alpine Linux without root

Martijn Braam — Wed, 20 Mar 2024 23:50:30 -0000

Creating a chroot in Linux is pretty easy: put a rootfs in a folder and run the sudo chroot /my/folder command. But what if you don't want to use superuser privileges for this?

This is not super simple to fix, not only does the chroot command itself require root permissions but the steps for creating the rootfs in the first place and mounting the required filesystems like /proc and /sys require root as well.

In pmbootstrap the process for creating an installable image for a phone requires setting up multiple chroots and executing many commands in those chroots. If you have the password timeout disabled in sudo you will notice that you will have to enter your password tens to hundreds of times depending on the operation you're doing. An example of this is shown in the long running "pmbootstrap requires sudo" issue on Gitlab. In this example sudo was called 240 times!

Now it is possible with a lot of refactoring to move batches of superuser-requiring commands into scripts and elevate the permissions of that with a single sudo call but to get this down to a single sudo call per pmbootstrap command would be really hard.

Another approach

So instead of building a chroot the "traditional" way what are the alternatives?

The magic trick to get this working are user namespaces. From the Linux documentation:

User namespaces isolate security-related identifiers and attributes, in particular, user IDs and group IDs (see credentials(7)), the root directory, keys (see keyrings(7)), and capabilities (see capabilities(7)). A process's user and group IDs can be different inside and outside a user namespace. In particular, a process can have a normal unprivileged user ID outside a user namespace while at the same time having a user ID of 0 inside the namespace; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace.

It basically allows running commands in a namespace where you have UID 0 on the inside without requiring to elevate any of the commands. This does have a lot of limitations though which I somehow all manage to hit with this.

One of the tools that makes it relatively easy to work with the various namespaces in Linux is unshare. Conveniently this is also part of util-linux so it's a pretty clean dependency to have.

Building a rootfs

There's enough examples of using unshare to create a chroot without sudo but those all assume you already have a rootfs somewhere to chroot into. Creating the rootfs itself has a few difficulties already though.

Since I'm building an Alpine Linux rootfs the utility I'm going to use is apk.static. This is a statically compiled version of the package manager in Alpine which allows building a new installation from an online repository. This is similar to debootstrap for example if you re more used to Debian than Alpine.

There's a wiki page on running Alpine Linux in a chroot that documents the steps required for setting up a chroot the traditional way with this. The initial commands to aquire the apk.static binary don't require superuser at all, but after that the problems start:

$ ./apk.static -X ${mirror}/latest-stable/main -U --allow-untrusted -p ${chroot_dir} --initdb add alpine-base

This creates the Alpine installation in ${chroot_dir}. This requires superuser privileges to set the correct permissions on the files of this new rootfs. After this there's two options of populating /dev inside this rootfs which both are problematic:

$ mount -o bind /dev ${chroot_dir}/dev
mounting requires superuser privileges and this exposes all your hardware in the chroot

$ mknod -m 666 ${chroot_dir}/dev/full c 1 7
$ mknod -m 644 ${chroot_dir}/dev/random c 1 8
... etcetera, the mknod command also requires superuser privileges

The steps after this have similar issues, most of them for mount reasons or chown reasons.

There is a few namespace options from unshare used to work around these issues. The command used to run apk.static in my test implementation is this:

$ unshare \
    --user \
    --map-users=10000,0,10000 \
    --map-groups=10000,0,10000 \
    --setuid 0 \
    --setgid 0 \
    --wd "${chroot_dir}" \
    ./apk-tools-static -X...etc

This will use unshare to create a new userns and change the uid/gid inside that to 0. This effectively grants root privileges inside this namespace. But that's not enough.

If chown is used inside the namespace it will still fail because my unprivileged user still can't change the permissions of those files. The solution to that is the uid remapping with --map-users and --map-groups. In the example above it sets up the namespace so files created with uid 0 will generate files with the uid 100000 on the actual filesystem. uid 1 becomes 100001 and this continues on for 10000 uids.

This again does not completely solve the issue though because my unprivileged user still can't chown those files, doesn't matter if it's chowning to uid 0 or 100000. To give my unprivileged user this permission the /etc/subuid and /etc/subgid files on the host system have to be modified to add a rule. This sadly requires root privileges once to set up this privilege. To make the command above work I had to add this line to those two files:

martijn:100000:10000

This grants the user with the name martijn the permission to use 10.000 uids starting at uid 100.000 for the purpose of userns mapping.

The result of this is that the apk.static command will seem to Just Work(tm) and the resulting files in ${chroot_dir} will have all the right permissions but only offset by 100.000.

One more catch

There is one more complication with remapped uids and unshare that I've skipped over in the above example to make it clearer, but the command inside the namespace most likely cannot start.

If you remap the uid with unshare you get more freedom inside the namespace, but it limits your privileges outside the namespace even further. It's most likely that the unshare command above was run somewhere in your own home directory. After changing your uid to 0 inside the namespace your privilege to the outside world will be as if you're uid 100.000 and that uid most likely does not have privileges. If any of the folders in the path to the executable you want unshare to run for you inside the namespace don't have the read and execute bit set for the "other" group in the unix permissions then the command will simply fail with "Permission denied".

The workaround used in my test implementation is to just first copy the executable over to /tmp and hope you at least still have permissions to read there.

Completing the rootfs

So after all that the first command from the Alpine guide is done. Now there's only the problems left for mounting filesystems and creating files.

While /etc/subuid does give permission to use a range of uids as an unprivileged user with a user namespace it does not give you permissions to create those files outside the namespace. So the way those files are created is basically the complicated version of echo "value" | sudo tee /root/file:

$ echo "nameserver a.b.c.d" | unshare \
    --user \
    --map-users=10000,0,10000 \
    --map-groups=10000,0,10000 \
    --setuid 0 \
    --setgid 0 \
    --wd "${chroot_dir}" \
    sh -c 'cat > /etc/resolv.conf'

This does set-up and tear down the entire namespace for every file change or creation which is a bit inefficient, but inefficient is still better than impossible. Changing file permissions is done in a similar way.

To fix the mounting issue there's the mount namespace functionality in Linux. This allows creating new mounts inside the namespace as long as you still have permissions on the source file as your unprivileged user. This effectively means you can't use this to mount random block devices but it works great for things like /proc and loop mounts.

There is a --mount-proc parameter that will tell unshare to set-up a mount namespace and then mount /proc inside the namespace at the right place so that's what I'm using. But I still need other things mounted. This mounting is done as a small inline shell script right before executing the commands inside the chroot:

$ unshare \
    --user \
    --fork \
    --pid \
    --mount \
    --mount-proc \
    --map-users=10000,0,10000 \
    --map-groups=10000,0,10000 \
    --setuid 0 \
    --setgid 0 \
    --wd "${chroot_dir}" \
    -- \
    sh -c " \
    	mount -t proc none proc ; \
        touch dev/zero ; \
        mount -o rw,bind /dev/zero dev/zero ;\
        touch dev/null ; \
        mount -o row,bind /dev/null dev/null ;\
        ...
        chroot . bin/sh \
        "

The mounts are created right between setting up the namespaces but before the chroot is started so the host filesystem can still be accessed. The working directory is set to the root of the rootfs using the --wd parameter of unshare and then bind mounts are made from /dev/zero to dev/zero to create those devices inside the rootfs.

This combines the two impossible options to make it work. mknod can still not work inside namespaces because it is a bit of a security risk. mount'ing /dev gives access to way too many devices that are not needed but the mount namespace does allow bind-mounting the existing device nodes one by one and allows me to filter them.

Then finally... the chroot command to complete the journey. This has to refer to the rootfs with a relative path and this also depends on the working directory being set by unshare since host paths are breaking with uid remapping.

What's next?

So this creates a full chroot without superuser privileges (after the initial setup) and this whole setup even works perfectly with having cross-architecture chroots in combination with binfmt_misc.

Compared to pmbootstrap this codebase does very little and there's more problems to solve. For one all the filesystem manipulation has to be figured out to copy the contents of the chroot into a filesystem image that can be flashed. This is further complicated by the mangling of the uids in the host filesystem so it has to be remapped while writing into the filesystem again.

Flashing the image to a fastboot capable device should be pretty easy without root privileges, it only requires an udev rule that is usually already installed by the android-tools package on various Linux distributions. For the PinePhone flashing happens on a mass-storage device and as far as I know it will be impossible to write to that without requiring actual superuser privileges.

The code for this is in the ~martijnbraam/ambootstrap repository, hopefully in some time I get this to actually write a plain Alpine Linux image to a phone :D

BrixIT Blog

Building a browser game based on KiCad

The game

Internals

Conclusion

Megapixels 2.0: Small fixes and GTK breakage

GTK throwing a wrench into the development process

The future

After-FOSDEM videobox updates

Next up after FOSDEM

Audio firmware changes

VLAN support for the network switch

Hardware improvements

See also

BodgeOS pt.4: A working browser

The path to Firefox is paved with many dependencies

Desktop audio

Building Firefox

Increasing the difficulty and building for ARM

Continuing on

BodgeOS pt.3: Graphical desktop

The desktop

Sway

Font rendering

First attempt at booting

Sway starts

Branding

Continuing on

Megapixels 2.0 alpha release

Megapixels 2.0.0-alpha1

Many thanks

The release

BodgeOS pt.2: Running on real hardware

The Initramfs

Debugging the boot

Cleaning up

Next steps

Conjuring a Linux distribution out of thin air

Picking distros once again

Artisanal home-grown Linux

The bootstrap build

Building packages

Getting the first install to run

Beyond the first run

So why?

Building a timeseries database for fun

A prototype

The database

The querying

Is this better?

Making a Linux-managed network switch

Making my own gigabit network switch

Connecting it to Linux

Limitations

Building a DSMR reading board

Electrical design

The soldering

The case

The software side

Automatic case design for KiCad

Turbocase

Case outlines

Connector holes

Exporting the case

Further improvements

Megapixels contributions

Video recording

Many thanks

Moving to a RTOS on the RP2040

Initial software

FreeRTOS

Apache NuttX

Zephyr

Conclusion

Bootstrapping Alpine Linux without root

Another approach

Building a rootfs

One more catch

Completing the rootfs

What's next?