Linux - BrixIT Blog

Bootstrapping Alpine Linux without root

Martijn Braam — Wed, 20 Mar 2024 23:50:30 -0000

Creating a chroot in Linux is pretty easy: put a rootfs in a folder and run the sudo chroot /my/folder command. But what if you don't want to use superuser privileges for this?

This is not super simple to fix, not only does the chroot command itself require root permissions but the steps for creating the rootfs in the first place and mounting the required filesystems like /proc and /sys require root as well.

In pmbootstrap the process for creating an installable image for a phone requires setting up multiple chroots and executing many commands in those chroots. If you have the password timeout disabled in sudo you will notice that you will have to enter your password tens to hundreds of times depending on the operation you're doing. An example of this is shown in the long running "pmbootstrap requires sudo" issue on Gitlab. In this example sudo was called 240 times!

Now it is possible with a lot of refactoring to move batches of superuser-requiring commands into scripts and elevate the permissions of that with a single sudo call but to get this down to a single sudo call per pmbootstrap command would be really hard.

Another approach

So instead of building a chroot the "traditional" way what are the alternatives?

The magic trick to get this working are user namespaces. From the Linux documentation:

User namespaces isolate security-related identifiers and attributes, in particular, user IDs and group IDs (see credentials(7)), the root directory, keys (see keyrings(7)), and capabilities (see capabilities(7)). A process's user and group IDs can be different inside and outside a user namespace. In particular, a process can have a normal unprivileged user ID outside a user namespace while at the same time having a user ID of 0 inside the namespace; in other words, the process has full privileges for operations inside the user namespace, but is unprivileged for operations outside the namespace.

It basically allows running commands in a namespace where you have UID 0 on the inside without requiring to elevate any of the commands. This does have a lot of limitations though which I somehow all manage to hit with this.

One of the tools that makes it relatively easy to work with the various namespaces in Linux is unshare. Conveniently this is also part of util-linux so it's a pretty clean dependency to have.

Building a rootfs

There's enough examples of using unshare to create a chroot without sudo but those all assume you already have a rootfs somewhere to chroot into. Creating the rootfs itself has a few difficulties already though.

Since I'm building an Alpine Linux rootfs the utility I'm going to use is apk.static. This is a statically compiled version of the package manager in Alpine which allows building a new installation from an online repository. This is similar to debootstrap for example if you re more used to Debian than Alpine.

There's a wiki page on running Alpine Linux in a chroot that documents the steps required for setting up a chroot the traditional way with this. The initial commands to aquire the apk.static binary don't require superuser at all, but after that the problems start:

$ ./apk.static -X ${mirror}/latest-stable/main -U --allow-untrusted -p ${chroot_dir} --initdb add alpine-base

This creates the Alpine installation in ${chroot_dir}. This requires superuser privileges to set the correct permissions on the files of this new rootfs. After this there's two options of populating /dev inside this rootfs which both are problematic:

$ mount -o bind /dev ${chroot_dir}/dev
mounting requires superuser privileges and this exposes all your hardware in the chroot

$ mknod -m 666 ${chroot_dir}/dev/full c 1 7
$ mknod -m 644 ${chroot_dir}/dev/random c 1 8
... etcetera, the mknod command also requires superuser privileges

The steps after this have similar issues, most of them for mount reasons or chown reasons.

There is a few namespace options from unshare used to work around these issues. The command used to run apk.static in my test implementation is this:

$ unshare \
    --user \
    --map-users=10000,0,10000 \
    --map-groups=10000,0,10000 \
    --setuid 0 \
    --setgid 0 \
    --wd "${chroot_dir}" \
    ./apk-tools-static -X...etc

This will use unshare to create a new userns and change the uid/gid inside that to 0. This effectively grants root privileges inside this namespace. But that's not enough.

If chown is used inside the namespace it will still fail because my unprivileged user still can't change the permissions of those files. The solution to that is the uid remapping with --map-users and --map-groups. In the example above it sets up the namespace so files created with uid 0 will generate files with the uid 100000 on the actual filesystem. uid 1 becomes 100001 and this continues on for 10000 uids.

This again does not completely solve the issue though because my unprivileged user still can't chown those files, doesn't matter if it's chowning to uid 0 or 100000. To give my unprivileged user this permission the /etc/subuid and /etc/subgid files on the host system have to be modified to add a rule. This sadly requires root privileges once to set up this privilege. To make the command above work I had to add this line to those two files:

martijn:100000:10000

This grants the user with the name martijn the permission to use 10.000 uids starting at uid 100.000 for the purpose of userns mapping.

The result of this is that the apk.static command will seem to Just Work(tm) and the resulting files in ${chroot_dir} will have all the right permissions but only offset by 100.000.

One more catch

There is one more complication with remapped uids and unshare that I've skipped over in the above example to make it clearer, but the command inside the namespace most likely cannot start.

If you remap the uid with unshare you get more freedom inside the namespace, but it limits your privileges outside the namespace even further. It's most likely that the unshare command above was run somewhere in your own home directory. After changing your uid to 0 inside the namespace your privilege to the outside world will be as if you're uid 100.000 and that uid most likely does not have privileges. If any of the folders in the path to the executable you want unshare to run for you inside the namespace don't have the read and execute bit set for the "other" group in the unix permissions then the command will simply fail with "Permission denied".

The workaround used in my test implementation is to just first copy the executable over to /tmp and hope you at least still have permissions to read there.

Completing the rootfs

So after all that the first command from the Alpine guide is done. Now there's only the problems left for mounting filesystems and creating files.

While /etc/subuid does give permission to use a range of uids as an unprivileged user with a user namespace it does not give you permissions to create those files outside the namespace. So the way those files are created is basically the complicated version of echo "value" | sudo tee /root/file:

$ echo "nameserver a.b.c.d" | unshare \
    --user \
    --map-users=10000,0,10000 \
    --map-groups=10000,0,10000 \
    --setuid 0 \
    --setgid 0 \
    --wd "${chroot_dir}" \
    sh -c 'cat > /etc/resolv.conf'

This does set-up and tear down the entire namespace for every file change or creation which is a bit inefficient, but inefficient is still better than impossible. Changing file permissions is done in a similar way.

To fix the mounting issue there's the mount namespace functionality in Linux. This allows creating new mounts inside the namespace as long as you still have permissions on the source file as your unprivileged user. This effectively means you can't use this to mount random block devices but it works great for things like /proc and loop mounts.

There is a --mount-proc parameter that will tell unshare to set-up a mount namespace and then mount /proc inside the namespace at the right place so that's what I'm using. But I still need other things mounted. This mounting is done as a small inline shell script right before executing the commands inside the chroot:

$ unshare \
    --user \
    --fork \
    --pid \
    --mount \
    --mount-proc \
    --map-users=10000,0,10000 \
    --map-groups=10000,0,10000 \
    --setuid 0 \
    --setgid 0 \
    --wd "${chroot_dir}" \
    -- \
    sh -c " \
    	mount -t proc none proc ; \
        touch dev/zero ; \
        mount -o rw,bind /dev/zero dev/zero ;\
        touch dev/null ; \
        mount -o row,bind /dev/null dev/null ;\
        ...
        chroot . bin/sh \
        "

The mounts are created right between setting up the namespaces but before the chroot is started so the host filesystem can still be accessed. The working directory is set to the root of the rootfs using the --wd parameter of unshare and then bind mounts are made from /dev/zero to dev/zero to create those devices inside the rootfs.

This combines the two impossible options to make it work. mknod can still not work inside namespaces because it is a bit of a security risk. mount'ing /dev gives access to way too many devices that are not needed but the mount namespace does allow bind-mounting the existing device nodes one by one and allows me to filter them.

Then finally... the chroot command to complete the journey. This has to refer to the rootfs with a relative path and this also depends on the working directory being set by unshare since host paths are breaking with uid remapping.

What's next?

So this creates a full chroot without superuser privileges (after the initial setup) and this whole setup even works perfectly with having cross-architecture chroots in combination with binfmt_misc.

Compared to pmbootstrap this codebase does very little and there's more problems to solve. For one all the filesystem manipulation has to be figured out to copy the contents of the chroot into a filesystem image that can be flashed. This is further complicated by the mangling of the uids in the host filesystem so it has to be remapped while writing into the filesystem again.

Flashing the image to a fastboot capable device should be pretty easy without root privileges, it only requires an udev rule that is usually already installed by the android-tools package on various Linux distributions. For the PinePhone flashing happens on a mass-storage device and as far as I know it will be impossible to write to that without requiring actual superuser privileges.

The code for this is in the ~martijnbraam/ambootstrap repository, hopefully in some time I get this to actually write a plain Alpine Linux image to a phone :D

The dilemma of tagging library releases

Martijn Braam — Sun, 14 Jan 2024 16:11:17 -0000

I've been working on the libmegapixels library for quite a bit now. The base of the library is pretty solid which is configuring a V4L2 pipeline so you can get camera frames on modern ARM platforms. Most of the work on the library side is figuring the AWB/AE/AF code and how that will fit together with applications.

Due to the AAA code not working yet and the API not being being fully defined on how those parts will fit together I've been holding of on tagging an actual release on the libmegapixels library.

A lot of my projects, especially libraries, are written in Python so I've long enjoyed the luxury of APIs being duck-typed and having the possibility of adding optional arguments to methods in the future. Sadly in C libraries I can't get away with never defining the types for arguments that might change in the future or adding optional arguments.

My original plan was to tag a release on libmegapixels together with the first 2.x release of Megapixels since these pieces of software are intended to fit together but after thinking about it some more (and some convincing from other people interested in the libmegapixels release) I've decided to tag a 0.1 release.

In an ideal world I can just release code when it's fully done and tested. In this case the long time it takes to get everything ready for use will mean that potential contributors to the code will also be held back from experimenting with the codebase. Especially since a large part of libmegapixels is the config files it ships for specific hardware configurations. If I wouldn't make any releases then at some point users/developers will be forced to just ship random git commits which is a way worse situation to be in for bug tracking.

With this 0.1 release I want to make it possible to start writing config files for various phones and platforms to test camera pipelines. Hopefully this will also mean any issues with the configuration file format that people might hit will be figured out before I have to tag a "final" 1.x release.

The release

So the initial tagged release of libmegapixels:

located at https://gitlab.com/megapixels-org/libmegapixels/-/tags/0.1.0
Build instructions at https://libme.gapixels.me/building.html
Comes with absolutely no guarantee of stability for the C api of the library
Most likely the config file format is stable but might have small tweaks before the 1.x release

Hopefully this will allow people to start experimenting with the codebase and generate some feedback on it so I'm not just developing this for months and completely overfitting it to the three devices I'm testing on.

I'm planning to make a similar release for libdng soon. That library is also mostly stable but I need to fix up the last parts of the API to allow reading and writing all the required metadata.

The MNT keyboard reviewed

Martijn Braam — Tue, 19 Dec 2023 23:59:01 -0000

MNT Research is one of those few companies that actually releases open source hardware. Instead of just getting a schematic with your hardware (which is great even by itself) there's the full sources for that schematic, the Kicad parts libraries, the sources for the firmware and even documentation how to use that code.

I received my MNT Standalone Keyboard V3 a few days ago so I've been typing on it now for a bit. This is all happening while I'm recovering from covid so I hope if I read back this post in a few days it is actually somewhat coherent :)

This being a more niche product sadly does make it a bit on the expensive side. But I must say this is by far the most solid keyboard I've owned. My main keyboard on my desktop is an Das Keyboard 4 ultimate. It's a nice keyboard but it doesn't compare to the full machined aluminium frame on the MNT keyboard.

The whole keyboard is mounted on what's basically a 4mm slab of aluminium which has a nice MNT logo machined on it on the bottom

This makes the keyboard feel incredibly solid, even with the rest of the frame taken off it's practically impossible to even bend the keyboard. The second half of the frame is the top edge that screws on the base plate with 8 screws.

This is another very carefully designed aluminium part In the close-up above you can see the opening for the USB-C connection for the keyboard and the internal cutouts for the display daughterboard with the screw mounting.

The electronics

This keyboard is based around an Atmega32U4 microcontroller. This is the same keyboard PCB as what's shipped in the MNT Reform laptop so there are two connectors on this board. The USB-C connector is what's exposed on the standalone keyboard and the laptop presumably uses the USB header that's beside it.

Beside the USB header is one of the dip switches. SW36 is labeled "STANDALONE" here. This switches the board to use USB power instead of the 3.3V supplied by the laptop mainboard. The ribbon connector is the connection to the OLED display board.

On the left side of the display board there is an empty footprint for the standard Atmega programming header and a serial port that's used to connect to the laptop mainboard. Additionally there's a reset button and SW84 which has the confusing label "RG".

Thanks to the schematics being available in the manual it's easy to find that this is the switch to enable programming. The rest of the interesting parts is hidden somewhere below the display board or on the bottom side of the PCB possibly. I have not taken the keyboard further apart for this review since all the information I'd ever want is already available in the schematics. The keyboard matrix itself is read out by the Atmega directly which provides the full keyboard functionality and the OLED display is on a small daughterboard to slightly rise it towards the front bezel.

Firmware

Since this is one of the 8-bit Atmel parts it's very easy to build firmware using the gcc-avr compiler packaged in various distributions. All the source files are stored in the firmware repository for the various MNT products.

Checking the version of the firmware is pretty easy. With the circle key on the top-right corner of the keyboard the menu on the display opens. You can use the arrow keys to browse to the "System Status" option or just press the "s" key on the keyboard.

Which shows the hardware revision this firmware was build for and the version that was specified when building:

It seems like the "g" at the start of the commit has was accidental here and it refers to commit 7e73483 in the firmware repository. This seems to be the newest tag when the keyboard was shipped so that makes sense.

So lets change something! The key in the bottom left corner of the keyboard is the Hyper key instead of Ctrl as you'd expect from most keyboards. The Ctrl key is moved in place of the Caps Lock button on normal keyboard layouts which is great for a lot of uses. I never use Hyper though so I want to change that key to be my second Ctrl key.

The readme specifies that the keyboard layout is defined in the various matrix_* files so after reading around a bit it seems like I have to edit matrix_3.h for my keyboard.

Reading the manual again I realized that doing this makes me lose access to the media keys since those are defined as "Hyper+F*" for the various media actions. To fix that I changed the right control button into the Hyper key, this is the button with the three dots on it. My resulting code change:

diff --git a/reform2-keyboard-fw/matrix_3.h b/reform2-keyboard-fw/matrix_3.h
index bb72f6d..f9db133 100644
--- a/reform2-keyboard-fw/matrix_3.h
+++ b/reform2-keyboard-fw/matrix_3.h
@@ -25,7 +25,7 @@
 
 // Sixth row
 #define MATRIX3_DEFAULT_ROW_6 \
-  HID_KEYBOARD_SC_EXECUTE,\
+  HID_KEYBOARD_SC_LEFT_CONTROL,\
   HID_KEYBOARD_SC_LEFT_GUI,\
   HID_KEYBOARD_SC_LEFT_ALT,\
   KEY_SPACE,\
@@ -33,7 +33,7 @@
   KEY_SPACE,\
   KEY_SPACE,\
   HID_KEYBOARD_SC_RIGHT_ALT,\
-  HID_KEYBOARD_SC_RIGHT_CONTROL,\
+  HID_KEYBOARD_SC_EXECUTE,\
   HID_KEYBOARD_SC_LEFT_ARROW,\
   HID_KEYBOARD_SC_DOWN_ARROW,\
   HID_KEYBOARD_SC_RIGHT_ARROW

Now to build this there's a simple Makefile. Since I've already programmed Atmega parts on this machine I already have the compiler installed making this very quick and easy.

I ended up compiling with the following command:

$ make REFORM_KBD_OPTIONS="-DKBD_VARIANT_3 -DKBD_MODE_STANDALONE -DKBD_FW_VERSION=\\\"Martijn\\\""

This is straight from the readme with an additional define to set the firmware version to "Martijn". After building this I got the keyboard.hex file that can be flashed.

The flashing is as simple as running the flash.sh script. This will instruct you to press "Circle + X" to enter flashing mode and then run the neccesary commands to flash the keyboard. After running this I noticed that the delete key on the keyboard was no longer a delete key. It turns out I don't have VARIANT_3 but instead VARIANT_3_US. A quick rebuild and reflash also fixes that.

The brightness differences on the display are a camera artifact

Tadaa! My own name in the firmware version field. It's super easy to mess with this firmware.

The keyboard itself

Well the keyboard works just fine as a keyboard. Typing on this keyboard takes a few minutes to get used to compared to my normal keyboard since all the keys are slightly closer together. The split spacebar is also annoying me a bit. It turns out that the left split in the spacebar is exactly the spot where I normally hit the spacebar with my thumb.

The switches are nice and clicky (but silent, I have the version with brown switches in them). Overall the keyboard just does what it needs to. The standard layout is quite unusual but everything can be changed with open firmware so I'm confident I can get to a layout I'm 100% happy with.

Conclusion

This is an extremely solid and very compact keyboard I can easily throw in my backpack. It being an USB-C keyboard makes it fit neatly with all my other random cables I usually take with me.

It might be slightly more expensive than similar keyboards, but I don't know of similar keyboards with a case this rugged and the display functionality (I forgot to mention you can use HID reports from the host to write custom content to the display from your computer). The openness of this product makes the extra cost certainly worth it for me.

I'll probably be messing with the firmware for this keyboard a bit more while I use it. There's some small things to fix like the device reporting the name "LUFA Keyboard Demo Application" in Linux instead of a neater "MNT Keyboard" or something.

Looking closer at the syslog

Martijn Braam — Mon, 11 Dec 2023 15:43:17 -0000

The syslog protocol, it's one of the ancient protocols in the Unix world. For a long time the logging was handled by daemons like syslog-ng and rsyslog, this has now been taken over by journald on a lot of systems. But have you ever wondered how your log messages even end up in /var/log in the first place?

I've started looking into syslog implementations when building a replacement for the use of busybox syslogd in postmarketOS. In postmarketOS this daemon is configured to just send syslog messages to a in-memory buffer for logging and never store anything on disk in /var/log. This is mainly to make sure there's no unneeded writes to the flash storage in a lot of the old phones that are supported by postmarketOS. There's a few downsides to this logging implementation though:

No persistent logging of system messages across reboots. This would be easy to check if certain log messages were present on earlier boots when debugging.
Completely unstructured logs while people are pretty much used to journald logging with nice filters

As a replacement I wrote logbookd. It's a tiny syslog daemon that supports disk and memory logging and provides some nice filtering options to be closer to journald. The bulk of this work is handled by doing structured logging into SQLite.

So how does the syslog work

The way the syslog works is incredibly simple. The syslog daemon opens an unix domain socket at /dev/log. Applications connect to this socket and write log messages in the syslog format and the syslog daemon takes care of filtering those out and putting it in the various files in /var/log.

The complication of this is that there is no real syslog protocol. There are two standards for it though. There is RFC 3164 and RFC 5424 which both describe the syslog protocol. The 3164 document was only created in 2001 and describes what various implementations are doing in the wild. It's RFC 5424 that actually nails down a specific format.

I wrote parser for the 5424 format initially since that's the newest standard and it's by far the easiest to parse. An RFC 5424 log message looks like this:

<13>1 2023-12-11T14:56:59.0189+01:00 laptop test - - [timeQuality tzKnown="1"] Hi

The first part here between the angular brackets is the PRI value. It encodes the logging facility and severity as one number. The least significant 3 bits encode the severity on a scale of 1-8 and the other bits encode one of the 23 facilities that are defined. Some examples of the facilities are:

0 for kernel messages
1 for generic userspace messages
2 for the mail system

Most of the other numbers are for more old services like UUCP and FTP and for some numbered user-defined codes. In the example above the 13 means facility 1 (user) and severity 5 (notice).

The other parts of this message are in order:

The protocol version number which is set to 1 here.
A timestamp with timezone for the log message
The laptop is the hostname for the message, this will be set to - when NULL
The test part is the application name. This can also be - for NULL
The next field is called PROCID and is set to - for NULL in my case. According to the standard it might be used for the pid but is mostly implementation defined.
The second null is the MSGID, it can define a message type from the specific service, it will also be null in most cases.
The next part is [timeQuality tzKnown="1"] which is the STRUCTURED-DATA field. It can contain any implementation defined data. This is a subset of the structured data created by the logger tool used to create test messages. This field can also be just - for no structured data.
Finally the actual log message. That's just Hi in this case.

Writing a parser for this format is relatively straightforward. In the logbookd implementation there's a row for every one of these fields in the logging table and the message is split up according to these rules.

There is a fatal flaw in the RFC 5424 specification though: nobody is using it. None of the log messages on my running systems are actually in this format. It looks like practically all software uses RFC 3164, which is a fancy way of saying they do whatever they want.

RFC 3164

So this is actually the true specification for syslog messages being used in the wild. Let's look at one of these messages:

<13>Dec 11 15:21:50 laptop test: Hi

It's a lot simpler! But not actually. This is a pretty minimal message. The initial part is the same as the RFC 5424 message, the PRI is luckily parsed the exact same way. There is no version indicator though and it does not use an ISO timestamp format.

The more problematic issues with this format though is that it does support a lot more data but it's pretty badly defined. Even all the parts shown in the example above are optional. The most minimal syslog message that is still up to this spec is Hi.

It's also somewhat valid to send messages with a badly formatted timestamp and it's up to the syslogd to fix up the timestamp in the message. This also makes it very easy to make it actually parse parts of the timestamp as the hostname since this is all badly defined and space separated.

Since there is no official field for the pid of a process this is usually appended to the application name in square brackets.

The logbookd implementation is mostly based on the way these old messages are parsed in rsyslog and tries to not guess parts. This means only the timestamp, app, hostname and message fields are filled in.

Kernel logging

Not all logging in the system comes from userspace. On Linux there's also the kernel log ringbuffer that can be read from /dev/kmsg. Reading from this file will return all the log messages in the kernel ringbuffer and also makes it possible to stream new log messages with further reads. The log messages from the kernel are in a similar but different format than the syslog socket:

6,1004,5150172365,-;hid-generic 0003: hiddev96,hidraw2: USB HID device on usb-0000:00:14
 SUBSYSTEM=hid
 DEVICE=+hid:0003

The first field in the kernel message is again the PRI. This follows the same numbering as the syslog RFCs but it's not in angular brackets this time. In this case it's facility 0 (the kernel) and severity 6 (info). The second field is the KSEQ. This is a number that counts up for every log message since boot. The logbookd implementation uses this to de-duplicate the kernel log messages after opening the file since it will always return the old kernel log messages first.

After that comes the timestamp. Instead of string parsing this is a straight up unix timestamp so it's way easier to deal with. The field after the timestamp is - indicating NULL, this is the flags field.

After the semicolon the actual kernel log message starts. This is the message as is rendered in the dmesg utility. After the log message there's a newline but the log line doesn't end there! The structured data is defined as indented continuation lines after the message itself and this contains some easier machine-parsable data that is usually hidden in dmesg.

Systemd journald

So everything changed when journald was introduced. Figuring out how this all works involves diving into the systemd source code. Systemd provides several unix sockets related to logging in /var/run/systemd/journal:

dev-log this is symlinked to /dev/log and receives syslog formatted lines and writes it to the journal
stdout is a socket that receives logs from systemd units. This is what the systemd-cat command connects to. It writes a header on connection to give the application metadata and then the stdout or stderr is just connected straight to this socket.
socket receives the log messages in the binary journald format

There is a few other fancy things that journald does. It is possible to filter your log messages with the --boot argument. If no argument is supplied it will only show messages from the current boot. If you specify a negative number it's possible to get only log messages from a specific previous bootup.

The way this is done is by reading from /proc/sys/kernel/random/boot_id. This is a value generated by the kernel on bootup. It is a UUID generated from random data. These are also the values you see when you run journalctl --list-boots. The BOOT_ID value shown there is this UUID with the dashes removed.

My logbookd implementation also reads the boot_id on startup and stores it with the logs, this allows filtering in the exact same way with the logread -b parameter.

Logging to a database

So the main departure journald and logbookd do from the older syslog daemons is that they don't log to plain-text files. Journald has a custom database format the messages are stored in and logbookd stores messages in an SQLite database.

Structured logging to a database has a few nice upsides. The main one is being able to do way more detailed filtering than what is reasonably possible with grep. It's a lot easier to filter on a specific date and time range in a database and due to database indexes this is still fast.

One of the other main reasons for using SQLite in logbookd is that the implementation in postmarketOS was configured to only log to memory. Using SQLite as logging back-end meant that it's easily possible to replicate this by writing to an in-memory database which is already supported by SQLite.

The final thing added to logbookd is the middle ground between in-memory and on-disk logging: the reduced writes mode. In this mode the syslog is written to an in-memory database but when receiving a SIGINT, SIGTERM or SIGUSR1 signal the logbookd daemon will open the on-disk database and lets SQLite do a database migration. This means that SQLite will append the write the new loglines to the disk without rewriting all the existing logs there. On bootup this database is restored again so the logging system behaves as-if it's configured to do normal on-disk logging.

You can use this now

If you're running postmarketOS edge and you have updated to the latest version your installation should've migrated to the logbookd logging daemon. The logread utility implements the common options the busybox logread command already had. For normal use this means that there's not much difference except that the log output from the logread utility is now colored and contains kernel logs.

Some examples of the new things that are now possible:

$ logread -b list
ID   BOOT ID                                FIRST ENTRY       LAST ENTRY
 0   05c3f283-3bae-4b2a-8431-210dd63310e0   Dec 11 16:33:59   Dec 11 16:34:05
-1   f3ea2fa1-6f9e-4e82-bd0b-201091fcb5b4   Dec 07 18:21:06   Dec 07 18:25:50

$ logread -b 0 -n 2
[Dec 11 16:35:09] daemon dleyna-renderer-service[18060]: Client :1.166 lost
[Dec 11 16:35:10] daemon dleyna-renderer-service[18060]: dLeyna: Exit

$ logread -b 1 -n 1
[Dec 07 18:25:18] kern kernel: perf: interrupt took too long (2531 > 2500), lowering kernel.perf_event_max_sample_rate to 79000

$ logread -b -1 -u logbookd -n 1
[Dec 07 18:25:37] syslog logbookd: Ready to process log messages

Being able to see interleaved kernel and userspace messages also makes certain scenarios a lot easier to debug.

Hopefully this makes a few things easier to debug. There's a bunch of software that also logs directly into /var/log in seperate files, this has not been replaced by logbookd and is also not directly query-able by this new system. For the rest of the log messages enjoy the new colors :)

Megapixels 2.0: DNG exporting

Martijn Braam — Sat, 18 Nov 2023 14:17:38 -0000

It seems overkill to make a whole seperate library dedicated to replacing 177 lines of code in Megapixels that touches libtiff, but this small section of code causes significant issues for distribution packaging and compatability with external photo editing software. Most importantly the adjusted version in Millipixels for the Librem 5 does not output DNG files that are close enough to the Adobe specifications to be loaded into the calibration software.

Making this a seperate library would make it easier to test. In the Adobe DNG SDK there is a test utility that can verify if a TIFF file is up to DNG spec and it can (with a lot of complications) be build for Linux.

The spec

The first thing after copying over the code block from Megapixels to a seperate project is reading the Adobe DNG specification.

When I wrote the original export code in Megapixels it was based around some example code I found on Github for using Libtiff that I can no longer find and it results in something that's close enough to a valid DNG file for the dcraw utility. This is also a DNG 1.0 file that is generated.

I have spend the next day reading the DNG 1.4 specification from Adobe to understand what a valid DNG file is absolutely minimally required to have. These are my notes from that:

## Inside a DNG file

* SubIFDType 0 is the original raw data
* SubIFDType 1 is the thumbnail data
* The recommendation is to store the thumbnail as the first IFD
* TIFF metdata goes in the first IFD
* EXIF tags are preferred
* Camera profiles are stored in the first IFD

## Required tags

* DNGVersion
* UniqueCameraModel

Validation

I also spend a long time to build the official Adobe DNG SDK. This is mostly useless for developing any open source software due to licensing but it does provide a nice dng_validate utility that can be used to actually test the DNG files. Building this utility is pretty horrifying since it requires some specific versions of dependencies and some patches to work on modern compilers.

The libdng codebase now has the adobe_dng_sdk.sh script that will build the required libraries and the validation binary.

with the Megapixels code adjusted with the info from the documentation above I fed some random noise as data to the library to generate a DNG file and run it through the validator.

$ dng_validate out.dng
Validating "out.dng"...
*** Warning: This file has Chained IFDs, which will be ignored by DNG readers ***
*** Error: Unable to find main image IFD ***

Well that's not a great start... There's also a -v option to get some more verbose info

$ dng_validate -v out.dng
Validating "out.dng"...

Uses little-endian byte order
Magic number = 42

IFD 0: Offset = 308, Entries = 10

NewSubFileType: Preview Image
ImageWidth: 20
ImageLength: 15
BitsPerSample: 8
Compression: Uncompressed
PhotometricInterpretation: RGB
StripOffsets: Offset = 8
StripByteCounts: Count = 300
DNGVersion: 1.4.0.0
UniqueCameraModel: "LibDNG"
NextIFD = 10042

Chained IFD 1: Offset = 10042, Entries = 6

NewSubFileType: Main Image
ImageWidth: 320
ImageLength: 240
Compression: Uncompressed
StripOffsets: Offset = 441
StripByteCounts: Count = 9600
NextIFD = 0

*** Warning: This file has Chained IFDs, which will be ignored by DNG readers ***
*** Error: Unable to find main image IFD ***

Let's have a look at what the DNG spec says about this:

DNG recommends the use of SubIFD trees, as described in the TIFF-EP specification. SubIFD chains are not supported.

The highest-resolution and quality IFD should use NewSubFileType equal to 0. Reduced resolution (or quality) thumbnails or previews, if any, should use NewSubFileType equal to 1 (for a primary preview) or 10001.H (for an alternate preview).

DNG recommends, but does not require, that the first IFD contain a low-resolution thumbnail, as described in the TIFF-EP specification.

So I have the right tags and the right IFDs but I need to make an IFD tree instead of chain in libtiff. I have no idea how IFD trees work so up to the next specification!

It seems like TIFF trees are defined in the Adobe PageMaker 6 tech notes from 1995. That document describes that the NextIFD tag that libtiff used for me is used primarily for defining multi-page documents, not multiple encodings of the same document like what happens here with a thumbnail and the raw data. You know this is a 1995 spec because it gives a Fax as example of a multi-page document.

In the examples provided in that specification the first image is the main image and the NextIFD tag is just replaced by a subIFD tag. In case of DNG the main image is the thumbnail for compatibility with software that can't read the raw camera data.

Switching over to a SubIFD tag is suprisingly simple, just badly documented. Libtiff will create the NextIFD tag automatically for you but if you create an empty SubIFD tag then libtiff will fill in the offset for the next IFD for you when closing the file:

TIFF *tif = TIFFOpen(path, "w");

// Set the tags for IFD 0 like normal here
TIFFSetField(tif, TIGTAG_SUBFILETYPE, DNG_SUBFILETYPE_THUMBNAIL);

// Create a NULL reference for one SubIFD
uint64_t offsets[] = { 0L };
TIFFSetField(tif, TIFFTAG_SUBIFD, 1, &offsets);

// Write the thumbnail image data here

// Close the first IFD
TIFFWriteDirectory(tif);

// Start IFD1 describing the raw data
TIFFSetField(tif, TIFFTAG_SUBFILETYPE, DNG_SUBFILETYPE_ORIGINAL);
// write raw data and close the directory again
TIFFWriteDirectory(tif);

// Close the tiff, this will cause libtiff to patch up the references
TIFFCLose(tif);

So with the code updated the validation tool neatly shows the new SubIFD tags and finds actual errors in my DNG file data now

Uses little-endian byte order
Magic number = 42

IFD 0: Offset = 308, Entries = 11

NewSubFileType: Preview Image
ImageWidth: 20
ImageLength: 15
BitsPerSample: 8
Compression: Uncompressed
PhotometricInterpretation: RGB
StripOffsets: Offset = 8
StripByteCounts: Count = 300
SubIFDs: IFD = 10054
DNGVersion: 1.4.0.0
UniqueCameraModel: "LibDNG"
NextIFD = 0

SubIFD 1: Offset = 10054, Entries = 6

NewSubFileType: Main Image
ImageWidth: 320
ImageLength: 240
Compression: Uncompressed
StripOffsets: Offset = 453
StripByteCounts: Count = 9600
NextIFD = 0

*** Error: Missing or invalid SamplesPerPixel (IFD 0) ***
*** Error: Missing or invalid PhotometricInterpretation (SubIFD 1) ***

Ah, so these two tags are actually required but not described as such in the DNG specification since these are TIFF tags instead of DNG tags (while it does explicitly tells other TIFF required data).

Patching up these errors is easy, just slightly annoying since the validation tool seemingly gives only a single error per IFD requiring to iterate on the code a bit more. After a whole lot of iterating on the exporting code I managed to get the first valid DNG file:

Raw image read time: 0.000 sec
Linearization time: 0.002 sec
Interpolate time: 0.006 sec
Validation complete

Now the next step is adding all the plumbing to make this usable as library and making an actually nice command line utility.

First actual test

Now I have written the first iterations of libmegapixels and libdng it should be possible to actually load a picture in some editing software. So let's try some end-to-end testing with this.

With the megapixels-getframe utility from libmegapixels I can get a frame from the sensor (In this case the rear camera of the Librem 5) and then feed that raw data to the makedng utility from libdng.

$ getframe -o test.raw
Using config: /usr/share/megapixels/config/purism,librem5.conf
received frame
received frame
received frame
received frame
received frame
Stored frame to: test.raw
Format: 4208x3120
Pixfmt: GRBG
$ makedng -w 4208 -h 3120 -p GRBG test.raw test.dng
Reading test.raw...
Writing test.dng...

No errors and the file passes the DNG validation, let's load it into RawTherapee :)

The first frame loaded into RawTherapee

I had to boost the exposure a bit since the megapixels-getframe tool does not actually control any of the sensor parameters like the exposure time so the resulting picture is very dark. There's also no whitebalance or autofocus happening so the colors look horrible.

But...

The colors are correct! The interpetation of the CFA pattern of the sensor and the orientation of the data is all correct.

Integration testing

The nice thing about having the seperate library is that testing it becomes a lot easier than testing a GTK4 application. I have added the first simple end-to-end test to the codebase now that feeds some data to makedng and checks if the result is a valid DNG file using the official Adobe tool.

#!/bin/bash
set -e

if [ $# -ne 1 ]; then
  echo "Missing tool argument"
  exit 1
fi
makedng="$1"
echo "Running tests with '$makedng'"

# This testsuite runs raw data through the makedng utility and validates the
# result using the dng_validate tool from the Adobe DNG SDK. This tool needs
# to be manually installed for these tests to run.

# Create test raw data
mkdir -p scratch
magick -size 1280x720 gradient: -colorspace RGB scratch/data.rgb

# Generate DNG
$makedng -w 1280 -h 720 -p RG10 scratch/data.rgb scratch/RG10.dng

# Validate DNG
dng_validate scratch/RG10.dng

This is launched from ctest in my cmake files for now since I'm developing most of this stuff using CLion which only properly supports cmake projects. This is why a lot of my C projects have both meson and cmake files to build them but only the meson project file has install commands in it.

For more advanced testing it would be neat to have raw sensor dumps of several sensors in different formats which are all pictures of a colorchecker like the picture above. Then have some (probably opencv) utility that can validate that a colorchecker is present in the picture with the right colors.

There also needs to be a non-adobe-propriatary validation tool that can be easily run as testsuite for distribution packaging so at build time it's possible to validate that the combination of libdng and the distribution version of libtiff can produce sane output. This has caused several issues in Megapixels before after all.

Overall architecture

I've spent too much time drawing this

With the addition of libdng the architecture for Megapixels 2.0 starts to look like this. Megapixels no longer has any pipeline manipulation code, that is all handled by the library which after configuration just passes the file descriptor for the sensor node to Megapixels to handle the realtime control of the sensor parameters.

The libdng code replaces the plain libtiff exporting done in Megapixels and generate the DNG files that will be read by postprocessd. Postprocessd reads the dng files with the help of the dcraw library which already has custom DNG reading code that does not use libtiff.

The next steps now is to flesh out the library public interface for libdng so it can do all the DNG metadata that Megapixels requires and then hooking it up to Megapixels to actually use it.

Funding update

Since my previous post about the libmegapixels developments and the Megapixels 2.0 post I wrote before that I've almost doubled the funding for actually working on all the FOSS contributions. I'm immensely thankful for all the new patrons and it also made me notice that the donations page on this site was no longer being regenerated. That is fixed now.

I'm also still trying to figure out if I can add some perks for patrons to all of this but practically all options just amount to making things slightly worse for non-patrons. I hope just making the FOSS ecosystem better one of code line at a time is enough :)

Adding hardware to libmegapixels

Martijn Braam — Mon, 13 Nov 2023 17:59:48 -0000

Since in the last post I only showed off the libmegapixels config format and made some claims about configurablility without demonstrating it. I thought that it might be a good idea to actually demonstrate and document it.

As example device I will use my Xiaomi Mi Note 2 with a broken display, shown above. Also known in PostmarketOS under the codename xiaomi-scorpio. I picked this device as demo since I have already used this hardware in Megapixels 1.x so I know the kernel side of it is functional. I have not run any libmegapixels code on this device before writing this blogpost so I'm writing it as a I go along debugging it. Hopefully this device does not require any ioctl that has not been needed by the existing supported devices.

What makes it possible to get camera output from this phone is two things:

The camera subsystem in this device is supported pretty well in the kernel, in this case it's a Qualcomm device which has a somewhat universal driver for this
The sensor in this phone has a proper driver

The existing devices that I used to develop libmegapixels are based around the Rockchip, NXP and Allwinner platforms so this will be an interesting test if my theory works.

The config file name

Just like Megapixels 1.x the config file is based around the "compatible" name of the device. This is defined in the device tree passed to Linux by the bootloader. Since this is a nice mainline Linux device this info can be found in the kernel source: https://github.com/torvalds/linux/blob/b85ea95d086471afb4ad062012a4d73cd328fa86/arch/arm64/boot/dts/qcom/msm8996pro-xiaomi-scorpio.dts#L17

compatible = "xiaomi,scorpio", "qcom,msm8996pro", "qcom,msm8996";

This device tree specifies three names for this device ranking from more specific to less specific. xiaomi,scorpio is the exact hardware name, qcom,msm8996pro is the variant of the SoC and the qcom,msm8996 name is the inexact name of the SoC. Since this configuration defined both the SoC pipeline and the configuration for the specific sensor module the only sane option here is xiaomi,scorpio since that describes that exact hardware configuration. Other msm8996 devices might be using a completely different sensor.

The most specific option is not always the best option, in the case of the PinePhone for example the compatible is:

"pine64,pinephone-1.1", "pine64,pinephone", "allwinner,sun50i-a64";

In this hardware the camer system for the 1.0, 1.1 and 1.2 revision is identical so the config file just uses the pine64,pinephone name.

Knowing this the config file name will be xiaomi,scorpio.conf and can be placed in three locations. /usr/share/megapixels/config, /etc/megapixels/config and just the plain filename in your current working directory.

Now we know what the config path is the hard part starts, figuring out what to put in this config file.

The media pipeline

The next step is figuring out the media pipeline for this device. If the kernel has support for the hardware in the device it should create one or more /dev/media files. In the case of the Scorpio there's only a single one for the camera pipeline but there might be additional ones for stuff like hardware accelerated video encoding or decoding.

You can get the contents of the media pipelines with the media-ctl utility from v4l-utils. Use media-ctl -p to print the pipeline and you can use the -d option to choose another file than /dev/media0 if needed. For the Scorpio the pipeline contents are:

Media controller API version 6.1.14

Media device information
------------------------
driver          qcom-camss
model           Qualcomm Camera Subsystem
serial          
bus info        platform:a34000.camss
hw revision     0x0
driver version  6.1.14

Device topology
- entity 1: msm_csiphy0 (2 pads, 5 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		[fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb]
		<- "imx318 3-001a":0 [ENABLED,IMMUTABLE]
	pad1: Source
		[fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb]
		-> "msm_csid0":0 []
		-> "msm_csid1":0 []
		-> "msm_csid2":0 []
		-> "msm_csid3":0 []

[ Removed A LOT of entities here for brevity ]

- entity 226: imx318 3-001a (1 pad, 1 link)
              type V4L2 subdev subtype Sensor flags 0
              device node name /dev/v4l-subdev19
	pad0: Source
		[fmt:SRGGB10_1X10/5488x4112@1/30 field:none colorspace:raw xfer:none]
		-> "msm_csiphy0":0 [ENABLED,IMMUTABLE]

- entity 228: ak7375 3-000c (0 pad, 0 link)
              type V4L2 subdev subtype Lens flags 0
              device node name /dev/v4l-subdev20

The header shows that this is a media device for the qcom-camss system, which handles cameras on Qualcomm devices. There is also a node for the imx318 sensor which further confirms that this is the right media pipeline.

Analyzing the pipeline in this format is pretty hard when there's more than two nodes though, that's why there is a neat option in media-ctl to output the mediagraph as an actual graph using Graphviz.

$ apk add graphviz
$ media-ctl -d 0 --print-dot | dot -Tpng > pipeline.png

Which produces this image:

In a bunch of cases you can copy most of the configuration of this graph from another device that uses the same SoC but since this is the first Qualcomm device I'm adding I have to figure out the whole pipeline.

The only part that's really specific to the Xiaomi Scorpio is the top two nodes. The imx318 is the actual camera module in the phone connected with mipi to the SoC. The ak7375 is listed as a "Motor driver". This means that it is the chip handeling the lens movements for autofocus. There are no connections to this node since this device does not handle any graphical data, the entity only exists so you can set v4l control values on it to move the focus manually.

All the boxes in the graph are called entities and correspond with the Entity blocks in the media-ctl -p output. The boxes are yellow if they are entities with the type V4L, these are the nodes that will show up als /dev/video nodes to actually get the image data out of this pipeline.

The lines between the boxes are called links, the dotted lines are disabled links and solid lines are enabled links. On this hardware a lot of the links are created by the kernel driver and are hardcoded. These links show up in the text output as IMMUTABLE and mostly describe fixed hardware paths for the image data.

The goal of configuring this pipeline is to get the image data from the IMX sensor all the way down to one of the /dev/video nodes and figuring out the purpose of the entities in between. If you are lucky there is actual documentation for this. In this case I have found documentation at https://www.kernel.org/doc/html/v4.14/media/v4l-drivers/qcom_camss.html which is for the v4.14 kernel but for some reason is removed on later releases.

This documentation has neat explanations for these entities:

2 CSIPHY modules. They handle the Physical layer of the CSI2 receivers. A separate camera sensor can be connected to each of the CSIPHY module;
2 CSID (CSI Decoder) modules. They handle the Protocol and Application layer of the CSI2 receivers. A CSID can decode data stream from any of the CSIPHY. Each CSID also contains a TG (Test Generator) block which can generate artificial input data for test purposes;
ISPIF (ISP Interface) module. Handles the routing of the data streams from the CSIDs to the inputs of the VFE;
VFE (Video Front End) module. Contains a pipeline of image processing hardware blocks. The VFE has different input interfaces. The PIX (Pixel) input interface feeds the input data to the image processing pipeline. The image processing pipeline contains also a scale and crop module at the end. Three RDI (Raw Dump Interface) input interfaces bypass the image processing pipeline. The VFE also contains the AXI bus interface which writes the output data to memory.

This documentation is not for this exact SoC so the amount of entities of each type is different.

Configuring the pipeline and connecting it all up is now just a lot of trial and error, in the case of the Scorpio it has already been trial-and-error'd so there is an existing config file for the old Megapixels at https://gitlab.com/postmarketOS/megapixels/-/blob/master/config/xiaomi,scorpio.ini

In this old pipeline description format the path is just enabling the links between the first csiphy, csid, ispif and vfe entity. Since this release of Megapixels did not really support further configuration it just tried to then set the resolution and pixel format for the sensors on all entities after it and hoped it worked. On an unknown platform just picking the left-most path will pretty likely bring up a valid pipeline, the duplicated entities are mostly useful for cases where you are using multiple cameras at once.

Initial config file

The first thing I did is creating a minimal config file for the scorpio that had the minimal pipeline to stream unmodified data from the sensor to userspace.

Version = 1;
Make: "Xiaomi";
Model: "Scorpio";

Rear: {
    SensorDriver: "imx318";
    BridgeDriver: "qcom-camss";

    Modes: (
    {
	Width: 3840;
	Height: 2160;
	Rate: 30;
	Format: "RGGB10";
	Rotate: 90;

	Pipeline: (
	    {Type: "Link", From: "imx318", FromPad: 0, To: "msm_csiphy0", ToPad: 0},
	    {Type: "Link", From: "msm_csiphy0", FromPad: 1, To: "msm_csid0", ToPad: 0},
	    {Type: "Link", From: "msm_csid0", FromPad: 1, To: "msm_ispif0", ToPad: 0},
	    {Type: "Link", From: "msm_ispif0", FromPad: 1, To: "msm_vfe0_rdi0", ToPad: 0},
	    {Type: "Mode", Entity: "imx318"},
	    {Type: "Mode", Entity: "msm_csiphy0"},
	    {Type: "Mode", Entity: "msm_csid0"},
	    {Type: "Mode", Entity: "msm_ispif0"},
	);
	},
    );
};

This can be tested with the megapixels-getframe command.

$ ./megapixels-getframe
Using config: /etc/megapixels/config/xiaomi,scorpio.conf
[libmegapixels] Could not link 226 -> 1 [imx318 -> msm_csiphy0] 
[libmegapixels] Capture driver changed pixfmt to UYVY
Could not select mode

This command tries to output as much debugging info as possible, but the reality is that you'll most likely need to look at the kernel source to figure out what is happening and what arbitrary constraints exist.

So the iterating and figuring out errors starts. First the most problematic line is the UYVY format one. This most likely means that the pipeline pixelformat I selected was not correct and to fix that the kernel helpfully selects a completely different one. getframe will detect this and show this happening. In this case the RGGB10 format is wrong and it should have been RGGB10p. The kernel implementation is a bit inconsistent about which format it actually is while MIPI only allows one of these two in the spec. Changing that removes that error.

The other interesting error is the link that could not be created. If you look closely at the Graphviz output you'll see that this link is already enabled by the kernel and in the text output it is also IMMUTABLE. This config line can be dropped because this is not configurable.

$ ./megapixels-getframe
Using config: /etc/megapixels/config/xiaomi,scorpio.conf
VIDIOC_STREAMON failed: Broken pipe

Progress! At least somewhat. The mode setting commands succeed but now the pipeline can not actually be started. This is because some drivers only validate options when starting the pipeline instead of when you're actually setting modes. This is one of the most annoying errors to fix because there's no feedback whatsoever on what or where the config issue is.

My suggestion for this is to first run media-ctl -p again and see the current state of the pipeline. This output shows the format for the pads of the pipeline so you can find a connection that might be invalid by comparing those. My pipeline state at this point is:

imx318: SRGGB10_1X10/3840x2160@1/30
csiphy0: SRGGB10_1X10/3840x2160
csid0: SRGGB10_1X10/3840x2160
ispif0: SRGGB10_1X10/3840x2160
vfe0_rdi0: UYVY8_2X8/1920x1080

AHA! the last node is not configured correctly. It's always the last one you look at. It turns out the issue was that I'm simply missing a mode command in my config file that sets the mode on that entity so it's left at the pipeline defaults. Let's test the pipeline with that config added:

$ /megapixels-getframe
Using config: /etc/megapixels/config/xiaomi,scorpio.conf
received frame
received frame
received frame
received frame
received frame

The pipeline is streaming! This is the bare minimum configuration needed to make Megapixels 2.0 use this camera. For reference after all the changes above the config file is:

Version = 1;
Make: "Xiaomi";
Model: "Scorpio";

Rear: {
    SensorDriver: "imx318";
    BridgeDriver: "qcom-camss";

    Modes: (
        {
            Width: 3840;
            Height: 2160;
            Rate: 30;
            Format: "RGGB10p";
            Rotate: 90;
            
            Pipeline: (
                {Type: "Link", From: "msm_csiphy0", FromPad: 1, To: "msm_csid0", ToPad: 0},
                {Type: "Link", From: "msm_csid0", FromPad: 1, To: "msm_ispif0", ToPad: 0},
                {Type: "Link", From: "msm_ispif0", FromPad: 1, To: "msm_vfe0_rdi0", ToPad: 0},
                {Type: "Mode", Entity: "imx318"},
                {Type: "Mode", Entity: "msm_csiphy0"},
                {Type: "Mode", Entity: "msm_csid0"},
                {Type: "Mode", Entity: "msm_ispif0"},
                {Type: "Mode", Entity: "msm_vfe0_rdi0"},
            );
        },
    );
};

Camera metadata

The config file not only stores information about the media pipeline but can also store information about the optical path. Every mode can define the focal length for example because changing the cropping on the sensor will give you digital zoom and thus a longer focal length. With modern phones with 10 cameras on the back it is also possible to define all of them as the "rear" camera and have multiple modes with multiple focal lengths so camera apps can switch the pipeline for zooming once zooming is implemented in the UI.

Finding out the values for this optical path is basically just using search engines to find datasheets and specs. Sometimes the pictures generated by android have the correct information for this in the metadata as well.

This information is also mostly absent from sensor datasheets since that only describe the sensor itself, you either need to find this info from the camera module itself (which is the sensor plus the lens) or the specifications for the phone.

From spec listings and review sites I've found that the focal length for the rear camera is 4.06mm and the aperture is f/2.0. This can be added to the mode section:

Width: 3840;
Height: 2160;
Rate: 30;
Format: "RGGB10p";
Rotate: 90;
FocalLength: 4.06;
FNumber: 2.0;

Reference for pipeline commands

Since this is now practically the main reference for writing config files until I get documentation generation up and running for libmegapixels I will put the complete documentation for the various commands here.

While parsing the config file there are four values stored as state : width, height, format and rate. The values for these default to the ones set in the mode and they are updated whenever you define one of these values explicitly in a command. This prevents having to write the same resolution values repeatedly on every line but it still allows having entities in the pipeline that scale the resolution.

Link

{
  Type: "Link",
  From: "msm_csiphy0", # Source entity name, required
  FromPad: 1,          # Source pad, defaults to 0
  To: "msm_csid0",     # Target entity name, required
  ToPad: 0             # Target pad, defaults to 0
}

Translates to an MEDIA_IOC_SETUP_LINK ioctl on the media device.

Mode

{
  Type: "Mode", 
  Entity: "imx318"  # Entity name, required
  Width: 1280       # Horisontal resolution, defaults to previous in pipeline
  Height: 720       # Vertical resolution, defaults to previous in pipeline
  Pad: 0            # Pad to set the mode on, defaults to 0
  Format: "RGGB10p" # Pixelformat for the mode, defaults to previous in pipeline
}

Translates to an VIDIOC_SUBDEV_S_FMT ioctl on the entity.

Rate

{
  Type: "Rate",
  Entity: "imx318",  # Entity name, required
  Rate: 30           # FPS, defaults to previous in pipeline
}

Translates to an VIDIOC_SUBDEV_S_FRAME_INTERVAL ioctl on the entity.

Crop

{
  Type: "Crop",
  Entity: "imx318", # Entity name, required
  Width: 1280       # Area width, defaults to previous width in pipeline
  Height: 720       # Area height resolution, defaults to previous height in pipeline
  Top: 0            # The vertical offset, defaults to 0
  Left: 0           # The horisontal offset, defaults to 0
  Pad: 0            # Pad to set the crop on, defaults to 0
}

Translates to an VIDIOC_SUBDEV_S_CROP ioctl on the entity.

The future of libmegapixels

It has been quite a bit of work to create libmegapixels and it has been a mountain of work to rework Megapixels to integrate it. The first 90% of this is done but the trick is always in getting the second 90% finished. In the Megapixels 2.0 post I already mentioned this has burned me out. On the other hand it's a shame to let this work go to waste.

There is a few parts of autofocus, autoexposure and autowhitebalance that are very complicated and math heavy to figure out, I can't figure it out. The loop between libmegapixels and Megapixels exists to pass around the values but I can't stop the system from oscillating and can't get it to settle on good values. There seems to be no good public information available on how to implement this in any case.

Another difficult part is sensor calibration. I have the hardware and software to create calibration profiles but this system expects the input pictures to come from... working cameras. The system completely lacks proper sensor linearisation which makes setting a proper whitebalance not really possible. You might have noticed the specific teal tint that gives away that a picture is taken on a Librem 5 for example. If that teal tint is corrected for manually then the midtones will look correct but highlights will become too yellow. Maybe there's a way to calibrate this properly or maybe this just takes someone messing with the curves manually for a long while to get correct.

There also needs to be an alternative to writing dng files with libtiff so for my own sanity it is required to write libdng. The last few minor releases of libtiff have all been messing with the tiff tags relating to DNG files which have caused taking pictures to not work for a lot of people. The only way around this seems to be stop using libtiff like all the Linux photography software has already done. This is not a terribly hard thing to implement, it just has been prioritized below getting color correct so far and I have not had the time to work on it.

There is also still segfaults and crashes relating to the GPU debayer code in Megapixels for most of the pixel formats. This is very hard to debug due to the involvement of the GPU in the equation.

How can you help

If you know how to progress with any of this I gladly accept any patches for this to push it forward.

The harder part of this section is... money. I love working on photography stuff, I can't believe the Megapixels implementation has even gotten this far but it basically takes me hyperfocusing for weeks for 12 hours per day on random camera code to get to this point, that is not really sustainable. It's great to work on this for some days and making progress, it's really painful to work for weeks on that one 30 line code block and making no progress whatsoever. At some point my dream is that I can actually live off doing open source work but so far that has still been a distant dream.

I've had the donations page now for some years and I'm incredibly happy that people are supporting me to work on this at all. It's just forever stuck on receiving enough money that you feel like a responsibility to produce progress but not nearly enough to actually fund that progress. So in practice only extra pressure.

So I hate asking for money, but it would certainly help towards the dream of being an actual full time FOSS developer :)

Megapixels 2.0

Martijn Braam — Thu, 09 Nov 2023 18:33:39 -0000

The Megapixels camera application has long been the most performant camera application on the original PinePhone. I have not gotten the Megapixels application to that point alone. There have been several other contributors that have helped slowly improving performance and features of this application. Especially Benjamin has leaped it forward massively with the threaded processing code and GPU accelerated preview.

All this code has made Megapixels very fast on the PinePhone but also has made it quite a lot harder to port the application to other hardware. The code is very much overfitted for the PinePhone hardware.

Finding a better design

To address the elephant in the room, yes libcamera exists and promises to abstract this all away. I just disagree with the design tradeoffs taken with libcamera and I think that any competition would only improve the ecosystem. It can't be that libcamera got this exactly right on the first try right?

Instead of the implementation that libcamera has made that makes abstraction code in c++ for every platform I have decided to pick the method that libalsa uses for the audio abstraction in userspace.

Alsa UCM config files are selected by soundcard name and contain a set of instructions to bring the audio pipeline in the correct state for your current usecase. All the hardware specific things are not described in code but instead in plain text configuration files. I think this scales way better since it massively lowers the skill floor needed to actually mess with the system to get hardware working.

The first iteration of Megapixels has already somewhat done this. There's a config file that is picked based on the hardware model that describes the names of the device nodes in /dev so those paths don't have to be hardcoded and it describes the resolution and mode to configure. It also describes a few details about the optical path to later produce correct EXIF info for the pictures.

[device]
make=PINE64
model=PinePhone

[rear]
driver=ov5640
media-driver=sun6i-csi
capture-width=2592
capture-height=1944
capture-rate=15
capture-fmt=BGGR8
preview-width=1280
preview-height=720
preview-rate=30
preview-fmt=BGGR8
rotate=270
colormatrix=1.384,-0.3203,-0.0124,-0.2728,1.049,0.1556,-0.0506,0.2577,0.8050
forwardmatrix=0.7331,0.1294,0.1018,0.3039,0.6698,0.0263,0.0002,0.0556,0.7693
blacklevel=3
whitelevel=255
focallength=3.33
cropfactor=10.81
fnumber=3.0
iso-min=100
iso-max=64000
flash-path=/sys/class/leds/white:flash

[front]
...

This works great for the PinePhone but it has a significant drawback. Most mobile cameras require an elaborate graph of media nodes to be configured before video works, the PinePhone is the exception in that the media graph only has an input and output node so Megapixels just hardcodes that part of the hardware setup. This makes the config file practically useless for all other phones and this is also one of the reason why different devices have different forks to make Megapixels work.

So a config file that only works for a single configuration is pretty useless. Instead of making this an .ini file I've switched the design over to libconfig so I don't have to create a whole new parser and it allows for nested configuration blocks. The config file I have been using on the PinePhone with the new codebase is this:

Version = 1;
Make: "PINE64";
Model: "PinePhone";

Rear: {
    SensorDriver: "ov5640";
    BridgeDriver: "sun6i-csi";
    FlashPath: "/sys/class/leds/white:flash";
    IsoMin: 100;
    IsoMax: 64000;

    Modes: (
        {
            Width: 2592;
            Height: 1944;
            Rate: 15;
            Format: "BGGR8";
            Rotate: 270;
            FocalLength: 3.33;
            FNumber: 3.0;

            Pipeline: (
                {Type: "Link", From: "ov5640", FromPad: 0, To: "sun6i-csi", ToPad: 0},
                {Type: "Mode", Entity: "ov5640", Width: 2592, Height: 1944, Format: "BGGR8"},
            );
        },
        {
            Width: 1280;
            Height: 720;
            Rate: 30;
            Format: "BGGR8";
            Rotate: 270;
            FocalLength: 3.33;
            FNumber: 3.0;

            Pipeline: (
                {Type: "Link", From: "ov5640", FromPad: 0, To: "sun6i-csi", ToPad: 0},
                {Type: "Mode", Entity: "ov5640"},
            );

        }
    );
};

Front: {
    SensorDriver: "gc2145";
    BridgeDriver: "sun6i-csi";
    FlashDisplay: true;

    Modes: (
        {
            Width: 1280;
            Height: 960;
            Rate: 60;
            Format: "BGGR8";
            Rotate: 90;
            Mirror: true;

            Pipeline: (
                {Type: "Link", From: "gc2145", FromPad: 0, To: "sun6i-csi", ToPad: 0},
                {Type: "Mode", Entity: "gc2145"},
            );
        }
    );

Instead of having a hardcoded preview mode and main mode for every sensor it's now possible to make many different resolution configs. This config recreates the 2 existing modes and Megapixels now picks faster mode for the preview automatically and use higher resolution modes for the actual picture.

Every mode now also has a Pipeline block that describes the media graph as a series of commands, every line translates to one ioctl called on the right device node just like Alsa UCM files describe it as a series of amixer commands. Megapixels no longer has the implicit PinePhone pipeline so here it describes the one link it has to make between the sensor node and the csi node and it tells Megapixels to set the correct mode on the sensor node.

This simple example of the PinePhone does not really show off most of the config options so lets look at a more complicated example:

Pipeline: (
    {Type: "Link", From: "imx258", FromPad: 0, To: "rkisp1_csi", ToPad: 0},
    {Type: "Mode", Entity: "imx258", Format: "RGGB10P", Width: 1048, Height: 780},
    {Type: "Mode", Entity: "rkisp1_csi"},
    {Type: "Mode", Entity: "rkisp1_isp"},
    {Type: "Mode", Entity: "rkisp1_isp", Pad: 2, Format: "RGGB8"},
    {Type: "Crop", Entity: "rkisp1_isp"},
    {Type: "Crop", Entity: "rkisp1_isp", Pad: 2},
    {Type: "Mode", Entity: "rkisp1_resizer_mainpath"},
    {Type: "Mode", Entity: "rkisp1_resizer_mainpath", Pad: 1},
    {Type: "Crop", Entity: "rkisp1_resizer_mainpath", Width: 1048, Height: 768},
);

This is the preview pipeline for the PinePhone Pro. Most of the Links are already hardcoded by the kernel itself so here it only creates the link from the rear camera sensor to the csi and all the other commands are for configuring the various entities in the graph.

The Mode commands are basically doing the VIDIOC_SUBDEV_S_FMT ioctl on the device node found by the entity name. To make configuring modes on the pipeline not extremely verbose it implicitly takes the resolution, pixelformat and framerate from the main information set by the configuration block itself. Since several entities can convert the frames into another format or size it automatically cascades the new mode to the lines below it.

In the example above the 5th command sets the format to RGGB8 which means that the mode commands below it for rkisp1_resizer_mainpath also will use this mode but the rkisp1_csi mode command above it will still be operating in RGGB10P mode.

Splitting of device management code

Testing changes in Megapixels is pretty hard. To develop the Megapixels code I'm building it on the phone and launching it over SSH with a bunch of environment variables set so the GTK window shows up on the phone and I get realtime logs on my computer. If there's anything that's going on after the immediate setup code it is quite hard to debug because it's in one of the three threads that process the image data.

To implement the new pipeline configuration I did that in a new empty project that builds a shared library and a few command line utilities that help test a few specific things. This codebase is libmegapixels and with it I have split off all hardware access from Megapixels itself making both these codebases a lot easier to understand.

It has been a lot easier to debug complex camera pipelines using the commandline utilities and only working on the library code. It should also make it a lot easier to make Megapixels-like applications that are not GTK4 to make it integrate more with other environments. One of the test applications for libmegapixels is getframe which is now all you need to get a raw frame from the sensor.

Since this codebase is now split into multiple parts I have put it into a seperate gitlab organisation at https://gitlab.com/megapixels-org which hopefully keeps this a bit organized.

This is also the codebase used for https://fosstodon.org/@martijnbraam/110775163438234897 which shows off libmegapixels and megapixels 2.0 running on the Librem 5.

Burnout

So now the worse part of this blog post. No you can't use this stuff yet :(

I've been working on this code for months, and now I've not been working on this code for months. I have completely burned out on all of this.

The libmegapixels code is in pretty good state but the Megapixels rewrite is still a large mess:

Saving pictures doesn't really work and I intended to split that off to create libdng
The QR code support is not hooked up at all at the moment
Several pixelformats don't work correctly in the GPU decoder and I can't find out why
Librem 5 and PinePhone Pro really need auto-exposure, auto-focus and auto-whitebalance to produce anything remotely looking like a picture. I have ported the auto-exposure from Millipixels which works reasonably well for this but got stuck on AWB and have not attempted Autofocus yet.

The mountain of work that's left to do to make this a superset of the functionality of Megapixels 1.x and the expectations surrounding it have made this pretty hard to work on. On the original Megapixel releases nothing mattered because any application that could show a single frame of the camera was already a 100% improvement over the current state.

Another issue is that whatever I do or figure out it will always be instantly be put down with "Why are you not using libcamera" and "libcamera probably fixes this".

Some things people really need to understand is that an application not using libcamera does not mean other software on the system can't support libcamera. If Firefox can use libcamera to do videocalls that's great, that's not the usecase Megapixels is going for anyway.

What also doesn't help is receiving bugreports for the PinePhone Pro while Megapixels does not support the PinePhone Pro. There's a patchset added on top to make in launch on the PinePhone Pro but there's a reason this patchset is not in Megapixels. The product of the Megapixels source with the ppp.patch added on top probably shouldn't've been distributed as Megapixels...

What also doesn't help is that if Megapixels 2.0 were finished and released it would also create a whole new wave of criticism and comparisons to libcamera. I would have to support Megapixels for the people complaining that it's not enough... You could've not had a camera application at all...

It also doesn't help that the libcamerea developers are also the v4l2 subsystem maintainers in the kernel. I have during development of libmegapixels tried sending a simple patch for an issue I've noticed that would massively improve the ease of debugging PinePhone Pro cameras. I've sent this 3 character patch upstream to the v4l2 mailing lists and it got a Reviewed-by in a few days.

Then after 2 whole months of radio silence it got rejected by the lead developer of libcamera on debatable grounds. Now this is only a very small patch so I'm merely dissapointed. If I had put more work into the kernel side improving some sensor drivers I might have been mad but at this point I'm just not feeling like contributing to the camera ecosystem anymore.

Edit: I've been convinced to actually try to do this full-time and push the codebase forward enough to make it usable. This is continued at https://blog.brixit.nl/adding-hardware-to-libmegapixels/

Making an USB Ethernet adapter work [SR9700]

Martijn Braam — Sat, 28 Oct 2023 17:41:51 -0000

I just needed a simple USB to Ethernet adapter for testing. It does not need to be fast, it does not need to reach USB 3.0 speeds or gigabit speeds. I own two other USB ethernet adapters that have various reliability issues so I got a random cheap one from ebay.

The adapter I ended up getting was the SR-QF9700 that does not have an actual brand on it. When plugging it into my laptop there was a slight issue though. It shows up as a CD drive...

This is one of those annoying features where USB dongles will present a CD drive with drivers to windows to make them work, ignoring that this is just an rndis device that has worked without special drivers for decades. This wouldn't be problematic if it exposed both the ethernet interface and the driver CD at the same time so I could just ignore it.

In theory this can be made working by hacks such as usb_modeswitch but I've seen several forum posts online that mention it not working on this hardware. Even if that did solve the issue, it still makes this device useless for me because I don't want to mess with special software on every device I plug it into.

I was ready to add this adapter to the e-waste problem but then I thought about how this would be handled from the manufacturing side. The chip is generic but those driver installers are usually branded. There needs to be somewhere to actually store the drivers on the device...

Taking it apart

This thing is very easy to open. There are no screws or clips in it at all, it's all held together by the sticker with the model number.

Inside the adapter is a simple PCB that has a crystal and the USB ethernet module. This particular adapter contains a CoreChips SR9700 single-chip ethernet controller. Doing anything interesting with this would be hard, there's not really public documentation for this chip except for the pin descriptions. The SR9900 datasheet mentions some internal one-time programmable memory, but it being one-time programmable does not really help me.

So lets look at the back of the PCB

Another chip! This is an 4MBit SPI Nor flash chip. This must be the chip actually storing the windows drivers.

So what is the behavior of the ethernet controller chip if this chip is absent...

SPI flash pinouts are pretty standardized, but I looked up the datasheet for this specific chip to be sure:

There's many ways to make a SPI flash chip temporarily not function. I decided to jam a screwdriver between the CS and SO pin to make the chip unable to respond to SPI communication. With the screwdriver in place I plugged in the USB cable and behold:

Ethernet! Well that was easy. Now to make this permanent I shorted together the same two pins with my soldering iron since removing the whole chip is way harder.

So instead of messing with any device/computer I want to plug this ethernet adapter into I now have a normal USB ethernet adapter with just one extra solder connection :)

Developers are lazy, thus Flatpak

Martijn Braam — Sat, 03 Jun 2023 15:58:47 -0000

In the last decade I have seen a very slow but steady shift to solutions for packaging software that try to isolate the software from host systems to supposedly make things easier. My first experience with this was Docker, now Flatpak is the thing for desktop applications.

The promise of Flatpak

So the thing Flatpak is supposed to fix for me as developer is that I don't need to care about distributions anymore. I can bolt on whatever dependencies I want to my app and it's dealt with. I also don't need to worry about having software in distributions, if it's in Flatpak it's everywhere. Flatpak gives me that unified base to work on and everything will be perfect. World hunger will be solved. Finally peace on earth.

Sadly there's reality. The reality is to get away from the evil distributions the Flatpak creators have made... another distribution. It is not a particularly good distribution, it doesn't have a decent package manager. It doesn't have a system that makes it easy to do packaging. The developer interface is painfully shoehorned into Github workflows and it adds all the downsides of containerisation.

Flatpak is a distribution

While the developers like to pretend real hard that Flatpak is not a distribution, it's still suspiciously close to one. It lacks a kernel and a few services and it lacks the standard Linux base directory specification but it's still a distribution you need to target. Instead of providing seperate packages with a package manager it provides a runtime that comes with a bunch of dependencies. Conveniently it also provides multiple runtimes to make sure there's not actually a single base to work on. Because sometimes you need Gnome libraries, sometimes you need KDE libraries. Since there's no package manager those will be in seperate runtimes.

While Flatpak breaks most expectations of a distribution it's still a collection of software and libraries build together to make a system to run software in, thus it's a distribution. A really weird one.

No built in package manager

If you need a dependency that's not in the runtime there's no package manager to pull in that dependency. The solution is to also package the dependencies you need yourself and let the flatpak tooling build this into the flatpak of your application. So now instead of being the developer for your application you're also the maintainer of all the dependencies in this semi-distribution you're shipping under the disguise of an application. And one thing is for sure, I don't trust application developers to maintain dependencies.

This gets really nuts by looking at some software that deals with multimedia. Lets look at the Audacity flatpak. It builds as dependency:

wxwidgets
ffmpeg
sqlite
chrpath
portaudio
portmidi

So lets look at how well dependencies are managed here. Since we're now almost exactly half a year into 2023 I'll look at the updates for the last 6 months and compare it to the same dependencies in Alpine Linux.

audacity has been updated 4 times in the flatpak. It has been updated 5 times on Alpine.
ffmpeg has been updated to 6.0 in both the flatpak and Alpine, but the ffmpeg package has had 9 updates because if codecs that have been updated.
sqlite hasn't been updated in the flatpak and has been updated 4 times in Alpine
wxwidgets hasn't been updated in the flatpak and has been updated 2 times in Alpine
chrpath hasn't had updates
portaudio hasn't had updates in flatpak and Alpine.
portmidi hasn't had updates

This is just a random package I picked and it already had a lot more maintainance of the dependencies than the flatpak has. It most likely doesn't scale to have all developers keep track of all the dependencies of all their software.

The idea of isolation

One of the big pros that's always mentioned with Flatpak is that the applications run in a sandbox. The idea is that this sandbox will shield you from all the evil applications can do so it's totally safe to trust random developers to push random Flatpaks. First of all this sandbox has the same issue any permission system that exists also has. It needs to tell the user about the specific holes that have been poked in the sandbox to make the application work in a way that end users understand what the security implications of those permissions are.

For example here's Gnome Software ready to install the flatpak for Edge:

I find the permission handleing implemented here very interesting. There's absolutely no warning whatsoever about the bypassed security in this Flatpak untill you scroll down. The install button will immediately install it without warning about all the bypassed sandboxing features.

So if you do scroll down there's more details right? Sure there is!

There's a nice red triangle with the words Unsafe! pfew, everyone is fine now. So this uses a legacy windowing system which probably means it uses X11 which is not secure and breaks the sandbox. Well if that's the only security issue then it might be acceptable? Let's click that button.

Well yeah... let's hide that from users. Of course the browser needs to write to /etc. This is all unimportant to end users.

The even worse news is that since this is proprietary software it's not really possible to audit what this would do, and even if it's audited it's ridiculously easy to push a new more evil version to Flathub since practically only the first version of the app you push is thorougly looked at by the Flathub maintainers.

Even if there weren't so many holes in the sandbox. This does not stop applications from doing more evil things that are not directly related to filesystem and daemon access. You want analytics on your users? Just requirest the internet permission and send off all the tracking data you want.

So what about traditional distributions

I've heard many argument for Flatpaks by users and developers but in the end I can't really say the pros outweigh the cons.

I think it's very important that developers do not have the permissions to push whatever code they want to everyone under the disguise of a secure system. And that's my opinion as a software developer.

Software packaged by distributions has at least some degree of scrutiny and it often results in at least making sure build flags are set to disable user tracking and such features.

I also believe software in general is better if it's made with the expectation that it will run outside of Flatpak. It's not that hard to make sure you don't depend on bleeding edge versions of libraries while that's not needed. It's not that hard to have optional dependencies in software. It's not that hard to actually follow XDG specifications instead of hardcoding paths.

But packaging for distributions is hard

That's the best thing! Developers are not supposed to be the ones packaging software so it's not hard at all. It's not your task to get your software in all the distributions, if your software is useful to people it tends to get pulled in. I have software that's packaged in Alpine Linux, ALT Linux, Archlinux AUR, Debian, Devuan, Fedora, Gentoo, Kali, LiGurOS, Nix, OpenMandriva, postmarketOS, Raspbian, Rosa, Trisquel, Ubuntu and Void. I did not have to package most of this.

The most I notice from other distributions packaging my software is patches from maintainers that improve the software, usually in dealing with some edge case I forgot with a hardcoded path somewhere.

The most time I've ever spent on distribution packaging is actually the few pieces of software I've managed to push to Flathub. Dealing with differences between distributions is easy, dealing with differences between runing inside and outside Flatpak is hard.

But Flatpaks are easier for end users

I've ran into enough issues as end user of flatpaks. A package being on Flathub does not mean that it will be installable for an end user. I've ran into this by installing packages on the PineBook Pro which generated some rather confusing error messages about the repositories missing. It turns out that the Aarch64 architecture was missing for those flatpaks so the software was just not available. Linux distributions generally try to enable as much architectures as possible when packaging, not just x86_64.

A second issue I've had on my Pinebook Pro is that it has a 64GB rootfs. Using too many flatpaks is just very wasteful of space. In theory you have a runtime that has your major dependencies and then a few Megabytes of stuff in your application flatpak. In practice I nearly have an unique platform per flatpak installed because the flatpaks depend on different versions of that platform or just on different platforms.

Another issue is with end users of some of my Flatpaks. Flatpak does not deal well with software that communicates with actual hardware. A bunch of my software uses libusb to communicate with sepecific devices as a replacement for some Windows applications and Android apps I would otherwise need. The issue end users will run in to is that they first need to install the udev rules in their distribution to make sure Flatpak can access those USB devices. For the distribution packaged version of my software it Just Works(tm)

Flatpak does have it's uses

I wouldn't say Flatpak is completely useless. For certain usecases it is great to have available. It think Flatpak makes most sense for when closed source software would need to be distributed.

I would like to see this be more strict though. I wouldn't want to have flatpaks with holes in the sandbox with a proprietary license for example. Which is exactly what the Edge flatpak is.

It's quite sad that Flatpak madness has gone so deep into the Gnome ecosystem that it's now impossible to run the nice Gnome Builder IDE without having your application in a flatpak. (EDIT: Turns out that using Builder without Flatpak is possible again)

I don't think having every app on a Linux machine being Flatpak is anything I'd want, If I wanted to give developers that much power to push updates to anywhere in my system without accountability I'd just go run Windows.

Alpine Linux is pretty neat

Martijn Braam — Wed, 01 Feb 2023 15:44:57 -0000

I've used various Linux distributions in the past, starting with a Knoppix live CD a long time ago. For a long time I was an Ubuntu user (with compiz-fusion ofcourse), then I used Arch Linux for years thinking it was the perfect distribution. Due to postmarketOS I found out about Alpine Linux and now after using that for some years I think I should write a post about it.

Installing

Ubuntu has the easy graphical installer of course. Installing Arch Linux the first time is quite an experience the first time. I believe Arch since has added a setup wizard now but I have not tried it.

Installing Alpine Linux is done by booting a live cd into a shell and installing from there just like Arch but it provides the setup-alpine shell script that runs you through the installation steps. It's about as easy as using the Ubuntu installer if you can look past the fact that it's text on a black screen.

A minimal Alpine installation is quite small, that combined with the fast package manager makes the install process really really quick.

Package management

The package management is always one of the big differentiators between distributions. Alpine has it's own package manager called APK, the Alpine Package Keeper. While it's now confused with the android .apk format it predates Android by two years.

The package management is pretty similar to Archlinux in some aspects. The APKBUILD package format is very similar to the pkgbuild files in Arch and the packages support similar features. The larger difference is the packaging mentality: Archlinux prefers to never split packages, just one .pkg.tar.zst file that contains all the features of the application and all the docs and development headers. Alpine splits out all these things to subpackages and the build system warns when the main package contains any documentation or development files.

For a minimal example of this let's compare the tiff library. In Alpine Linux this is split into 5 packages:

tiff, the main package that contains libtiff.so.6 [460 kB]
tiff-dev, the development headers [144 kB]
libtiffxx, the c++ bindings [28 kB]
tiff-doc, the documentation files [5.21 MB]
tiff-tools, command line tools like ppm2tiff [544 kB]

In Arch Linux this is a single package called libtiff that's 6.2 MB. For most Linux users you'd never need the library documentation which takes the most space in this example.

The end result is that my Archlinux installations are using around 10x the disk space my Alpine installations use if I ignore the home directories.

Some more differences are that Alpine provides stable releases on top of the rolling edge release branch. This improves reliablity a lot for my machines. You wouldn't normally put Arch Linux on a production server but I found Alpine to be almost perfect for that usecase. Things like the /etc/apk/world file makes management machines easier. It's basically the requirements.txt file for your Linux installation and you don't even need to use any extra configuration management tools to get that functionality.

There's also some downsides to apk though. Things I'm missing is optional packages and when things go wrong it has some of the most useless error messages I've encountered in software: temporary error (try again later) . Throwing away the original error and showing "user friendly" messages usually does not improve the situation.

Glibc or not to glibc

One of the main "issues" that get raised with Alpine is that it does not use glibc. Alpine Linux is a musl-libc based distribution. In practice I don't have many problems with this since most my software is just packaged by in the distribution so I wouldn't ever see that it's a musl distribution.

Issues appear mostly when trying to run proprietary software on top of Alpine or software that's so hard to build that you're in practice just getting the prebuilds. The solution to proprietary software is... don't use proprietary software :)

For the cases where that's not possible there's always either Flatpak or making a chroot with a glibc distribution in it.

Systemd

Beside not using glibc there's also no systemd in Alpine. This is one of the things I miss the most actually. I don't enjoy the enormous amount of different "best practices" for specifying init scripts and the bad documentation surrounding it. So far by best solution for creating idiomatic init scripts for alpine is just submitting something to the repository and wait until someone complains about style issues.

Beside that I'm pretty happy with the tools openrc provides for manageing services. The rc-update tool gives a nice consise overview of enabled boot services and the service tool just does what I expect. It seems like software is starting to depend on systemd restarting it instead of fixing memory leaks which causes me some issues sometimes.

Conclusion

Alpine Linux is neato. I try to use it everywhere I can.

Configuring Blackmagic Design converters in Linux

Martijn Braam — Thu, 18 Aug 2022 23:35:25 -0000

Blackmagic Design makes a lot of video converters in various sizes and feature sets. These are relatively cheap boxes that convert between HDMI, SDI and optical fiber for video signals. There's also some neat converters that can do audio embedding and de-embedding.

Normally these converters are configured using a Windows or OSX utility. Just plug in a converter with an USB cable and change one of the few settings. This is pretty nice except there's no utility for Linux.

Making a Linux utility

I already maintain pyatem/openswitcher which is a reverse engineered protocol implementation for the ATEM video switcher devices from Blackmagic Design. I decided to also take a look at the USB protocol for the converters to add this to the pyatem python library.

I started by looking at the BiDirectional SDI/HDMI 12G micro converter. To look at the traffic I installed USBPCap on a Windows machine. This is not the most convenient tool but it's the quickest way to get USB logs.

With USBPCap running I started the official software and changed every setting once and then ended the capture. With this captured traffic I could start to look at how the protocol works using Wireshark.

Looking at an USB dump using Wireshark

The protocol is relatively simple, it only uses USB control transfers and retrieving/writing settings is done by sending a setting name to the device and then reading or writing the value.

The more difficult part of this design is that you need to know in advance which settings exist on which converter. There does not seem to be a way to enumerate a list of the settings from the device.

Using this information I wrote the pyatem.converter module that implements the USB protocol using libusb and also added a definition that describes all the settings for the BiDirectional converter I used.

With the library implemented I greated a simple GTK3 application that presented the settings in a similar way to the official software:

Converter Setup on Linux

Supporting more hardware

A setup utility that only works with one model of converters is not super useful. It also creates the risk of overfitting the library to some specific protocol quirk that one model has, and this is exactly what happened in this case.

I ordered the HDMI to SDI and SDI to HDMI 3G converters since these are cheap and really common. They are also an older generation of converters. For those not familiar with this hardware: the 12G converters are the 4K capable ones and the 3G converters top out at 1080p30. The number refers to the link speed.

I created another packet dump using Windows to find which fields are supported and to my suprise the protocol for these converters is completely different.

The protocol for these 3G converters does not work by sending a setting name and then a value. Instead the settings are numbers and the operation is done in a single usb transaction. This is possible since a numeric setting can be set in the wValue field of a read or write operation to signify what you want to read or write.

Due to this I had to refactor the whole library to support multiple protocol backends. This would also make it possible to support more kinds of Blackmagic USB protocols for their different products like the ATEM Mini hardware and the monitors. These are normally configured using seperate setup utilities.

Going mobile

Since I also work on postmarketOS I tried running my Linux app on a PinePhone. The only change I had to make it removing the "Blackmagic Design" name from the converter name because the product names are so ridiculously long. After that the Linux desktop setup utility I made Just Works(tm) on Linux phones:

Converter Setup on postmarketOS

After sharing this picture there seemed a bit of interest for having a mobile app for configuring the converters. This makes a lot of sense, the converters are usually spread out all over the place and dragging a laptop around to configure them is annoying.

While it's nice that this works on postmarketOS supported phones, the majority of people still have Android and iOS phones. Since I already know how the protocol works and doing a simple Android app shouldn't be too hard... I decided to learn Android development.

The Android app

I had tried to do Android development before and always got stuck very early in the process making me never continue it. It seems like Kotlin is a great improvement over the Java issues I was having though. I managed to get a proof of concept up and running that read the settings in a single converter in a day and continued developing. I also signed up for a Google Play developer account and went through the verification steps. It turns out dealing with the Google Play Console is definetly the hardest and most time consuming part of development.

It took about a week to get everything on the google side verified and reviewed. Luckily I did this with by first hello world demo and didn't wait until I had finished the rest of the app otherwise I would've lost a lot more time on this. In the week of waiting I fixed up the multi-protocol setup and refactored the code of the app to be actually nice instead of one large MainActivity.kt file with hardcoded values.

The Converter Configuration Android app

My plan with the Android app is to make it paid and keep the opensource Linux app and libraries free. This makes me able to fund the development of the open source side through the Android app and give a wider audience the ability to actually use the app. The nice thing is that the Play store already takes care of all the payment and licensing for this.I can just keep using the funds from the Android app to get more hardware to reverse engineer and add to the app.

The Android app is now in open beta. The beta can be joined through https://play.google.com/apps/testing/nl.brixit.setup. Currently the app only supports the three converters mentioned above but I hope to provide an update soon to at least get the UpDownCross converter working.

I'm not planning to make an iOS app for this though, that would require me to buy into the Apple ecosystem with multiple devices to even get started and learn a whole new software stack. I'm also not entirely sure if apps controlling non-class-compliant usb hardware is even possible in the Apple mobile ecosystem.

Why I left PINE64

Martijn Braam — Wed, 17 Aug 2022 09:47:04 -0000

Linux hardware projects are made or broken by their community support. PINE64 has made some brilliant moves to build up a mobile Linux community, and has also made some major mistakes. This is my view on how PINE64 made the PinePhone a success, and then broke that again through their treatment of the community.

I want to start by pointing out that this is me leaving PINE64 and not the projects I'm involved in like postmarketOS. These opinions are my own yadda yadda...

Community Editions and the PinePhone's early life

The original PinePhone was brought up on the existing Linux Mobile projects like Ubuntu Touch, postmarketOS, and Maemo Leste, and also spawned new Linux distributions like Mobian and Danctnix ARM. This grew until there were 25 different projects working on the PinePhone — an apparently thriving community.

Following the initial set of Developer Editions, intended for community hackers to build the software with, the first consumer-targeted PinePhone devices were the Community Editions. A batch of PinePhones was produced with a special branded back cover for five community projects: UBPorts, postmarketOS, Mobian, Manjaro, and KDE Plasma Mobile. Every Community Edition phone sold also sent $10 to their respective Linux distribution projects.

Working together through these Community Editions, Linux distributions built a software ecosystem that works pretty well on the PinePhone.

The end of community editions

In February 2021, PINE64 announced the end of the community editions. At this moment, PINE64's focus shifted from supporting a diverse ecosystem of distributions and software projects around the PinePhone to just supporting Manjaro Linux alone.

The fact that a useful software ecosystem for the PinePhone exists at all is thanks to the diverse strategy employed by PINE64 in supporting many distributions working together on each of the many facets of the software required. Much of the original hardware bring-up was done by Ubuntu Touch. Mobian developers built the telephony stack via their eg25-manager project. And in my role for the postmarketOS distribution, I developed the camera stack.

Manjaro Linux has been largely absent from these efforts. The people working on many of the Linux distributions represented in the community editions tend to work not just on packaging software, but on building software as well. This is not the case for Manjaro, which focuses almost entirely on packaging existing software. Supporting Manjaro has historically done very little to facilitate the development of the software stack which is necessary for these devices to work. In some cases the Manjaro involvement actually causes extra workload for the developers by shipping known broken versions of software and pointing to the developers for support. Which is why https://dont-ship.it/ was started.

Regardless, Manjaro is now the sole project endorsed and financially supported by PINE64, at least for the Linux capable devices. As a consequence it has a disproportionate level of influence in how PINE64 develops its products and manages the ecosystem.

The last straw

With community members influence in PINE64 diminished in favor of a Manjaro mono-culture, what was once a vibrant ecosystem has been reduced to a bunch of burnt-out and maligned developers abandoning the project. The development channels are no longer the great collaboration between various distributions developing PinePhone components and there are now only a small number of unpaid developers working on anything important. Many of PINE64's new devices, such as the PinePhone Pro, PineNote, and others, have few to no developers working on the software — a potential death blow for PINE64's model of relying on the community to build the software.

Everyone has had a different "last straw". For me, it was the SPI flash situation.

There is a substantial change to booting between the PinePhone and PinePhone Pro. Previously, each distribution could develop a self-contained eMMC or microSD card image, including a compatible bootloader and kernel distribution. Installation is as simple as flashing a microSD card with the desired distribution and popping it in.

On the PinePhone Pro, the hardware works differently: it prefers to load the bootloader from the eMMC instead of the microSD. This means that when the PinePhone Pro shipped from the factory with Manjaro on the eMMC it will always boot the Manjaro u-Boot, even when booting from a microSD card. We no longer have any control over the bootloader for these devices.

There is a solution, however. The hardware can have an SPI flash chip that gives a bit of storage to put U-Boot in and that storage is always preferred over the eMMC and microSD storage. The problem with this is that all the distributions need to agree on a U-Boot build to put in there, and agree to never overwrite it with a distribution-specific version.

The solution to this is Tow-Boot: a distribution of U-Boot that can be put in that flash chip. With this the U-Boot firmware can just be treated like system firmware and be updated through fwupd independent of what distributions ship. This would work not only for the PinePhone Pro, but would also enable things like installing your preferred Linux distribution on a PineBook Pro by popping in a flash drive with a UEFI installer, much like you can on any other laptop.

Negotiating this solution was hell. Manjaro is incentivized not to agree to this, since it cedes their sole control over the bootloader, and PINE64 listens to Manjaro before anyone else. Furthermore, PINE64 does not actually want to add SPI flash chips to their hardware. Apparently, there has been some issues with people using SPI flash as RW storage on the A64-LTS boards, which would be a support issue.

After months of discussions between the community, Manjaro, and PINE64 leadership, we finally were able to convince them to ship the PinePhone Pro with an SPI flash chip with Tow-Boot installed on it.

But the Pinebook Pro has a similar boot configuration, and thus a similar problem. Some time after the PinePhone Pro was shipped, it was time for a new Pinebook Pro batch, and this discussion started again. The same arguments were re-iterated by all sides all over again, and the discussion went nowhere. PINE64 representatives went so far as to say, quote, "people who want [an SPI chip] can just solder one on". This batch of Pinebook Pros has ended up shipping without Tow-Boot flashed.

So I left

This is the moment I left. I left all the official channels, stepped down as PINE64 moderator. Left the private developer chat rooms. PINE64 cares only about Manjaro, and Manjaro does not care about working with any other distributions. This is no longer a community that listens to software developers. As a representative of postmarketOS, there is no further reason for me to be directly involved with PINE64 if the only opinions that matter are those of Manjaro.

Like many others, I have become burnt out on this ecosystem. So I quit. I am no longer getting random requests to make Manjaro's factory software work. No longer am I enduring the stress and frustration of these meaningless discussions behind the scenes, and after not being in the PINE64 for some weeks I can definitely say I'm way less stressed out.

Now I can just focus on making postmarketOS work better. On the PINE64 hardware, and all the many other devices supported by postmarketOS.

I hope that future vendors will make better choices, and listen to the actual whole community. Maybe even help with the software development side.

The end of the nice GTK button

Martijn Braam — Thu, 24 Mar 2022 18:43:07 -0000

The above picture is a GTK3 button with the Adwaita theme. I like this button. I would even say this is by far my favorite button.

I personally feel like Adwaita has been the pinnacle of Linux design and that the Gnome design guidelines are absolutely great. Apps are laid out in a reasonably consistent way. The purpose of every widget is very clear. The design even works very well on Linux phone formfactors.

The design for Gnome apps has stayed fairly consistent for at least a decade, there's been some minor design changes over the years which you can easily see by opening Gnome Control Center since it seems to be in a perpetual cycle of moving some panels to a new styling but leaving other ones untouched.

Here is an example of an older panel. Nice boxes with square corners.

This is a newer panel. It uses the ListBox with nice rounded corners. I quite like this design.

I think Gnome has really nailed this design. But in practice I'm using Adwaita dark on most systems. The dark theme, while not officially supported as a normal application theme, works absolutely brilliantly and is a great example of how to design a dark theme.

Everything is still very clear in the dark variant and the general theme of Adwaita has been nicely maintained. It doesn't look like it's just an inverted color scheme for Adwaita or Adwaita with the brightness lowered.

Thanks to Libhandy it's even possible to make nice responsive layouts that scale between desktop use and mobile use without looking out of place. The new widgets introduced with Libhandy neatly fit in with the normal GTK3 theming of widgets. I've been making use of these new widgets a lot when designing applications and I think some of these make desktop design a lot easier and should've been part of GTK3 at the start.

GTK 4

Now GTK3 is the "Old school" way to do UI. GTK4 has been in development for a bit and has improved a lot of the internals. One of the great upsides is that it can take more advantage of the GPU when rendering the UI.

The Adwaita theme has also been nicely carried over and looks very similar to the GTK3 counterpart.

This has all the features I like from GTK 3. The only issue with it is that font rendering looks horrific, but that might just be my machine. Also the dark theme is quite nice.

Libadwaita

So now we're in the other part of my blog post. The things I don't like. When I want to make a gtk4 application for mobile I would need libhandy, but libhandy for gtk4 is not a thing. The "solution" is libadwaita. This provides the widgets I need but it comes with the downside of having some of the worst decisions in application theming.

This is a libadwaita application. With the new libadwaita theming it falls in the pitfalls of modern design. Everything is flat now. Borders on widgets have been eradicated! Blood for the flat design gods!

I personally don't like the flat look. I think Microsoft was stupid for doing it. I think Apple is stupid for doing it and I've been praising Adwaita for being the sane option in an insane world. This is sadly over now.

I don't think removing all borders and gradients counts as design. That's just lazy and completely devoid of having any personality.

My design opinion aside, I think the theme has objective issues as well. This is just a huge decrease in accessibility. The buttons in the headerbar look the same as titles now. The only hint that New List is a button is because it happens to be a split button and has a dropdown icon. It looks identical to the window title otherwise.

For some reason the only button controls in the headerbar with a slight hint of buttoniness are the window controls because they have a background. Ironically these are the only controls in the old theme that didn't look like buttons.

I feel like the designers of this new theme have never sit down with anyone who's not a "techie" to explain to them how to use a computer. While a lot of people now instinctively hunt for labels that have hover effects, for a lot of people who are just not represented in online computer communities because they're just using the computer as a tool this is completely weird. I have had to explain to people tons of times that the random word in the UI somewhere in an application is actually a button they can press to invoke an action.

The dark theme

I've not been able to make the libadwaita apps on my system change theming or even switch to the dark variant, so I'll only have screenshots from around the internet.

In this dark theme the edges and divider in the theme suddenly become light. I find this really jarring and it looks like it's just an inverted theme. This is a minor complaint and otherwise the dark theme is pretty well done except for the flatness issues also present in the light theme.

The community

Now one of the worst parts is that everywhere I only even hint at not completely loving the new libadwaita theme I instantly get shut down and disagreed with before I can even get the chance to give some feedback. Apparently not liking flat themes makes me a madman in this world. Why am I not allowed to even have opinions about the look of the operating system I'm using?

The feedback I get is that I should move to QT/KDE, but I think that theming has had the same issues for way longer already and I do really like the Gnome HIG.

I'm also an application developer, the only choice I have is either staying on developing gtk3/libhandy apps for as long that's supported or start with making libadwaita applications which means I'll be making apps that I don't like the look of, which is incredibly depressing.

Gnome activists have already been doing the whole "Don't theme my app" thing, why are they theming my app now? this goes both ways!

Going IPv6 only

Martijn Braam — Sat, 19 Feb 2022 18:37:41 -0000

Since we're slowly coming up to 25 years of having IPv6, but still not actually having IPv6, I tried making a seperate local network that only has v6 access. In theory quite a few of the popular internet services should have IPv6 connectivity working.

From previous network configuration mistakes, I've learned that it takes quite a while before I notice the loss of IPv4 connectivity if IPv6 is still working. Usually, I notice that by having the search engine work perfectly fine, but none of the websites I click loading.

Setting up the network

It was quite easy to set up the v6-only network for me. I already have all the network traffic running through a Linux router so I just made a new bridge that gets IPv6 addresses only and drop all IPv4 traffic. I've assigned a network interface to this bridge and also an SSID on the wireless network.

$ brctl addbr v6only
$ iptables -A FORWARD -i v6only -j DROP
$ iptables -A INPUT -i v6only -j DROP
$ ip6tables -A FORWARD -i v6only -j ACCEPT
$ ip6tables -A INPUT -i v6only -j ACCEPT

First problems

The first problem didn't take very long, I had a complete lack of ipv6 reachable nameservers in my network.

The nameserver in my network is unbound running on a Linux machine. It's configured to do recursive resolving and caching and I had never set it up for IPv6.

I ran into a few interesting issues with setting up unbound. I had created an ACL that allowed the IPv6 subnet on my LAN to use the recursive resolver by marking that subnet as a private network. It took quite a bit of debugging to figure out why it refused to return AAAA records for my local services. I had created an AAAA record for my local Matrix server and I could resolve it just fine through third-party nameservers, but got no results at all from unbound. It turns out that marking an IPv6 subnet as private also makes DNS responses that return an address in that subnet dissapear due to the scrubber.

The scrubber makes a lot of sense. It prevents a website called evil.com to make a DNS record called local.evil.com pointing to 127.0.0.1 or any other local address. The issue with such a record is that browsers will allow JavaScript to connect to services on your machine this way. This works absolutely fine for IPv4 for me, but on IPv6 the public address for the service is the same as my local address for it since there's no NAT involved. After quite a bunch of searching the fix turned out to be adding a single line to the config:

private-domain: brixit.nl

This allowed my matrix.brixit.nl AAAA record to function both inside and outside the network.

Infrastructure fixes

Now I've been connected to the internet again I ran into my second issue. While all the postmarketOS infrastructure has IPv6 addresses, none of it had it's AAAA records created after the last infrastructure migration. This was luckily quite easy to fix.

Beside the infrastructure that postmarketOS runs itself there's also gitlab.com/postmarketOS. Luckily IPv6 is also fully supported on Gitlab. It looks like Github and Sourcehut are behind in this and are not reachable over IPv6 at all.

This means most postmarketOS work is possible on an IPv6-only connection. Issues are mostly that the sources for quite a lot of project are on github or sourcehut which means builds are failing locally. Also the build infrastructure for postmarketOS is running on Sourcehut builds which means I can't see the logs of build failures (but the artefacts are on the postmarketOS infrastructure and accessible again)

Inaccessible websites

So in my testing I come across several websites that didn't work yet.

Reddit
Hackernews
Twitter
Github
Sourcehut
Hackaday
Stackoverflow

I guess killing a bunch of the social media would make me a bit more productive. The code forge access is quite a lot more problematic. I can totally understand though that Microsoft doesn't have the funds to deploy IPv6, it's a tiny startup after all.

So is this useful at all

Running an IPv6-only network isn't super useful for daily use. There's simply too many services that still fully rely on IPv4 to function.

Occasionally plugging into the IPv6 testnet though is very useful. It's great for finding bugs, the kinds of bugs that lead to still not being able to use IPv6 after decades. Since practically all IPv6 deployments are dualstack it's not very noticable when IPv6 is misconfigured because practically all software silently falls back to IPv4.

Just using this network for an hour was enough to iron all the IPv6 issues on the postmarketOS and my own infrastructure so I'd say this was worth it.

PinePhone Camera pt5

Martijn Braam — Sun, 13 Feb 2022 20:20:05 -0000

It's been a while since I've written anything about the Megapixels picture processing. The last post still showcases the old GTK3 version of Megapixels even!

In the meantime users figured out how to postprocess the images better to get nicer results from the PinePhone camera. One of the major improvements that has landed was the sigmoidal contrast curve in ImageMagick.

convert img.tiff -sharpen 0x1.0 -sigmoidal-contrast 6,50% img.jpg

This command slightly sharpens the image and adds a nice smooth contrast curve to the image. This change has a major issue though, this is a fixed contrast curve added to all images and it does not work that great for a lot of cases. The best result was running this against pictures that were taken with the manual controls in megapixels so they have the right exposure.

On the PinePhone the auto exposure in the sensor tends to overexpose images though. Adding more contrast after that will just make the issues worse. In the header image of this post there's three images shown generated from the same picture. The first one is the unprocessed image data, the second one is the .jpg created by the current version of Megapixels and the third one is the same data with my new post-processing software.

Waveform visualisation of the banner image

This screenshot shows the waveform of the same header image. This visualizes the distribution of image data on the horizontal axis it's the horizontal position of the image and on the vertical axis it's the brightness of all the pixels in that column plotted. Here you can still see the 3 distinct images from the header image but with different distribution of the color/brightness data.

One of the main issues with the data straight from the sensor is that it's mostly in the upper part of the brightness range, there's no data at all in the bottom quarter of the brightness range and this is visible as images that have no contrast and look grayish.

The sigmoidal contrast curve in the middle image takes the pixels above the middle line and makes them brighter and pixels below the middle line and makes them darker. The main part that's improving is the data extending further in the lower part here, but due to the curve the bright parts of the image become even brighter and the top line shows that the data is clipping.

The third image with the new algorithm instead moves the data down by keeping the bright pixels in the same spot but "stretching" the image to the bottom. This corrects for the blacklevel of the sensor data and also creates contrast without clipping the data.

How

This started with me trying to make the postprocessing faster. Currently the postprocessing is done with a shell script that calls various image manipulation utilities to generate the final image.

Megapixels will take a burst of pictures and saves those as seperate .dng files in a temporary directory. From that series the second one is always used and the rest is ignored. With dcraw the image will be converted to rgb data and stored as tiff. Imagemagick will take that and apply the sharpness/contrast adjustment and save a .jpg

Because these tools don't preserve the exif data about the picture exiftool is ran last to read the exif from the original .dng files and save that in the final .jpg

Importing and exporting the image between the various stages is not really fast, and for some reason the processing in Imagemagick is just really really slow. My plan was to replace the 3 seperate utilities with a native binary that uses libraw, libjpeg, libtiff and libexif to deal with this process instead.

version 1 of postprocessd

The new tool is postprocessd (because it's supposed to run in the background and queue processing) It uses libraw to get rgb data, this is the same library that's used in dcraw. Then the resulting data is written directly to to libjpeg to create the final jpegs without any processing in between. This is what actually generated the first image shown in the banner. Processing a single .dng to a .jpg in this pipeline is pretty fast compared to the old pipeline, a full processing takes 4 seconds on the PinePhone.

The downside is that the image looked much worse due to the missing processing. Also just having a bunch of .jpeg files isn't ideal. The solution I wanted is still the image stacking to get less noise. With the previous try to get stacking running with HDR+ it turned out that that process is way way way too slow for the PinePhone and the results just weren't that great. In the meantime I've encountered https://github.com/luigi311/Low-Power-Image-Processing which uses opencv to do the stacking instead. This seemed easy to fit in.

Version 2 with opencv for stacking

This new code takes all the frames and converts them with libraw. Then the opencv code filters out all the images that are low contrast or fully black, because sometimes Megapixels glitches out. The last .dng file is then taken as a reference image and all the other images are aligned on top of that with a full 4 point warping transform to account for the phone slightly moving between taking the multiple pictures. After the aligning the pictures are averaged together to get a much less noisy image without running an actual denoiser.

This process produced an image that's exactly the same as the output files from v1 but with less noise.

Before and after stacking

This is a zoomed in crop of a test image that shows the difference of noise. The results are amazing for denoising without having artifacts that make the image blurry. But for every upside there's a downside. This is very slow.

Stacking 2 images together with the current code takes 38 seconds. For great results it's better to stack 2 images though.

Color processing

Now the opencv dependency is added it's pretty easy to just use that to handle the rest of the postprocessing tasks.

The main improvement here is the automatic blacklevel and whitelevel correction. The code slightly blurs the image and then finds the darkest and brightest point. Then it's simply substracting the value of the darkest point to shift the colors in the whole image down and the colored haze is removed. Then the pixels get multiplied with a calculated value to make the brightest pixel pure white again which "stretches" the brightness range so it fills the full spectrum. This process adds the contrast like the old imagemagick code did, but in a way more carefully tuned way.

After this a regular "unsharp mask" sharpening filter is run that's fairly agressive, but tuned for the sensor in the PinePhone so it doesn't look oversharpened.

A last thing that's done is a slight gamma correction to darken the middle gray brightness a bit to compensate for the PinePhone sensor overexposing most things. The resulting contrast is pretty close to what my other Android phones took, except the resolution for those phones is a lot better.

What's left to do

The proof of concept works, now the integration work needs to happen. The postprocessing is quite CPU intensive so one of the goals of postprocessd is to make sure it never processes multiple images at the same time but instead queues the processing jobs up in the background so the CPU is free to actually run Megapixels. There's also still some bugs with the exif processing and the burst length in the current version of Megpixels is a bit too short. This can probably be made dynamic to take more pictures in the burst when the sensor gain is set higher.