BrixIT Blog

Squirrel Strings

Martijn Braam — Tue, 05 Sep 2023 15:00:35 -0000

Don't you sometimes wonder why a whole class of security vulnerabilities exist in programming languages? Some languages try to fix issues like memory safety vulnerabilities, but one issue that's encountered on The Web™ more often is injection vulnerabilities.

username = request.get("username")
for row in execute(f"SELECT * FROM users WHERE username={username}"):
	frubnicate(row)

The possibility for injection vulnerabilities is everywhere. Most famously SQL injections where string templating or concatenation is used to build SQL commands. It's still very easy to create one of these vulnerabilities because the APIs are perfect for injection.

The PHP documentation for MySQL

The only mitigation for this is warnings in the documentation and lets be fair, a lot of people won't read documentation.

This is not only an issue for SQL. There's a lot of possibilities for injection like HTML injection in template engines making it possibly to perform XSS attacks. Or shell injection vulnerabilities in embedded devices.

server = requests.get("dns")
os.system(f'echo "{server}" > /etc/resolv.conf')

For running commands it's safer and recommended to use the APIs that expect a list of arguments instead of a string, bypassing string concatenation. For HTML injection template engines usually protect the application by escaping variables fed into it.

For SQL the solution is parameterized queries to seperate the actual SQL from the data supplied by users. But what if this distinction was a programming language feature instead?

Introducing squirrel strings

So why call it squirrel strings? Because naming things is hard and I thought of this for fixing SQL.

A squirrel string is a string that is made from a string template but not finalized into an actual string. In this example I'll show how something like this would work in Python since there it can fit in quite nicely.

username = request.get("username")
for row in execute(s"SELECT * FROM users WHERE username={username}"):
	frubnicate(row)

This is the exact same example as the worst-case example above but with a single letter changed. The f-string has become an s-string. Now it seems to be doing the exact same thing, but in a secure way. The major difference is hidden here in the execute method.

A hypothetical squirrel string language construct would function the same as a Python f-string but would not return a plain string, instead it returns an object containing the literal part of the template and the data injected.

>>> foo = "World"
>>> demo = s"Hello {foo}!"
>>> demo
<Squirrel "Hello ?!" ["World"]>
>>> demo.template
["Hello ", "!"]
>>> demo.data
["World"]

For this to not to introduce weird edge-cases where the squirrel-string is used after changing the data the squirrel string should evaluate the expressions in the template before storing it in the squirrel object.

>>> foo = s"1+2={1+2}"
>>> foo.template
["1+2=", ""]
>>> foo.data
[3]

An SQL query

So with this language feature it would be way easier to make a safer database API that accepts SQL strings. To ensure injection doesn't happen through regular strings type-checks can be used to check at runtime and type annotations can be used at "compile" time.

def execute(query: Squirrel):
    if not isinstance(query, Squirrel):
        raise SecurityError()
    sql = "?".join(query.template)
    return cur.execute(sql, query.data)
    
execute(s"UPDATE users SET password={password} WHERE id={userid}")
# Becomes UPDATE users SET password=? WHERE id=?

With this in place it would be safe to template SQL strings because the insecure user data never becomes actually part of the query, which I think is a way better solution than making developers use more verbose APIs to ensure the safety.

Another side benefit is that the type of the data placed into the template is preserved so numbers would not be passed as strings into the query. The constant part of the query could also be cached internally by the database API as a prepared statement without the developer having to use another API.

In the real world

So this is all a random shower thought. None of this is actually implemented and I might be missing some obvious flaw here that causes this to not work.

The API for consuming squirrel-strings is something I thought up while writing the post and for certain use-cases a completely different API might be more useful. Maybe have the literal string as "Hello !" and a dict with data and offsets into the template. Maybe the api should stringlify all incoming data anyway.

The examples also use Python here since it's relatively easy to fit this into Python. I don't know of any other programming language that has flags for string literals embedded into the syntax.

But even if this is a perfect idea with no downsides, it wouldn't fix injection vulnerabilities immediately. Languages would have to create some syntax for defining squirrel strings. Libraries need to start accepting squirrel strings and at some point compatability with existing code would need to be broken, otherwise the "legacy" codepath would still be the default insecure one.

But in any case I wanted to put the idea out there.

Update

It was pointed out to me that this feature is almost fully in javascript already. Javascript has Tagged Templates which allow something similar but not as an actual object, the string is instead converted in the arguments for a function.

The neat thing is that does make it possible to implement this in javascript using that language feature:

function squirrel(template, ...args) {
    // This should probably be something more proper but
    // I'm not a JS programmer
    let result = new Object();
    result.template = template;
    result.data = args;
    return result;
}

let name = "unsafe data";
let example = squirrel`SELECT * FROM people WHERE name=${name}`;
console.log(example.template) // ['SELECT * FROM people WHERE name=', '']
console.log(example.data)     // ['unsafe data']

This functionality is in fact used by some Javascript libraries already to run SQL queries.

Expensive cameras are actually better

Martijn Braam — Wed, 16 Aug 2023 00:01:48 -0000

It has been quite a while since I've bought a "new" camera. A lot of my photography has been done on a Nikon D3300. This is a 2014 entry-level DSLR. While this camera is marked as entry level it has practically all the features you'd ever need for photography and I never really have had any issues with picture quality of this camera.

Now the world is moving on to mirrorless cameras it has become almost affordable to buy a 10 year old second hand professional camera, so I snatched up the D750 right after Nikon introduced a new batch of Z-series mirrorless cameras. I've always wanted to use some full-frame cameras but the price difference has been absurd.

When I bought the D3300 somewhere in ~2015 If I remember correctly I did a comparison between specs of a bunch of the current Nikon cameras and came to the conclusion that getting something more expensive as the entry level just wasn't worth it. A comparison of the models at the time:

The D3300 "entry level" for ~300 euro at the end of 2015.
The D5300 "upper entry" for ~500 euro. This adds a tilting screen mostly.
The D7200 "enthusiast" for ~1000 euro. Higher max ISO and faster shutter.
The D610 "high end" for ~1500 euro. This one is the cheapest full frame model but was old-gen by that time already.
The D750 "high end" for ~2000 euro. The same generation but full frame.
The D810 "professional" for ~3000 euro. Takes both SD cards and Compact Flash
The D4s "flagship" for ~6000 euro. Still only takes Compact Flash cards.

So the most obvious spec in speclistings but the least important for 99% of the work, sensor resolution. Almost all these cameras are the same 24.7 megapixels resolution except for the D810 that is slightly newer and has 36 megapixels and the D4s that is a bit older and has 16 megapixels.

A lot of specifications in webshop spec lists for these models are almost or exactly the same between these models, mainly because these are mainly based around the Expeed 4 SoC. When I chose the D3300 I was upgrading from a second hand D70 and my budget allowed only choosing between the two cheapest options. The main differentiator betwen those two models are that the screen tilts and it has a lot more autofocus points, I deemed the tilting feature completely useless on a DSLR because the main selling point for these cameras is that you have an optical viewfinder. Since I didn't need more autofocus points than the D70 I just got the cheapest one.

The D3300 is a great camera, it has significantly better noise performance than the D70 which I practically had to keep below ISO 800 to don't get super noisy results.

Looking at some benchmarks

I'm not sure if this data was available yet when I was deciding between models almost a decade back, but there is an amazing camera comparison website called DXOMARK. It has a spec comparison between the D3300 and the D750. This gives a better picture of camera differences through actual measurements of the sensor.

For some reason it's really hard or impossible to find which sensor is in which model camera which seems like an obvious spec to list since it's the most important component of the camera. Some of the major sensor differences between these two cameras:

The old one is a DX size sensor and the D750 is an FX camera. This has some upsides and downsides I'll get at later.
The bit depth is 12 vs 14 bit. And since since that's not super intuitive, this means it captures 4x more levels, not 1.16x the amount.
11 vs 51 autofocus points.
Up to 5dB improvement in noise level at high-iso use. This is almost half the noise on max ISO.
12.7dB vs 14.5dB of dynamic range at the minimum ISO. This means I can have a lot brighter highlights in my shot without the sensor clipping.

After using the D750 for some weeks these differences turned out to make quite a bit of difference in which shot I could still make without things going noisy or blurry.

User experience differences

Another difference you won't easily see from spec sheets is the massive difference in UX.

The D750 on the left and the D3300 on the right

On first glance the UI on the devices is pretty similar and on photos from camera reviews you'd think these behave pretty similar. It could not be more different.

The D3300 turns on the screen for practically any action you do that doesn't require looking through the viewfinder. The bottom two rows are the quick-access settings for changing some of the main behaviors of the camera like autofocus/manual focus. The exposure and focus areas and the ISO. Some of the settings can be changed with holding a combination of hardware buttons and scrolling the thumbwheel but most of the configuration happens through this UI. The way you use those software quick-settings is with the d-pad to navigate through it.

The D750 does seem to have roughly the same buttons on the back but the UI is missing the two bottom rows of settings. It's not only missing those two rows but it's actually not possible at all to change many of these settings using the d-pad and the screen. This pretty quickly forced me to learn all the hardware buttons which is probably a good thing since this is a lot faster to use.

A few examples of the usage differences:

To change ISO on the D3300 I have to first move the camera out of the automatic modes using the mode dial on the top and then use the "i" button beside the screen to enter the actual quick settings menu. Then With the d-pad I move to the ISO option and with the "ok" button it opens a dialog to select the ISO with the d-pad buttons again. This does not allow changing between auto/manual ISO, that happens a few layers deep in the main menu.For the D750 I hold the button on the back that has "iso" above it and then I use the thumbwheel to scroll through the options. I for this action the back screen lights up quickly to show what I'm doing but there's also a tiny ISO indicator in the viewfinder to see the options while I'm scrolling through them. The list of options include "auto" so no seperate menu path is needed.
To change between autofocus and manual focus on the D3300 there's another quick-setting menu that showing "MF" in the picture. Using the same procedure as the ISO option this can be navigated to switch between "MF", "AF-S" and "AF-C". The availability of these options is dependent on which picture mode the camera is in.On the D750 there's an actual physical toggle on the side to switch between MF and AF very quickly. There's a bunch more AF variants in this camera and also more focus point modes. These extra options can be accessed by pressing the button in the center of the toggle and using the thumbwheel the AF mode can be changed between continuous, single and auto. The second wheel on the front can change the focus point mode at the same time between single-point, all points, groups and 3d-tracking.

Changing the shutter speed or aperture on the D3300 depends on the mode. Since there's a single thumbwheel on this camera it will pick the "main" setting on the mode you're in. On shutter-priority mode it will control the shutter and on apererture-priority mode it will control the aperture. In program or manual mode the wheel controls the shutter speed. To access the aperture setting the exposure compensation button has to be held downOn the D750 there's two wheels so the rear one is always shutter speed and the front one is always aperture control.
To change between release modes like single-photo, burst, timer and quiet mode the D3300 has the release mode button below the d-pad that pops up another menu to navigate.The D750 has a second ring below the mode selector dial that you can rotate to select these modes between Single, Continuous low-speed, Continuous high-speed, Quiet, Quiet-continuous, Timer and mirror-lock-up.

The mode dial on the D750 has more "professional" options than the D3300. The D3300 mode dial is filled with magic useless modes like portrait, sport, landscape, baby?, macro and night-mode. These options are basically very slight variations on the auto program. It also has a dedicated position to open the in-camera manual for some reason.The D750 mode dial has all the special-auto modes under a single "scene" position so you can easily skip them and it adds the U1 and U2 positions instead. These are fully programmable positions that store the entire camera configuration. I have one of these programmed for focus trapping and one fixed ISO 100 and 1/160 shutter. Another small detail is that the mode dial has a locking button that prevents accidentally changing the mode.

There is a lot more of these differences, practically all the settings you'd ever need are accessible using some button combination and the wheels and you can see the settings you're adjusting through the viewfinder to never lose sight of the subject of the picture. Some of these settings are especially nicely done and I really like how it spells out "3D" using the focus points when selecting the 3D focus tracking mode.

The D750 viewfinder when selecting the 3d focus mode

Another difference isn't the added physical buttons but there's a whole second display, I was used to having this display on the D70 but the smaller entry cameras are sadly missing this.

The top LCD on the D750

This top display is an old-school LCD with an optional backlight that can be triggered to read it in the dark. This picture shows basically all the settings you can read on this display but one of the nice features is that this is an always-on display. When the camera is off the amount of remaining shots is still visible (which is the 1.2k above) and when the SD card is removed it shows [-E-] instead. This makes it really quick to see if the camera is ready while it's still in my case.

Using the camera

There's quite a bit of things I noticed while using this camera the first few days. One of the first things that was obvious is how much better the autofocus system on this camera is. Not only is there a bit more granularity with the extra autofocus points the lenses also get focus a lot quicker.

On the D3300 I mainly kept the camera in single-point single-push focus mode when possible because it relatively often picks a completely wrong autofocus point in the full-sensor mode. I also barely use the continuous focus modes because it does not super reliably pick the right moments to keep focussing and stopping it again when reframing shots.

Another point where the D3300 focus is lacking quite a bit is subjects that are moving quickly. The 3d tracking autofocus on the D750 can actually keep up with a puppy charging towards you at full speed.

Another big change is mainly due to switching to a much larger sensor. This changes the lens selection I use quite a bit. Since I've never planned to get a full-frame camera I never considered that as an option while buying lenses. My most used primes on the D3300 are the Nikor 35mm and 50mm and of those I mainly use the 50mm one. Due to the smaller sensor on the old camera it means that most of my photos are effectively shot on 75mm.

At first using the lenses was quite weird. The 50mm lens on the D750 was behaving more like the 35mm lens on the D3300 which would have a 52mm effective focal length in that setup. But when putting the 35mm lens on the D750 I got... also 52mm. This is because my 50mm prime is a full frame lens and the 35mm one is a DX lens. The D750 detects this and automatically crops down the pictures in this case.

This annoyed me for quite a bit and I considered getting a new prime in the 75-85mm range to get back my favorite focal length but then I realized that I could just use my 70-300mm telephoto lens as my "standard'" lens instead. While this would be quite unwieldy to use on the D3300 it works great on daily use of the D750 and since this is a lens with optical stabilization it helps even more with low light usage.

I'm also quite happy that my most expensive lens, the 150-600mm happened to also be a full-frame lens. Dragging both the camera and this lens around is quite heavy but the results are amazing. With the improved autofocus it becomes possible to snap pictures of birds in flight even!

This picture would've never been in focus on the D3300

The lower effective focal length using the FX sensor on the lenses, combined with a lower noise floor has helped a massive amount with getting a lot of pictures successfully I would've never even attempted on the D3300 because I would have to push the ISO so high the results become useless or I would not be able to focus quickly and accurately enough to get the shot.

Astrophotography

I find astrophotography interesting, it's incredible what you're able to see using a camera pointed at the sky that you could never see with the naked eye. While I find it interesting I never gotten into it enough to actually pick gear specifically for astrophotography qualities. A few days ago I tried the D750 for astrophotography for the first time and the results have blown me away.

One difference here is that the D3300 has a pentamirror and the D750 has a pentaprism. This is a part that sits between the lens and the eyepiece of the camera and is responsible for reflecting the image through the lens a few times so it lines up with your eye while looking through the eyepiece and the picture isn't mirrored.

The pentamirror is the cheaper option which consists of a few mirrors glued together. This means that the light has to go through a few more glass to air transitions before it reaches your eyes which results in the image through the viewfinder being slightly darker. I had not noticed any difference really when using the D750 for normal photography but for astrophotography it made seeing stars through the viewfinder a lot easier.

Another difference is that on the D750 the autofocus system is actually good enough to autofocus on stars which makes the whole process a lot less painful.

And finally due to better low-light performance I was able to get way better results than my previous tries with the D3300.

The Andromeda Galaxy with the D750

Here's an example of a stack of 30 pictures with an ISO of 12800 showing the Andromeda Galaxy. I would put a D3300 picture for comparison here but at this focal length and shutter duration it just doesn't get enough light to extract the data from.

Conclusion

I'm glad I've picked up this camera. I've long thought the pro cameras were just a few thousands of euro added price to get a second SD slot and a few more hardware buttons. The reviews of these cameras seems like a lot of cork sniffing but if you compare the entry and pro cameras there's actually some significant improvements. A lot of these improvements also are most likely artifical market segmentation since this has the same CPU in both models and a lot of user experience are just missing software features.

There's some things that are suprisingly similar. One example is the video recording feature. The recording quality is practically the same on both cameras and the experience with anything relating to the live view is equaly horrible. These cameras are not for recording video. This also makes the tilting screen a useless feature.

For most pictures the D3300 is absolutely fine and as bonus it's also a lot lighter and smaller.

Developers are lazy, thus Flatpak

Martijn Braam — Sat, 03 Jun 2023 15:58:47 -0000

In the last decade I have seen a very slow but steady shift to solutions for packaging software that try to isolate the software from host systems to supposedly make things easier. My first experience with this was Docker, now Flatpak is the thing for desktop applications.

The promise of Flatpak

So the thing Flatpak is supposed to fix for me as developer is that I don't need to care about distributions anymore. I can bolt on whatever dependencies I want to my app and it's dealt with. I also don't need to worry about having software in distributions, if it's in Flatpak it's everywhere. Flatpak gives me that unified base to work on and everything will be perfect. World hunger will be solved. Finally peace on earth.

Sadly there's reality. The reality is to get away from the evil distributions the Flatpak creators have made... another distribution. It is not a particularly good distribution, it doesn't have a decent package manager. It doesn't have a system that makes it easy to do packaging. The developer interface is painfully shoehorned into Github workflows and it adds all the downsides of containerisation.

Flatpak is a distribution

While the developers like to pretend real hard that Flatpak is not a distribution, it's still suspiciously close to one. It lacks a kernel and a few services and it lacks the standard Linux base directory specification but it's still a distribution you need to target. Instead of providing seperate packages with a package manager it provides a runtime that comes with a bunch of dependencies. Conveniently it also provides multiple runtimes to make sure there's not actually a single base to work on. Because sometimes you need Gnome libraries, sometimes you need KDE libraries. Since there's no package manager those will be in seperate runtimes.

While Flatpak breaks most expectations of a distribution it's still a collection of software and libraries build together to make a system to run software in, thus it's a distribution. A really weird one.

No built in package manager

If you need a dependency that's not in the runtime there's no package manager to pull in that dependency. The solution is to also package the dependencies you need yourself and let the flatpak tooling build this into the flatpak of your application. So now instead of being the developer for your application you're also the maintainer of all the dependencies in this semi-distribution you're shipping under the disguise of an application. And one thing is for sure, I don't trust application developers to maintain dependencies.

This gets really nuts by looking at some software that deals with multimedia. Lets look at the Audacity flatpak. It builds as dependency:

wxwidgets
ffmpeg
sqlite
chrpath
portaudio
portmidi

So lets look at how well dependencies are managed here. Since we're now almost exactly half a year into 2023 I'll look at the updates for the last 6 months and compare it to the same dependencies in Alpine Linux.

audacity has been updated 4 times in the flatpak. It has been updated 5 times on Alpine.
ffmpeg has been updated to 6.0 in both the flatpak and Alpine, but the ffmpeg package has had 9 updates because if codecs that have been updated.
sqlite hasn't been updated in the flatpak and has been updated 4 times in Alpine
wxwidgets hasn't been updated in the flatpak and has been updated 2 times in Alpine
chrpath hasn't had updates
portaudio hasn't had updates in flatpak and Alpine.
portmidi hasn't had updates

This is just a random package I picked and it already had a lot more maintainance of the dependencies than the flatpak has. It most likely doesn't scale to have all developers keep track of all the dependencies of all their software.

The idea of isolation

One of the big pros that's always mentioned with Flatpak is that the applications run in a sandbox. The idea is that this sandbox will shield you from all the evil applications can do so it's totally safe to trust random developers to push random Flatpaks. First of all this sandbox has the same issue any permission system that exists also has. It needs to tell the user about the specific holes that have been poked in the sandbox to make the application work in a way that end users understand what the security implications of those permissions are.

For example here's Gnome Software ready to install the flatpak for Edge:

I find the permission handleing implemented here very interesting. There's absolutely no warning whatsoever about the bypassed security in this Flatpak untill you scroll down. The install button will immediately install it without warning about all the bypassed sandboxing features.

So if you do scroll down there's more details right? Sure there is!

There's a nice red triangle with the words Unsafe! pfew, everyone is fine now. So this uses a legacy windowing system which probably means it uses X11 which is not secure and breaks the sandbox. Well if that's the only security issue then it might be acceptable? Let's click that button.

Well yeah... let's hide that from users. Of course the browser needs to write to /etc. This is all unimportant to end users.

The even worse news is that since this is proprietary software it's not really possible to audit what this would do, and even if it's audited it's ridiculously easy to push a new more evil version to Flathub since practically only the first version of the app you push is thorougly looked at by the Flathub maintainers.

Even if there weren't so many holes in the sandbox. This does not stop applications from doing more evil things that are not directly related to filesystem and daemon access. You want analytics on your users? Just requirest the internet permission and send off all the tracking data you want.

So what about traditional distributions

I've heard many argument for Flatpaks by users and developers but in the end I can't really say the pros outweigh the cons.

I think it's very important that developers do not have the permissions to push whatever code they want to everyone under the disguise of a secure system. And that's my opinion as a software developer.

Software packaged by distributions has at least some degree of scrutiny and it often results in at least making sure build flags are set to disable user tracking and such features.

I also believe software in general is better if it's made with the expectation that it will run outside of Flatpak. It's not that hard to make sure you don't depend on bleeding edge versions of libraries while that's not needed. It's not that hard to have optional dependencies in software. It's not that hard to actually follow XDG specifications instead of hardcoding paths.

But packaging for distributions is hard

That's the best thing! Developers are not supposed to be the ones packaging software so it's not hard at all. It's not your task to get your software in all the distributions, if your software is useful to people it tends to get pulled in. I have software that's packaged in Alpine Linux, ALT Linux, Archlinux AUR, Debian, Devuan, Fedora, Gentoo, Kali, LiGurOS, Nix, OpenMandriva, postmarketOS, Raspbian, Rosa, Trisquel, Ubuntu and Void. I did not have to package most of this.

The most I notice from other distributions packaging my software is patches from maintainers that improve the software, usually in dealing with some edge case I forgot with a hardcoded path somewhere.

The most time I've ever spent on distribution packaging is actually the few pieces of software I've managed to push to Flathub. Dealing with differences between distributions is easy, dealing with differences between runing inside and outside Flatpak is hard.

But Flatpaks are easier for end users

I've ran into enough issues as end user of flatpaks. A package being on Flathub does not mean that it will be installable for an end user. I've ran into this by installing packages on the PineBook Pro which generated some rather confusing error messages about the repositories missing. It turns out that the Aarch64 architecture was missing for those flatpaks so the software was just not available. Linux distributions generally try to enable as much architectures as possible when packaging, not just x86_64.

A second issue I've had on my Pinebook Pro is that it has a 64GB rootfs. Using too many flatpaks is just very wasteful of space. In theory you have a runtime that has your major dependencies and then a few Megabytes of stuff in your application flatpak. In practice I nearly have an unique platform per flatpak installed because the flatpaks depend on different versions of that platform or just on different platforms.

Another issue is with end users of some of my Flatpaks. Flatpak does not deal well with software that communicates with actual hardware. A bunch of my software uses libusb to communicate with sepecific devices as a replacement for some Windows applications and Android apps I would otherwise need. The issue end users will run in to is that they first need to install the udev rules in their distribution to make sure Flatpak can access those USB devices. For the distribution packaged version of my software it Just Works(tm)

Flatpak does have it's uses

I wouldn't say Flatpak is completely useless. For certain usecases it is great to have available. It think Flatpak makes most sense for when closed source software would need to be distributed.

I would like to see this be more strict though. I wouldn't want to have flatpaks with holes in the sandbox with a proprietary license for example. Which is exactly what the Edge flatpak is.

It's quite sad that Flatpak madness has gone so deep into the Gnome ecosystem that it's now impossible to run the nice Gnome Builder IDE without having your application in a flatpak. (EDIT: Turns out that using Builder without Flatpak is possible again)

I don't think having every app on a Linux machine being Flatpak is anything I'd want, If I wanted to give developers that much power to push updates to anywhere in my system without accountability I'd just go run Windows.

Can't get blinky on the BL602

Martijn Braam — Mon, 01 May 2023 16:46:41 -0000

One of the boards I have in my parts box is the PINE64 BL602 evaluation board. The "PineCone". It's a competitor to WiFi enabled microcontrollers like the Espressif products.

I put it aside since when I received it there was not really any software to go along with it and I don't have anywhere near the experience to bring up a board like this.

Getting blinky running 3 years laters

This product has been out for 3 years now so lets see if it has formed a community.

So if you are not familiar with blinky, it's the Hello World! of the microcontroller world. The goal is have a led connected to one pin of the board and toggle that pin on and off every second to make a blinking led.

Let's check the website first. The official page for this from PINE64 itself is https://pine64.com/product/pinecone-bl602-evaluation-board/. Since the only official thing is the webshop which doesn't list any documentation or links to progress further on how to use this thing I consider it a dead end. Let's move on to the pine64.org side.

The community side is the wiki page at https://wiki.pine64.org/wiki/PineCone. This has the specs again and a block diagram. It also has the datasheets for the components on the board and the schematics. Not anything I actually need to get started using this product.

Let's compare to the Raspberry Pi Pico, a competitor that's doing quite well. When you search for this you land on the official product page https://www.raspberrypi.com/products/raspberry-pi-pico/ that has the shop link, the specifications but more importantly it's a hub linking to the "Getting started" guide and the documentation for users.

So lets scroll down on the PineCone wiki page since it's my only hope for docs. Thee quarters down this long wiki page is a list of flashing utilities. I don't have anything to flash yet but I look forward to figuring which of the 6 listed flashers fits my personality best /s.

The rest of this page is links to random github repositories and blog articles. The top link being the link to the SDK for this board. It links to https://github.com/pine64/bl_iot_sdk and is labeled as "compilers, linkers, and all the code to build on Windows, Linux (x86_64), and MacOS". The last activity here was 2 years ago. This seems mostly focussed on the reverse engineering effort but lets have a look.

bl_iot_sdk

This repository has a README with again more links to a lot of different places. One of these is linking to https://pine64.github.io/bl602-docs/ which is a sphinx generated documentation site. This is several pages too deep from the official pine64 website but I forgive that since it has the magical words I've been looking for:

So I'm going down this path, it almost seems familiar to the way the Pi Pico sdk is used. Clone the repository and create an environment variable pointing to the checkout folder.

Next it tells me how to compile the example apps, but I don't want to compile the example apps, those probably work and they are in-tree so those probaby Just Work(tm) without setting up the scaffolding you should do for your own projects. I just want to build a main.c that does blinky. This part of the documentation sadly stops after compiling and flashing the built-in apps.

I have continued browsing this documentation for a bit but it does not show at all how to make a minimal blinky example and build it. The rest of the documentation is mainly using the advanced features, various flashing systems and debugging systems and image formats.

So to compare this with the Pi Pico documentation: The first thing I find when I read the documentation of the SDK is:

Blinky in the Pi Pico docs

Chapter 1.2 is the first chapter with actual documentation and it starts with all the things I want for getting started with yet another microcontroller board:

A very simple .c file that implements blinky without using any advanced features of the hardware.
The output of tree showing exactly which folder structure I should have and where the files go
The CMakeLists.txt that builds this application

The BL602 documentation has shown me none of the above yet. At this point I give up and just open 20 tabs in my browser with all the linked pages from the wiki.

All these links are random flashing utilities, porting new programming languages and porting new frameworks. Also a lot of them showing how to flash the example apps to the thing.

PEOPLE DON'T BUY MICROCONTROLLERS TO FLASH EXISTING EXAMPLES

Getting ANYTHING to run

So there is no documentation on how to make blinky for this platform whatsoever. I give up and figure it out myself from the sdk example apps.

There's no blinky in the examples, the simplest application I could find was the helloworld one that outputs some text over uart. This uses FreeRTOS to get that working so there's a complete lack of basic single .c file demo applications for this.

More issues

This SDK has multiple things I don't like. The most important one is that it ships prebuilt compilers in the source tree. This is not a proper SDK, this is some Android-level build mess.

The whole point of using nice RISC-V boards is that this is a standard architecture, all Linux distributions already ship a maintained RISC-V toolchain so why would I ever use some random precompiled binaries shipped by a Chinese vendor.

Why is this using Makefiles to build the thing instead of cmake which the Raspberry Pi foundation have already proven you can integrate way neater. This is just throwing hardware over the wall and linking to the chip vendor documentation, hoping the community can match a proper made tutorial, proper documentation and make a new SDK that's not a mess.

I guess I just dump this thing back in the parts bin and wait some years. Quite sad since this seems to be some really nice hardware and a great alternative to the existing microcontroller platforms.

Digital Aerochrome

Martijn Braam — Sun, 30 Apr 2023 11:11:55 -0000

A long long time ago in the 70s there was a lot of interesting film being made. At this time film was used for everything, professional and consumer cameras, movies, aerial photography. A lot of time has gone into recreating film looks for digital cameras now to reproduce the color response of the old film cameras.

One of the more popular examples of this is the Fujifilm X-series digital cameras that have film simulations built in. I've heard they are quite good but those cameras are quite expensive and I find it a bit useless to get a seperate camera for just that trick.

One legendary film stock you won't see simulated on anything like this is Kodak Aerochrome III Color Infrared. This film was originally produced for aerial photography. Most infrared film is black and white film that's also sensitive to infrared. What makes Aerochrome so special is that it's a color film that is infrared sensitive and uses false color to visualize the infrared spectrum.

While it's not possible to get real Aerochrome anymore it is possible to get the datasheet for it. This old datasheet has a great explanation of why this film exists, how it works and has color spectrum sensitivity data.

Color film is made from multiple layers of photosensitive material. Each layer has a different sensitivity spectrum and will produce a specific color when developed. For example one layer will be sensitive to red light and when developed will produce a cyan dye. This will create the negative image on the color negative. When this negative is inverted it the cyan will become the red part of the image again.

Sensitivity spectrum of Kodak Ektar 100 color negative film

But the color produced on the negative doesn't have to correlate with the original color that was recorded. When the spectrum of the sensitive layer doesn't correlate with the produced color dye it's called false color film.

Sensitivity spectrum for Kodak Aerochrome III Infrared color negative film

The scale for this datasheet is quite different because it has to include the infrared spectrum. One important thing to note is that the Aerochrome film is normally used in combination with a yellow filter on the lens, this will remove most of the blue light from the photograph. Then the false color is mapped so that infrared light will create cyan which inverts back to red. The red sensitive layer will produce magenta and develop to green. The green sensitive layer will produce yellow dye which produces blue on the final image. All the three layers are also sensitive to blue light which is why the filter is used together with this film.

Reproducing on a digital camera

The difficulty with reproducing this film is that cameras are designed to be not sensitive to infrared light. Even if the infrared light filter is removed from a camera the infrared light will just be detected as regular red light. This is not very useful because for the proper false color the visible red light should be shifted to green while the infrared should stay red. My solution to this is taking two pictures. One regular picture to capture the normal color spectrum and one with a R72 filter. This filter in front of the lens will let only light above 720nm pass which is exactly the part of the infrared spectrum I need.

To the naked eye this filter just looks black since human eyes aren't sensitive to infrared light. On my camera it produces a way more interesting result though.

This is the infrared light stored into the red channel of the picture. Now the trick to make this looking like aerochrome is combining the R and G channels from the normal picture and the R channel from the picture with the IR filter. This is annoyingly hard to do correctly with most software, especially if you need to align the images.

Custom software

So aligning two images and some very easy color manipulation is something I had already written before in postprocessd. I made a copy of this software and changed the pipeline to align the IR image on top of the RGB image. Since this is using libraw you can just feed it any supported camera raw file.

After stacking the images are combined into a single 4-channel RGBI image. Then a 4x3 matrix is run over this image to get the final 3-channel RGB output.

Yep, doing it this way is probably overkill compared to scripting a series of imagemagick commands. But it does allow the flexibilty of doing more complicated transforms than just moving the color channels around. The datasheet says to use a yellow filter because all the layers are sensitive to blue light, so what happens if I modify this matrix to leak blue light into it?

Yellow filter on the left, No yellow filter on the right

Mixing in the blue channel into the three other channels will lower the contrast of the image a bit, it also makes the sky a lot lighter again since... that was quite blue.

Since I have this option I added the -y argument to the tool to disable the yellow filter.

My software for this is available on https://git.sr.ht/~martijnbraam/aerochrome

Can this be done realtime?

One neat feature of the Fujifilm X-series cameras is that it gives a live preview of the film look while taking the picture. Since this process needs two pictures that would be impossible unless Fujifilm develops a sensor where IR cells are added to the bayer matrix.

Another option is hacking up somthing yourself. This is probably doable with a raspberry pi 4 compute module and two sensors, one regular IMX219 sensor and one NOIR option with the visible light filter added. To make it always align this can be shot through a beamsplitter prism which can be very small with these sensor sizes.

NitroKey disappoints me

Martijn Braam — Tue, 25 Apr 2023 18:39:47 -0000

There's an article making the rounds from NitroKey named "Smartphones With Popular Qualcomm Chip Secretly Share Private Information With US Chip-Maker".

This article is a marketing piece for selling their rebadged Pixel phones by picking a random phone and pointing at network traffic. It takes a look at a Sony Xperia XA2 and for some reason calls out Fairphone in particular.

The brand of the device should not really matter if this is a chipset issue as the article claims but it goes even further than just calling out other brands, it also additionally uses a custom rom to check these things instead of software supplied by those brands.

The second thing the article does is point out that WiFi geolocation exists and is done by Google and Apple by showing a screenshot from the Wiggle service that has nothing to do with that. Phones use Cell network, WiFi and network geolocation to get a rough location of a device, not for evil but for saving power. This prevents the need to run the GPS receiver 24/7 since most features don't need an exact location. There's no claims being made by NitroKey that their phone doesn't provide any of this.

After this we get to the main claim in the title of the article. The Qualcomm 630 chipset supposedly sharing private information with the manufacturer. The author of the article has found that the device connects to izatcloud.net and instead of doing the logical thing and opening izatcloud.net in a browser they do a whois request and then figure out it's from Qualcomm, They also proceed to contact Qualcomm lawyers instead of following the link on this page. The webpage hosted on this domain does conveniently explain who owns the domain and what it's purpose is and it's associated privacy policy. But that doesn't sound nearly as spooky.

The next section makes the claim that this traffic is HTTP traffic and is not encrypted. It proceeds to not show the contents of this HTTP request because it would show that it's not at all interesting. It does not contain any private data. It's just downloading an GPS almanac from Qualcomm for A-GPS.

The A-GPS data is only there to make getting a GPS fix quicker and more reliable. GPS signals are pretty weak and getting a lock indoors from a cold start (the device has been off for some time) is hard. Inside the GPS signal sent by the satellites there's occasional almanac data that compensates for things like atmospheric distortions, without the almanac your GPS position wouldn't even get within a few hundred meters of your actual position. Since this signal is only occasionally broadcast and you need to listen to a GPS sattelite for an extended amount of time (the broadcast takes around 10 minutes) it's easier for these modern devices to just fetch this information from the internet. Qualcomm provides this as a static file for their modems.

This feature isn't even only in the Qualcomm 630 chipset, it's in practically all Qualcomm devices. Some third party Android roms go as far as to obscure the IP address of your phone by proxying this http request with another server. The rom they have tested obviously didn't.

This feature is not even limited to Qualcomm devices, this practice happens in practically all devices that have both GPS and internet because people don't like waiting very long for their position when launching their navigation software. The NitroPhone has their GPS provided by Broadcom chips instead of Qualcomm ones so obviously it won't make the same HTTP requests, doesn't make it any more or less secure though.

Now the main issue, is this personal data? The thing that gets leaked is your IP address which is required because that's how you connect to things on the internet. This system does not actually send any of your private information like the title of the article claims.

I'm disappointed

The reason for articles like this is pretty obvious. They want to sell more of their phones for a massive profit margin. The sad thing about making these "Oh no all your data is leaking!!!" articles is that when there's actual leaks it won't stand out between all the marketing bullshit. The painful part is that it's actually working. See the outrage about companies not having ethics and not following laws.

This feature is not breaking laws, it's not unethical, it's not even made for eeeevill.

Reverse engineering the BMD camera HDMI control, part 2

Martijn Braam — Fri, 14 Apr 2023 11:32:24 -0000

In the previous part of this series I cut open an HDMI cable between a Blackmagic Design ATEM Mini and a Pocket 4k camera to attempt to figure out how the ATEM Mini can control the camera over the HDMI cable only.

While reverse-engineering that I looked into the CEC lines and was unable to get the DDC data at the same time because cutting up a HDMI cable is not a reliable way to probe data. The first thing I did was make a neat PCB for debugging this further.

HDMI Passthrough PCB

This is a very simple board that has two HDMI connectors with all the high speed pairs connected straight accross and all the slow speed signals broken out to two pin headers. One 7-pin header gives probing access to all the pairs, voltages and ground and a 2x6-pin header allows breaking the connection between the two HDMI ports for any of the data signals by removing the jumpers.

PulseView result of plugging in the camera

With this setup I'm able to dump all the traffic I'm interested in. Plugging in the camera shows up as a glitch in the HPD line, then the SDA/SCL lines are busy for a bit communicating the EDID for the "display" implemented in the ATEM Mini HDMI input. Then the CEC line starts showing activity with the port number the camera is plugged into and some vendor specific commands.

Looking back at my previous post after I've figured out more of this and after reading parts of the HDMI and EDID specs it turned out that I already had all the data, I just didn't recognize it yet.

On my initial look at the CEC data I did not know which bytes are transmitted by which device. With just removing the CEC jumper from by board it became quite visible which data was sent by the camera and even without the CEC line being connected to the ATEM Mini at all. Also I noticed that the camera still knew what camera number it had. I initially assumed the first bytes containing the camera number were coming from the ATEM side. Since that connection is severed it must be getting this data from the EDID blob.

The EDID specifications

PulseView only shows annotations for half the EDID blob it has received. So I turned to the suprisingly great EDID Wikipedia page which documents all the bytes. My first try to figure things out whas the parse-edid command on Linux from the read-edid package. This does parse all the monitor and resolution data I don't want but does not seem to decode all of it. I pasted the EDID blob in my hex dump slicer and started annotating the bytes according to the Wikipedia page.

The annotated part covers the initial 128 bytes of the EDID blob with the basic monitor information. The num_exts byte here is set to 0x01 so immediately following it is more data, in this case a CEA-861 extension block. This block can contain more detailed resolution info (and more importantly, resolutions for more modern monitors). It also has space for custom data blocks. The first blocks are the well documented Video block, Audio block and Speaker block. The fourth block that exists is the Vendor block.

I made a wrong assumption here. I thought since this is a vendor block it would be a block with undefined data from BlackMagic Design. This block also contains the only byte that changes between the 4 ports of the ATEM Mini. The fourth byte of this block was 0x10, 0x20, 0x30 and 0x40 for the four ports which confused me even further, why is this using the high four bits.

After having another coffee and reading the Wikipedia page on EDID a bit further I found out that the first three bytes of the vendor block are the vendor identification which makes sense if you can have multiple of these vendor blocks. To my suprise the example value of a vendor id is 0x000c03 which is in my packet dump.

Turns out I was reverse engineering the HDMI 1.4 specification here. It's even possible to just download this specification! The most useful part of that PDF is this:

The changing 4 bits is the one marked A in this table. And... the A,B,C,D fields are for setting the CEC physical address for the device connected to that port.

So suprisingly so far everything here is just in-spec and documented and I now learned more on how CEC works. The camera reads the EDID block to know the output timings and also reads the CEC physical address from the ATEM. In my case the port number wasn't 0x40, it was the 0x40 0x00 which translates to CEC address 4.0.0.0

So if I want to remotely control my camera I need to do EDID emulation and insert this data block if the upsteam device did not already set it.

The CEC part

So lets have a look at the CEC communication after the EDID exchange has happened. First quickly after connecting there's two very short bursts of CEC data. PulseView decodes this as CEC pings so I will ignore those for now. This is an export of the rest of the CEC data in this recording:

8185-700390 CEC: Frames: 1f:84:20:00
712370-745164 CEC: Frames: 1f:a0:7c:2e:0d:01:01
748596-757793 CEC: Frames: 01:01
761361-775144 CEC: Frames: 10:02:01
778761-802480 CEC: Frames: 01:50:3c:00:00
806749-864354 CEC: Frames: 01:70:25:70:20:43:41:4d:20:25:69:00
868623-906865 CEC: Frames: 0f:a0:7c:2e:0d:02:02:10
911133-939694 CEC: Frames: 01:68:70:74:73:00
943963-962842 CEC: Frames: 01:65:32:00

The CEC protocol uses addressing with 4 bits for everything. In my packet dump for this trace the camera was connected to port 2.0.0.0 of the ATEM. The first byte of the CEC frame is also address info, but this is the logical address instead of the physical address. This is split in 4 bits for the sender and 4 bits for the receiver. Address 0 is always the CEC root which is the ATEM and Address F is the broadcast address. The camera uses address 1 for its communication. The second byte if the packet is the CEC opcode.

The first packet the camera sends is opcode 0x84. This is a mandatory packet that is broadcasted from the device that's connected to tell the CEC network about the mapping between the physical address and logical address. In this case logical device 0x1 is broadcasting that the physical address is 0x2000 which is 2.0.0.0.

The second packet is opcode 0xa0 which is "Vendor Command With ID". Now I've entered the reverse engineering area again. The next 3 bytes is 0x7c 0x2e 0x0d which corresponds to the OID for BlackMagic Design and the vendor data is 0x01 0x01. After this packet has been sent the communication starts breaking the CEC specification and is now just sending BMD specific data. All the data PulseView is trying to decipher from the bytes are just red herrings.

Emulating the ATEM

So now the basics of the communication are known the next part is emulating an EDID and seeing what happens on the CEC line to get more information. For this I'm using a Raspberry Pi Pico hooked up to my HDMI passthrough board.

I removed all the jumpers from the passthrough board to isolate the slow signals in the HDMI connection and hooked them all up to the Pico. On the initial tests I could not get any signals from the camera this way, I was expecting just pulling the hot-plug-detect pin high would be enough to start the HDMI connection. It turns out that I need to have a monitor connected to the high speed pairs to make the camera happy.

The first thing the camera does is reading the EDID so I started with implementing EDID emulation. For this the Pico should act as an I2C slave which is suprisingly undocumented in the SDK manual. The only thing the manual says about it is using the i2c_set_slave_mode(i2c0, true, 0x50) command to put the hardware in the right mode. The rest is undocumented and requires poking registers outside of the SDK for now, hopefully that will get fixed in the future. With this I implemented an I2C device that responds on address 0x50 and has a block of 256 bytes of data. This just contains a copy of the EDID of one of the ATEM ports for now.

The harder part is doing CEC on the Pico. So far it seems like nobody has made a library for this and due to the existence of CEC on the Raspberry Pi SBCs it makes searching pretty hard. In theory it should be possible to implement this using a PIO block to handle the bitbanging part. For now I'm just using one of the cores to bitbang the protocol.

This implementation just supports receiving CEC data and sending the ACK back to make the camera continue sending data. The debug pin in this trace is used to debug the sample point and packet detection in my code. The bit is sampled on the rising edge and the start of a bit is on the falling edge. During the ACK bit the gpio on the Pico is switched to output to drive the CEC line low.

The packet shown above is the first packet the camera sends after sending the BMD vendor comand. When connected to the ATEM the next thing that happens is that the ATEM sends a few unknown bytes back. If I don't reply on this packet the camera will restart the whole CEC session after 1.5 seconds by broadcasting it's physical address again.

So the first thing I tried sending back to the camera is CEC operation 0x9F. This is requesting the CEC version of the device. It turns out the CEC implementation in Blackmagic Design cameras is quite out of spec. It acks my data and then proceeds to never respond to CEC data again. Technically following the CEC 1.4 specification it was already out-of-spec because of sending a vendor command without first requesting the vendor id of the device it's connected to.

So since it's no longer really speaking CEC at this point I started looking into replaying some of the messages I had captured to see how the camera behaves. There's a few things that are sent from the ATEM to the camera directly after startup that don't seem to be correlated to any camera config.

The first thing is operation 0x01 directly after sending the vendor command. Then operation 0x50 and 0x70 and lastly another full CEC vendor command but to the broadcast address instead. After some testing it looks like operation 0x01 is required to make the camera "accept" the connection. It stops the restart of the session after 1.5 seconds. I can't figure out what operation 50 and 70 do but leaving those out does not seem to change anything.

The broadcasted vendor command is the tally data which I also can ignore for now. The next command I sent is 0x01 0x52 0x00 which sets the gain of the camera. By sending this directly after receiving the 0x02 the camera sends on startup the gain on the camera display changes!

Figuring out more opcodes

Now I have a somewhat working setup I tried once again changing settings in Atem software control and observing the CEC data. With this process I figured out 35 opcodes.

The reference document for this is "Blackmagic Camera Control Developer Guide". This document does not have any information on the HDMI protocol but it does document how to control the camera over SDI. The most important thing is the list of parameters that can be sent to the device. I was hoping the bytes in the HDMI protocol relate to information in that document in any way but it seems not.

It looks like the developers at Blackmagic Design created a completely new protocol for the CEC communication to deal with the update speed. The CEC bus is roughly 300 baud and the commands that are sent over the very fast SDI interface are just too long to have reasonable response times. The gain command in the CEC protocol has only a single data byte and that takes nearly a tenth of a second to transmit, the same command over SDI is 8 bytes long already.

While dumping the CEC traffic I also noticed some commands are significantly longer. An int16 value being 10 bytes long in this case. All these long commands for various features are also all on opcode 0x04. After looking at the similarities and differences on these long packets I noticed that this is just a passthrough command for the older SDI commands.

# Parameter 0.4 on HDMI (Aperture ordinal)
01:04:81:03:ff:00:04:02:00:10:00

# Parameter 0.4 on SDI
03:05:00:00:00:04:02:10:00

The byes are in different order and there's some extra static bytes for some reason but it does seem to map to the documentation. Sending one of these packets takes roughly 300ms, which is why this is not used for parameters you control using sliders or wheels in the control interface.

The whole list of parameters I found is:

05 00 00 00 00 00 00 00 00 # Reset lift
06 00 00 00 00 00 00 00 00 # Reset gamma
07 00 00 00 00 00 00 00 00 # Reset gain
0D xx xx                   # Lift R
0E xx xx                   # Gamma R
0F xx xx                   # Gain R
11 xx xx                   # Lift G
12 xx xx                   # Gamma G
13 xx xx                   # Gain G
15 xx xx                   # Lift B
16 xx xx                   # Gamma B
17 xx xx                   # Gain B
19 xx                      # Lift W
1A xx                      # Gamma W
1B xx                      # Gain W
1D xx xx                   # Hue
1E xx xx                   # Saturation
1F xx xx                   # Pivot
20 xx xx                   # Contrast
21 xx xx                   # Luma mix
33 1E                      # Show bars
33 00                      # Hide bars
40 xx xx                   # Absolute focus (ID 0.0)
41 xx xx                   # Focus distance
42 XX XX                   # Iris (ID 0.2)
43                         # Trigger instant auto-iris (ID 0.5)
44 XX XX                   # Set absolute zoom in mm (ID 0.7)
46 xx xx                   # Zoom
47                         # Trigger autofocus
52 xx                      # Gain
54 xx                      # Temperature
55 xx                      # Tint
56                         # Trigger AWB (ID 1.3)
57 xx xx xx xx             # Shutter (ID 1.5)
58 xx                      # Detail 0, 1, 2, 3

The exact encoding for the bytes still need to be figured out but it seems to mostly follow the encoding described in the SDI command manual.

All that's left is implementing a nice API for this in the Pi Pico to automate camera control :)

Mobile Linux camera pt6

Martijn Braam — Wed, 08 Mar 2023 15:57:25 -0000

The processing with postprocessd has been working pretty well for me on the PinePhone. After I released it I had someone test it with the dng files from a Librem 5 to see how it deals with a completely different input.

To my suprise the answer was: not well. With the same postprocessing for the PinePhone and the Librem 5 the Librem 5 pictures are turning out way too dark and contrasty. The postprocessd code is supposed to be generic and has no PinePhone specific code in it.

Fast forward to some time later I now have a Librem 5 so I can do more camera development. The first thing to do is the sensor calibration process I did with the PinePhone in part 4 of this blog series. This involves taking some pictures of a proper calibration target which in my case is an X-rite ColorChecker Passport and feeding that into some calibration software.

Because aligning color charts and making sure all the file format conversions with the DCamProf calibration suite from RawTherapee is quite annoying I got the paid graphical utility from the developers. By analyzing the pictures the software will generate a lot of calibration data. From that currently only a small part is used by Megapixels: the ColorMatrix and ForwardMatrix.

Calibration output snippet

These are 3x3 matrices that do the colorspace conversion for the sensor. I originally just added these two to Megapixels because these have the least amount of values so they can fit in the camera config file and they have a reasonable impact on image quality.

The file contains two more important things though. The ToneCurve which converts the brightness data from the sensor to linear space and the HueSatMap which contains three correction curves in a 3 dimensional space of hue, saturation and brightness, this obviously is the most data.

What is a raw photo?

The whole purpose of Megapixels and postprocessd is take the raw sensor data and postprocess that with a lot of cpu power after taking the picture to produce the best picture possible. The processing of this is built on top of existing open source photo processing libraries like libraw.

The expectations this software has for "raw" image data is that it's high bit depth linear-light sensor data that has not been debayered yet. The data from the Librem 5 is exactly this, the PinePhone sensor data is weirder.

Unlike most phones that have the camera connected over MIPI-CSI which gives a nice high speed serial connection to push image data, the PinePhone is connected over a parallel bus.

Rear camera connection from the PinePhone 1.2 schematic

This parallel bus provides hsync/vsync/clock and 8 data lines for the image data. The ov5640 sensor itself has a 10-bit interface though:

The D[9:0] is the 10 image data lines from the sensor

Since only 8 of the 10 lines are available in the flatflex from the sensor module that has the ov5640 in it the camera has to be configured to output 8-bit data. I made the assumption the sensor just truncates two bits from the image data but from the big difference in the brightness response I have the suspicion that the image data is no longer linear in this case. It might actually be outputing an image that's not debayered but does have an sRGB gamma curve.

This is not really a case that raw image libraries deal with and it would not traditionally be labelled "raw sensor data". But it's what we have. But instead of making assumptions again lets just look at the data.

I have pictures of the colorchecker for both cameras and the colorchecker contains a strip of grayscale patches. With this it's possible to make a very rough estimation of the gamma curve of the picture. I cropped out that strip of patches from both calibration pictures and put them in the same image but with different colors. I also made sure to rescale the data to hit 0% and 100% with the darkest and brightest patch.

Waveform for the neutral patches, green is the PinePhone and pink is the Librem 5

The result clearly shows that the the data from the PinePhone is not linear. It also shows that the Librem 5 is also not linear but in the opposite direction.

These issues can be fixed though with the tonecurve calibration that's missing from the current Megapixels pictures

postprocessd is not generic after all

So what happened is that I saw the output of postprocessd while developing it and saw that my resulting pictures were way too bright. I thought I must've had a gamma issue and added a gamma correction to the code.

With this code added it looks way better for the PinePhone, it looks way worse for the Librem 5. This is all a side effect of developing it with the input of only one camera. The correct solution for this is not having this gamma correction and have the libraw step before it correct the raw data according to the tonecurve that's stored in the file.

Storing more metadata

The issue with adding more calibration metadata to the files is that it doesn't really fit in the camera ini file. I have debated just adding a quick hack to it and make a setting that generates a specific gamma curve to add as the tone curve. This will fix it for my current issue but to fix it once and for all it's way better to include all the curves generated by the calibration software.

So what is the output of this software? Lumariver Profiler outputs .dcp files which are "Adobe Digital Negative Camera Profile" files. I have used the profile inspection output that turns this binary file into readable json and extracted the matrices before. It would be way easier to just include the .dcp file alongside the camera configuration files to store the calibration data.

I have not been able to find any official file format specification for this DCP file but I saw something very familiar throwing the file in a hex editor... The file starts with II. This is the byte order mark for a TIFF file. The field directly after it is not 0x42 though which makes this an invalid TIFF file. It turns out that a DCP file is just a TIFF file with a modified header that does not have any image data in it. This makes the Megapixels implementation pretty easy: read the TIFF tags from the DCP and save them in the DNG (which is also TIFF).

In practice this was not that easy. Mainly because I'm using libtiff and DCP is almost a TIFF file. Using libtiff for DNG files works pretty well since DNG is a superset of the TIFF specification. The only thing I have to do is add a few unknown TIFF tags to the libtiff library at runtime to use it. DCP is a subset of the TIFF specification instead and it is missing some of the tags that are required by the TIFF specification. There's also no way in libtiff to ignore the invalid version number in the header.

So I wrote my own tiff parser for this. Tiff parsers are quite hard since there's an enormous amount of possiblities to store things in TIFF files. Since DCP is a smaller subset of TIFF it's quite reasonable to parse it manually instead. A parser for the DCP metadata is around 160 lines of plain C, so that is now embedded in Megapixels. The code searches for a .dcp files associated with a specific sensor and then embeds the calibration data into the generated DNG files. If the matrices are also defined in the camera ini files then those are overwritten by the ones from the DCP file.

Results

The new calibration work is now in megapixels#30 and needs to go through the testing and release process now. There's also a release for postprocessd upcoming that removes the gamma correction.

For the Librem 5 there's millipixels#88 that adds correct color matrices for now until that has the DCP code added.

Sensors and PCB design

Martijn Braam — Thu, 02 Mar 2023 20:09:12 -0000

I do a lot of software development, I do enough of it to be comfortable with the process. I have worked on enough projects and have made enough pieces of software that it's quite easy to quickly spin up a new project to fix a specific issue I'm having.

Hardware design is a whole other can of worms. I have played with things like the Arduino and later the STM32 chips, ESP8266 and RP2040. These things are quite neat from a programmers perspective. You write your code in c++ and the included toolchain figures out all the hard parts and flashes the board. Hardware design is also quite simple since it's mostly putting together different hardware modules and breakout boards. Only a basic understanding of the hardware busses is required to get stuff up and running.

Projects usually look like the picture above. No resistors, capacitors or other components are needed. Just a breadboard (also optional), jumper wires and the IO modules you need to make your idea work.

Power is dealt with by the microcontroller board, the modules already include the pull-up resistors and power regulators required and the programmer is also included already.

The software side is also simplified and abstracted to the point where it's harder to get your average javascript project working than the embedded projects. You'd never have to open the 440 page atmega 328p manual to write the firmware. You don't even need to know about the register map for GPIO pins since things like digitalWrite(pin, state) exist.

It's easy to get a complete prototype running like this but then what?

Putting things in production

By far the easiest way to put the project in production is just shoving the breadboard in a project box and declaring it done. It works but it's not neat. This works if you only need one and it's for yourself.

When you need 5? 10? 30? of the same thing then breadboarding it will become a lot less practical. In some cases it's easy enough to just stick the modules on a piece of protoboard and solder it together. When that doesn't scale another option is putting the modules on a custom PCB.

This is exactly what I did for a project. This is the first PCB I have ever designed and put into production:

It's a PCB that only has pinheaders and connectors on it. This board would have a nodeMCU module and an off-the-shelf rs485 module on it. The board itself is incredibly simple but it did give me a chance to run through the complete process of going from a schematic to an actual physical product. Since it has no actual components on it it has removed a lot of worry if the board will work at all.

In this case the board was designed because I needed 15 of this design and this makes a lot more reliable and easy to maintain with the possibility to have neat screw terminals on it for the external connections.

Optimizing the design more

The above design was simple enough to not worry about optimizing it, but another project I wanted to take on is replacing my various sensor nodes with a neater custom design.

These boards have been put together around various ESP8266 dev boards, most of them are nodeMCU v3 boards, some are off-the-shelf boards that I receive data from with an rtl-sdr.

My plan is to make the one board to rule them all. A custom designed board that makes it easy to connect my existing sensors and allows a bit of extension. The design is based around an ESP-12F module since all my existing sensor code is already written for the various ESP8266 boards. It's also in stock at LCSC so that makes it a lot easier to get fabricated.

The design goals for this custom board is:

On-board programmer for development
Have the ESP module hooked up so it can deep sleep properly
Have a built in temperature sensor. Most of my external sensor boards are already temperature/humidity sensors so it makes sense to just include it since it's a cheap chip.
Have a screw terminal for hooking up onewire devices
Expose a silkscreen marked SPI and I2C bus
Be able to be powered with an USB Type-C cable or a single lithium cell.
Have a built-in charger for the lithium cell.
The board should be able to be assembled by JLCPCB and Aisler

For implementing this I mostly looked at the nodeMCU v3 schematic as reference and also made the basic layout of the PCB similar to that board. Initially I wanted to add the very popular DS18B20 onewire temperature sensor on-board but that got replaced with the SHT30 module that also includes a humidity sensor and connects to the I2C bus instead.

For the programming and USB part I included the CP2102 USB-to-serial converter chip that was also included on the nodeMCU board. It is probably overkill for this application though since I'm not using any of the functionality of that chip beside the RX/TX pins. I dropped the auto-reset circuit since I was worried it would interfere with the deep sleep reset circuit that I found more important.

One simple change from the nodeMCU design is that I swapped the USB Micro-B connector for a Type-C connector which requires a bit more PCB routing to deal with the reversable plug and two extra resistors to make it up to USB spec. The Type-C spec requires a 5.1kΩ resistor to ground on both CC pins (individually, don't cheap out and connect both CC pins to the same resistor like on the Raspberry Pi)

Most of this design work is simply reading the datasheets for the components and following the recommendations there. Some datasheets even have bits of recommended PCB layout in it to make the PCB layout easier.

Since all the complicated ESP8266 circuitry is already handled on the SoM I used and the USB-to-serial converter is laid out, the rest is simple connectors. Except... there's power stuff to deal with.

Most of the difficulty with this design is figuring out the power design. If the battery connection was not included it would've been relatively straightforward. Just have a power regulator that converts the 5V from the USB connector to the 3.3V for the rest of the board. This is exactly what the nodeMCU board does.

In my case the regulator has to deal with power coming from either USB or a single Lithium bringing the input voltage range from 3.4-ish to 5V. There's also a charger chip included to charge the connected cell and a bit of circuitry to switch between the battery and USB power.

The block diagram for the sensor board

The diagram above shows the final power design. It has the MCP1700 3.3V regulator to replace the LM1117 regulator from the nodeMCU design, the MCP part has a way lower quiescent current which helps with the battery life. The MCP73832 is a single cell li-ion/li-po charger that requires very few external components. In this design it's programmed to charge at 400mA which is not super fast but it should last some months on a single charge anyway.

The magic part is this circuit for the power switching:

This receives 5V from the USB connector or the battery power (or both) and feeds them into the regulator to generate the 3.3v rail. The MOSFET will disconnect the battery power when 5V is supplied and the diode makes sure that the power of the battery won't ever flow back into the 5V part of the schematic.

Since there's also a single analog input on the ESP8266 I thought it would be a good idea to use it to read the battery level. This is yet again a decision that added a bunch more complexity. The theory for this is pretty simple: you add a resistor divider to bring the 0V-4.3V of the battery to the 0V-1V range of the analog input and then you'd read the battery voltage with a bit of code. The issue of this is that the resistor divider will not only do the voltage scaling for the ADC but will also be a path for the battery to leak power.

This is the part that's added to fix that. With a few extra parts the resistor divider can be completely disconnected from the battery stopping the power leakage. The software on the ESP will have to first enable the measuring circuit by toggleing a gpio, reading the value with the ADC and then disableing the gpio again. Since there's already 5.1kΩ resistors on the board I'm using another one here together with a 100kΩ. This re-use of resistor values is done since some manufacturers charge extra per distinct part model used. This brings the voltage range of the battery down to about 0-0.2v. This seemed fine at the time since the ESP has a 10-bit ADC.

With using only a fifth of the ADC range the ADC resolution is already brought down to slightly less than 8 bits. But batteries usually don't go down to zero volts. The normal voltage range is 3-4.2V giving a voltage difference of 0.06V after the voltage divider making the result slightly less than 6 bits of resolution. This does not account for ADC noise in the system yet. The result is that the final battery level graph is a bit choppy but at least it functions.

The last thing left is connectors. I added a footprint for some screw terminals for the OneWire connection. This footprint is basically the same as a pinheader except it has a fancy outline. For the battery I added a JST-PH connector because it seemed the most common one for batteries.

The I2C and SPI bus don't have standardized connectors though. But, through a bit of searching I found the pmod spec. This is an open specification made by Digilent to connect modules to their FPGA development boards. It's perfect for this usecase since the interface is just pinheaders with a defined pinout and it defines the power supply as 3.3V which I already have on the board. Some of the sensors I want are even available as pmod modules.

The two pmod headers on the board

Getting the boards made

After laying out the PCB another complicated process begins. Getting it actually fabricated and assembled. The board has been designed that it can be hand-soldered if needed. No BGA parts are used and the _handsolder variants of the footprints are used for the common parts, which are slightly larger.

To not have vendor lock-in the board is designed to be assembled by both JLCPCB and Aisler. JLCPCB is the very cheap option in this case and Aisler is quite neat since it's actually in The Netherlands. The Aisler design rules are less forgiving than JLCPCB so I used those for the board. It mostly mean that I can't use the absolutely tiny vias that JLCPCB can drill.

For the assembly service metadata has been added to the schematic. For JLCPCB a LCSC column was added to the part data that contains the part code for ordering from LCSC. For Aisler the parts come from the other big parts warehouses instead like Mouser. For that the MPN column is added that contains the manufacturer part number.

With this data and the selected footprints the board can be assembled. I left out all the part numberes for the through-hole parts since those are either more expensive or impossible to assemble and also pretty easy to solder manually if required on that specific board.

To actually get it made Aisler has a nice plugin for the PCB editor in Kicad: Aisler Push. With this it's just a single button and the Kicad file will be sent off to Aisler where their servers will generate the necessary fabrication files from it. From there it's using the Aisler website to fix up the MPN matching with actual parts from various suppliers and pressing order.

For JLCPCB the process is more complicated. The fabrication files have to be generated manually. There's a tutorial for going through the various steps of generating the ~10 files you need and then those can be zipped up and uploaded to the website. Since the LCSC part codes are completely unique the assembly step of this process Just Works(tm) without having to adjust anything on the website.

If Aisler had an option to specify the exact part match in the schematic metadata instead it would probably be the easiest option in this case so I hope that gets figured out in the future. Two weeks later I had the board :)

The Firmware

For the firmware of this board I just re-used the one I had previously written for one of the nodeMCU based sensor boards and extended it for the sleep features. It's pretty basic firmware that does the standard WiFi connection things for the ESP module and connects to my MQTT server to push the sensor readings in a loop.

For the battery operated boards that wouldn't be enough though. With that firmware the battery would run out in hours. There's a great series of blog posts from Oppoverbakke that go into great detail on how to optimize deep sleep on the ESP module. Especially the last post on avoiding WiFi scanning is very helpful for reducing power use.

To avoid as many delays as possible the WiFi accesspoint mac address and channel are saved into the memory of the RTC inside the module which is the only part of the chip left powered in the deepest sleep mode. Then with that information the module will save around one second of time being awake and connecting. Another optimisation is using a static ip address instead of DHCP to save another second.

I don't like having static ip addresses in my network. I have static DHCP leases for everything instead. This is why I extended the RTC connection code to do a DHCP request the first time and then save the received DHCP data in the RTC memory together with a counter to re-do DHCP after some time. This means that my board is still a fully DHCP compliant device without making a DHCP request every time it wakes up, this is what the lease time is for after all. Thanks to this I only have to power down the board when I need to change the IP address instead of reflashing it or building in some configuration web interface.

During deep sleep the whole board uses 19.1µA with a half charged battery in my tests and the transmission is down to to a bit less than a second at 70mA, but the power use of the transmissions varies quite a lot.

I had one module in production outdoors quickly to test. It sleeps for one minute in deep sleep and doesn't have any other optimisations. Looking at the data from that module it looks like it would last for about two weeks, even with the sub-zero temperatures during the night.

The battery life for the optimized version has yet to be seen. I have calculated it to be around 3 months but it would take a while to have conclusive results on that. This board uses a "3500mAh" 18650 cell which I assume in the calculations is actually 2000mAh.

Next revisions

During testing and programming I found out I had made a few mistakes with the board design. The most problematic one is that I put zener diodes on the UART lines between the USB-to-serial converter and the ESP module. This was to prevent power from the serial lines to flow into the CP2102 module and powering up that chip partially.

This did not work. When I got the boards I spend a bit of time figuring out why the programmer couldn't connect. The voltage drop through the diodes I picked is probably too much for the serial communication to work. To make things worse the diodes are completely unnecessary for this since the uart lines won't leak power in deep sleep anyway. Luckily this is easily fixable on my boards by making a solder bridge across the diodes.

Another issue on the board is that the battery charger chip gets very hot and is too close to the temperature sensor. I also forgot to add the thermal relief specified in the layout suggestions of the part so while charging that chip is practically constantly at 80°C making the value of the temperature sensor raise by around 5°C. Since this only affects the board when the battery is charging it's not a critical fault.

For cost optimization I skipped battery protection on the board. For my uses I just ordered a protected 18650 cell so it should be fine. But since I'm making a second revision board to fix these issues anyway I decided to include a battery protection chip this time around.

The protection chip sits between the ground terminal of the battery and the actual board ground and disconnects the battery pack in case of overcurrent, short circuits, overcharge or undercharge.

Another small change I did is add an extra pinheader to the board to access the UART lines directly, since pinheaders are basically free anyway.

You should make your own boards

Making boards is fun. I learned a lot during the process and now have some important skills to use in my other projects. Can you really call yourself a full stack programmer if you haven't designed the hardware yourself? :D

Alpine Linux is pretty neat

Martijn Braam — Wed, 01 Feb 2023 15:44:57 -0000

I've used various Linux distributions in the past, starting with a Knoppix live CD a long time ago. For a long time I was an Ubuntu user (with compiz-fusion ofcourse), then I used Arch Linux for years thinking it was the perfect distribution. Due to postmarketOS I found out about Alpine Linux and now after using that for some years I think I should write a post about it.

Installing

Ubuntu has the easy graphical installer of course. Installing Arch Linux the first time is quite an experience the first time. I believe Arch since has added a setup wizard now but I have not tried it.

Installing Alpine Linux is done by booting a live cd into a shell and installing from there just like Arch but it provides the setup-alpine shell script that runs you through the installation steps. It's about as easy as using the Ubuntu installer if you can look past the fact that it's text on a black screen.

A minimal Alpine installation is quite small, that combined with the fast package manager makes the install process really really quick.

Package management

The package management is always one of the big differentiators between distributions. Alpine has it's own package manager called APK, the Alpine Package Keeper. While it's now confused with the android .apk format it predates Android by two years.

The package management is pretty similar to Archlinux in some aspects. The APKBUILD package format is very similar to the pkgbuild files in Arch and the packages support similar features. The larger difference is the packaging mentality: Archlinux prefers to never split packages, just one .pkg.tar.zst file that contains all the features of the application and all the docs and development headers. Alpine splits out all these things to subpackages and the build system warns when the main package contains any documentation or development files.

For a minimal example of this let's compare the tiff library. In Alpine Linux this is split into 5 packages:

tiff, the main package that contains libtiff.so.6 [460 kB]
tiff-dev, the development headers [144 kB]
libtiffxx, the c++ bindings [28 kB]
tiff-doc, the documentation files [5.21 MB]
tiff-tools, command line tools like ppm2tiff [544 kB]

In Arch Linux this is a single package called libtiff that's 6.2 MB. For most Linux users you'd never need the library documentation which takes the most space in this example.

The end result is that my Archlinux installations are using around 10x the disk space my Alpine installations use if I ignore the home directories.

Some more differences are that Alpine provides stable releases on top of the rolling edge release branch. This improves reliablity a lot for my machines. You wouldn't normally put Arch Linux on a production server but I found Alpine to be almost perfect for that usecase. Things like the /etc/apk/world file makes management machines easier. It's basically the requirements.txt file for your Linux installation and you don't even need to use any extra configuration management tools to get that functionality.

There's also some downsides to apk though. Things I'm missing is optional packages and when things go wrong it has some of the most useless error messages I've encountered in software: temporary error (try again later) . Throwing away the original error and showing "user friendly" messages usually does not improve the situation.

Glibc or not to glibc

One of the main "issues" that get raised with Alpine is that it does not use glibc. Alpine Linux is a musl-libc based distribution. In practice I don't have many problems with this since most my software is just packaged by in the distribution so I wouldn't ever see that it's a musl distribution.

Issues appear mostly when trying to run proprietary software on top of Alpine or software that's so hard to build that you're in practice just getting the prebuilds. The solution to proprietary software is... don't use proprietary software :)

For the cases where that's not possible there's always either Flatpak or making a chroot with a glibc distribution in it.

Systemd

Beside not using glibc there's also no systemd in Alpine. This is one of the things I miss the most actually. I don't enjoy the enormous amount of different "best practices" for specifying init scripts and the bad documentation surrounding it. So far by best solution for creating idiomatic init scripts for alpine is just submitting something to the repository and wait until someone complains about style issues.

Beside that I'm pretty happy with the tools openrc provides for manageing services. The rc-update tool gives a nice consise overview of enabled boot services and the service tool just does what I expect. It seems like software is starting to depend on systemd restarting it instead of fixing memory leaks which causes me some issues sometimes.

Conclusion

Alpine Linux is neato. I try to use it everywhere I can.

Rewriting my blog again

Martijn Braam — Mon, 05 Dec 2022 18:17:29 -0000

For quite a while my blog has run on top of Ghost. Ghost is a blog hosting system built on top of nodejs. It provides a nice admin backend for editing your posts and it comes with a default theme that's good enough to not care about theming.

My blog has been through some rewrites, from Drupal to Jekyll to various PHP based CMS platforms and finally it ended up on Ghost. I like Ghost quite a lot but it has a few flaws that made me decide to replace it.

It's nodejs. I know a lot of people like javascript for some reason but it's quite horrific for system administration. I do not like having to add third party repositories to my system to get a recent enough nodejs build to host a simple website. I'm pretty sure this dependency on the latest and greatest is the main reason why everyone wants to stuff their project in docker and not care about the dependencies.
It only officially supports Ubuntu to install on (aside from Docker). It was already annoying to get it installed on Debian but now I've moved over most of my servers to Alpine this has become even more of a pain. There's no reason this has to integrate so deeply with the distribution that it matters what I'm running.
It depends on systemd, this is largely part of the Ubuntu dependency above and not needing to implement something else. But from running Ghost in openrc I've noticed that it depends on the automatic restarts done by systemd when the nodejs process crashes.
I'm just running a simple blog which is fully static content. this should not require running a permanent application server for availability of the blog and the dependency on Mysql is also very much overkill for something that's so read-heavy.
I don't particularly like the templating system for adjusting the theme. I've been running a fork of the casper default theme for a while that just rips out the javascript files. This breaks almost nothing on the default theme except the infinite scroll on the homepage.

Switching to something different

I could try to switch to yet another blogging platform. The issue is that I really really like writing posts in the Ghost admin backend. With a casual look at the browser dev console another solution became obvious...

The whole Ghost admin backend is neatly packaged into four files:

ghost.min.js
vendor.min.js
ghost.min.css
vendor.min.css

Since this is a fully standalone client side webapp thing I can just host those four files and then write a Python Flask application that implements the same REST api that the nodejs backend provided.

This is exactly what I did, the whole backend is now a small Flask application that implements the Ghost API in ~500 lines of Python. It's called Spook (since this is dutch for ghost).

This also uses an SQLite database as storage backend for the whole system. This Flask application also does not implement any of the website rendering. It just stores the state of the Ghost backend and does media uploads. To actually get a website from this it implements a hook script that gets called whenever anything on the website changes. With that it's possible to use any static site generator to generate a static website from this dataset.

Reimplementing casper

As static site generator I used a second Flask webapplication. This one uses the Flask-Frozen module to generate a static website, this is why this module has the name frozen_casper. This generator reads the SQLite database from Spook and generates static HTML compatible with the stylesheet from the stock casper theme from Ghost. It also generates an identical RSS feed so the RSS subscribers can be migrated over with identical post unique ids.

I did make some modifications to the generated html like implementing very basic paging on the posts list pages instead of relying on javascript infinite scrolling.

The spook and frozen_casper module together replaces a complete Ubuntu Server install, a mysql instance I had to keep giving RAM and a crashy nodejs webapplication. Since today this new system is hosting this blog :)

Taking a good picture of a PCB

Martijn Braam — Sun, 27 Nov 2022 10:49:01 -0000

Pictures of boards are everywhere when you work in IT. Lots of computer components come as (partially) bare PCBs and single board computers are also a popular target. Taking a clear picture of a PCB is not trivial though. I suspect a lot of these product pictures are actually 3d renders.

While updating hackerboards I noticed not all boards have great pictures available. I have some of them laying around and I have a camera... how hard could it be?

Definitely the worst picture is the one in the header above. Taken with a phone at an angle with the flash in bad lighting conditions. I've taken quite a bunch of pictures of PINE64 boards, like some of the pictures in the header of the pine64 subreddit and the picture in the sidebar. I've had mixed results with taking the pictures but the best results I've had with taking board pictures is using an external flash unit.

The ideal setup

So to create a great picture I've decided to make a better setup. I've used several components for this. The most important one is two external flashes controlled with a wireless transmitter. I've added softboxes to the flashers to minimize the sharp shadows usually created when using a flash. This produces quite nice board pictures with even lighting.

All dust looks 1000x worse on pictures :(

For all the pictures from this setup I've used a 50mm macro lens. Not only is it great for getting detail pictures of the small components, it's also the lens with the least distortion I have. Having less distortion in the lens is required to have a sharp picture all the way to the edges of the board and not have the edges of the board look curved. The curvature can be fixed in software but the focal plane not being straight to the corners can't be fixed.

It's possible to get even less shadows on the board by using something like a ring light, while this gets slightly more clarity I also find this just makes the pictures less aesthetically pleasing.

So how to deal with the edges of the board? For a lot of website pictures you'd want a white background. I have done this by just using a sheet of paper and cleaning up the background using photo editing software. This is quite time consuming though. The usual issues with this is that the background is white but not perfectly clipped to 100% pure white in the resulting picture. There's also the issue of the board itself casting a slight shadow.

I took my solution for this from my 3D rendering knowledge (which is not much). You can't have a shadow on an emitter. To do this in the real world I used a lightbox.

Lightboxes are normally for tracing pictures and are quite easy to get. It doesn't give me a perfectly white background but it gets rid of the shadows at least.

To get this from good to perfect there's another trick though. If I take a picture without the flashes turned on but everything else on the same settings I get a terribly underexposed picture... except for the background.

You never notice how many holes there are in a PCB until you put it on a lightbox

All I need to do to get a clean background is increasing the contrast of this picture to get a perfect mask. Then in gimp I can just overlay this on the picture with the layer mode set to lighten only.

The final composite

It's also possible to use the mask picture as the alpha channel for the color picture instead. This works great if there's a light background on the website, it shows the flaws though when the website has a dark background.

Let's create the worst-case scenario and use a pure black background:

Now edges are visible on the cutouts. Due to the mismatch in light color temperature with the light box the edges are also blue here. A lot of edges can be fixed by running the dilate filter in gimp on the mask layer to make the mask crop into the board by one pixel. it makes the holes in the board too large though. To get this perfect manual touchup is still required.

Automating it further

Now the input data is perfect enough that I can make the cutout with a few steps in gimp it's also possible to automate this further with the magic of ImageMagic.

$ convert board.jpg \( mask.jpg \
		-colorspace gray \
		-negate \
		-brightness-contrast 0x20 \
	\) \
	-compose copy-opacity \
	-composite board.png

This loads the normal picture from board.jpg and the backlit picture as mask.jpg and composites them together into a .png with transparency.

But it can be automated even further! I still have a bit of camera shake from manually touching the shutter button on the camera and I need to remember to take both pictures every time I slightly nudge the device I'm taking a picture of.

The camera I'm using here is the Panasonic Lumix GX7. One of the features of this camera is the built-in wifi. Using this wifi connection it's possible to use the atrocious Android application to take pictures and change a few settings.

After a bit of reverse engineering I managed to create a Python module for communicating with this camera. Now I can just script these actions:

import time
from remotecamera.lumix import Lumix

# My camera has a static DHCP lease
camera = Lumix("192.168.2.41")
camera.init()

camera.change_setting('flash', 'forcedflashon')
camera.capture()
time.sleep(1)
camera.change_setting('flash', 'forcedflashoff')
camera.capture()

Now I can just run the script and it will take the two pictures I need. It's probably also possible to fetch the images over wifi and automatically trigger the compositing but it sadly requires changing the wifi mode on the camera itself between remote control and file transfer.

Not just for PCBs

This setup is not only useful for PCB pictures. This is pretty great for any product picture where the product fits on the lightbox.

Here's the composite of a SHIFT 6mq. Pictures of the screen itself are still difficult due to the pixel pattern interfering with the pixels of the camera sensor and the display reflecting the light of the flashes. This can probably be partially fixed once I get a polarizing filter that fits this lens.

Finding an SBC

Martijn Braam — Tue, 15 Nov 2022 23:24:56 -0000

A long long time ago on a server far away there was a website called Board-DB. This website made in 2014 was a list of single board computers that became popular after the whole Raspberry Pi thing happened.

Board-DB in 2014

The website itself was pretty simple, just a list of devices and a simple search box with a few filters. The magic was in the well maintained dataset behind it though. After some time this website seemed to have gone away and I had not thought about it for quite a bit until I happened to learn that the owner of https://tuxphones.com/ originally ran the Board-DB website.

Rewriting

So a plan formed together with the old owner of the website. A good board search website is still after all these years a gap in the market. There's some websites that have these features but I keep running into missing filters. One of the things that annoy me the most about webshops and comparison sites is a lack of good filters and then including the most useless ones like a what color the product is.

To redesign this I wanted to go to the other extreme. If you compare two boards and there's not a single different listed spec, then something is missing. The schema for the rewritten database has 165 columns now and I still found some things that are missing in this dataset like detailed PCI-e data.

The 2022 version of the Board-DB website

This new website was written in a timespan of 10 months in some spare time together with Raffaele. At the time of writing it contains data about 466 boards. Not all data is perfect yet since this is a huge research and cleanup job but the website is certainly useful already. As the data in the backend gets more cleaned up more filters in the board list can be added to drill down on the specs you need.

Architecture

The website is a Python Flask webapplication that uses SQLite as storage backend. SQLite is perfect in this case since it's mostly a read-heavy website. The website is designed from the start to run behind a caching reverse proxy like Varnish and makes sure that all the URLs in the site generate the exact same content unless boards are changed.

The site is also designed to work without having javascript enabled but does use javascript to improve the experience. It even works in text-mode browsers. Due to it not using any javascript or css frameworks the website is also very light.

The dataset

At the start of the project I thought implementing the facetted search would be a large technical problem to solve but it that turned out to be relatively easy. By far hardest part of this system is adding and cleaning up the data. The original dataset is imported from a huge spreadsheet that with a bunch of heuristics gets converted into more specific data columns in the SQLite database that runs the website. The original dataset did not separate out specific details like which cores a SoC has and how many and what speed the ethernet ports were.

Parts of this data was recoverable by searching for keywords in the original description fields in the spreadsheet and other data is manually fixed up with some bulk editing tools.

One of the hacks in the CSV importer

This all ends up getting rendered nicely again on the board detail page

I'm quite happy with how the web frontend turned out. It has a decently detailed filter list, an usable spec listing page and a board comparison feature. Getting the project this far would've been impossible without the help of Raffaele.

The website is now online on https://hackerboards.com/

Automated Phone Testing pt.5

Martijn Braam — Tue, 25 Oct 2022 00:55:14 -0000

Now I've written all the parts for the Phone Test Setup in the previous parts, it's time to make the first deployment.

For the postmarketOS deployment I'm running the central controller and mosquitto in a container on the postmarketOS server. This will communicate with a low-power server in a test rack in my office. The controller hardware is an old passively cooled AMD board in a 1U rack case. I sadly don't have any part numbers for the rack case since I got a few of these cases second hand with VIA EPIA boards in them.

Ignore the Arduino on the left, that's for the extra blinkenlights in the case that are not needed for this setup

The specs for my controller machine are:

AMD A4-5000 APU
4GB DDR3-1600 memory
250GB SSD
PicoPSU

The specifications for the controller machine are not super important, the controller software does not do any cpu-heavy or memory-heavy tasks. The important thing is that it has some cache space for unpacking downloaded OS images and it needs to have reliable USB and ethernet. For a small setup a Raspberry Pi would be enough for example.

Ignore the Aperture sticker, this case is from a previous project

This server now sits in my temporary test rack for development. This rack will hold the controller PC and one case of test devices. This rack case can hold 8 phones in total by using 4 rack units.

Deploying the software

After running all the software for the test setup on my laptop for months I now started installing the components on the final hardware. This is also a great moment to fix up all the installation documentation.

I spend about a day dealing with new bugs I found while deploying the software. I found a few hardcoded values that had to be replaced with actual configuration and found a few places where error logging needed to be improved a lot. One thing that also took a bit of time is setting up Mosquitto behind an Nginx reverse proxy.

The MQTT protocol normally runs on plain TCP on port 1883 but since this involves sending login credentials its better to use TLS instead. The Mosquitto daemon can handle TLS itself and with some extra certificates will run on port 8883. This has the downside that Mosquitto needs to have access to the certificates for the domain and it needs to be restarted after certbot does its thing.

Since The TLS for the webapplication is already handled by Nginx running in reverse proxy mode it's easier to just set up Nginx to do reverse proxying for a plain TCP connection. This is the config that I ended up with:

stream {
  upstream mqtt_servers {
    server 10.0.0.107:1883;
  }
  server {
    listen 8883 ssl;
    proxy_pass mqtt_servers;
    proxy_connect_timeout 1s;
    
    ssl_certificate /etc/letsencrypt/.../fullchain.pem; 
    ssl_certificate_key /etc/letsencrypt/.../privkey.pem;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
  }
}

Another thing that had to be done for the deployment is writing actual service files for the components. The init files now sets up openrc in my Alpine installations to supervise the components, deal with logging and make sure the database schema migrations are run on restart.

Putting together the phone case

To neatly store the phones for the test setup I decided to use a 2U rack case since that's just high enough to store modern phones sideways. For this I'm using a generic 2U rackmount project box with the very easy to remember product code G17082UBK. This is a very cheap plastic case with an actual datasheet.

All the internal and external dimensions for this case are documented

I used this documentation to design a tray that fits between the screw posts in this case. The tray is printed in three identical parts and each tray has three slots. I use dovetail mounts to have the phone holders slide on this tray.

All this is designed using OpenSCAD. I never liked 3D modelling software but with this it's more like programming the shapes you want. This appeals a lot to me since... I'm a software developer.

The tray design in OpenSCAD

From this design I can generate an .stl file and send it to the 3D printer. The tray can print without any supports and takes about an hour on my printer. So 3 hours later I have the full base of the phone holder in my rack case.

To actually mount the phones there can be phone-specific models that grip into this baseplate and hold the phone and extra electronics. I made a somewhat generic phone mount that just needs two measurements entered to get the correct device thickness at two points. This is the holder I'm currently using for the PinePhone and the Oneplus 6.

The baseplates here are in blue and the phone holder is the green print. This version of the phone holder is designed to hold the Raspberry Pi Pico and has a ring for managing the soldered cables to the device. The size of the PinePhone is about the largest this case can hold. It will fill up the full depth of the case when the USB-C cable is inserted and it also almost hits the top of the case.

The PinePhone and Oneplus 6 in the test rack case

In this case I can hold 8 phones on the trays and have one of the slots on the tray left over to hold a future USB hub board that will have the 16 required USB ports to use all the devices in the case.

For small desk setups the a single tray is also pretty nice to hold devices and the phone holder itself will also stand on its own. This is great if you have one to three devices you want to hook up to your laptop for local development.

The tray is a bit flimsy without being screwed in the rack case but holds up phones great

Running the first test job

So now the case is together and the controller is deployed it's time to run an actual test. For this first test I'll be using Jumpdrive as the test image. This is by far the smallest image available for the PinePhone which makes testing a lot easier. It just boots a minimal kernel and initramfs and the greatest feature for this test: it spawns a shell on the serial console.

Since GitHub is not a great hosting platform and the bandwidth limit for the https://github.com/dreemurrs-embedded/Jumpdrive repository has been reached it's not possible to fetch the raw file from github without being logged in so this script uses my own mirror of the latest PinePhone jumpdrive image.

devicetype: pine64-pinephone

boot {
  rootfs: http://brixitcdn.net/pine64-pinephone.img.xz
}

shell {
  prompt: / #
  success: Linux
  script {
    ip a
    uname -a
  }
}

This will boot Jumpdrive and then on the root console that's available over the serial port it will run ip a to show the networking info and then uname -a to get the kernel info. Because the success condition is Linux it will mark the job successful if the uname output is printed.

The serial output also shows another feature of the controller: it will set the environment before executing the script, which in this case is just the CI variable and it will add || echo "PTS-FAIL" to the commands so non-zero exit codes of the commands can be detected. When PTS-FAIL is in the serial output the task will be marked as failed. Using the success: and fail: variables in the script the success and failure text can be set.

With these building blocks for test scripts it's now possible to implement a wide variety of test cases. Due to Jumpdrive not having enough networking features enabled in the kernel it's not yet possible to upgrade from the serial connection to a telnet connection in the test setup, this would make the test cases a bit more reliable since there's a very real possibility that a few bits are flipped right in the success or failure string for the testjob marking a successful job as failed due to timeouts.

Getting a generic test initramfs to combine with a kernel to do basic testing will be a good thing to figure out for part 6, Jumpdrive only has busybox utilities available and only has very limited platform support in the first place.

Trying Plasma Desktop again

Martijn Braam — Tue, 18 Oct 2022 23:51:04 -0000

So I'm trying KDE Plasma again, I hear 5.26 has many great improvements and the last time I ran KDE for more than a day was in 2014. I mainly run Gnome and Phosh and Sway on my devices and I feel like I don't use KDE enough for the amount of times I complain about it.

So I decided to put postmarketOS Plasma Desktop on my PineBook Pro. Mainly because one of my issues with KDE has been performance. I know that Gnome runs on the edge of smooth on this hardware so it's easy to compare if KDE will be slower or faster for me there. Testing on faster hardware would only hide performance issues.

Installation

So I installed Plasma with the postmarketOS installer on the Pinebook Pro, I don't have an nvme drive in this device, just running it from the stock 64GB eMMC module.

The only installation issue I had was the disk not resizing on first boot to fill the full eMMMC size, but that's a postmarketOS bug I need to fix, not a KDE issue.

So that continues me writing the rest of this blog post from my Plasma installation :)

First use

My issue with KDE Mostly is that it has many papercuts that I just don't have to deal with in the Gnome ecosystem. I don't want to make this sound like a big hate post on KDE so here's some good things about it first:

I like how the default theme looks in light and dark mode, I'll probably will swap out the cursor theme since it doesn't really fit but the rest looks nice and consistent
The notification system is quite an improvement. Gnome has the annoying tendency to pop up the notification right at the place where I'm working: the top center of the screen. If I get a few chat messages I have to manually click a few stacked notifications away unless I want to wait a few minutes for them to time out. The KDE notifications pop up in the bottom right corner and move out of the way, also the visual timeout indicator is great to have.
I like the set of stock wallpapers in Plasma a lot.
The default application launcher looks way more polished than I remember. Also I'm always happy when application launchers just pop up when you tap the meta key, which is still not a feature in all desktop environments.
That lockscreen fade in animation is nice
The performance has improved quite a lot. I feel like the performance of Plasma on this hardware was absolutely painful a year back when I ran it to get some photos and now it feels quite smooth in mosts places.

Nice notification, ignore inconsistent padding

Not everything is perfect but I'm quite pleased with how this is running. I know a lot of KDE is about everything being customizable, but the thing I like with Gnome is that it's already setup the way I want out of the box. Well mostly... For some reason on every desktop I install I have to re-add the ctrl+alt+t keybind to open a terminal. If I remember correctly this was a thing in older Ubuntu installations and nobody ships this keybind anymore.

Now for the papercuts I hit in the first hour:

The application launcher is slow to pop up. Feels like this should just always be there. But I have the feeling that a lot of animations have a half second delay before they start animating.
The terminal has way too many buttons and toolbars by default, luckily easy to remove again.
meta+up doesn't fullscreen the application, it moves it to the upper half of the screen instead. All the other environments I use fullscreen using this keyboard combination.
partitionmanager doesn't seem to have a function for "extend partition in all the free space after it", instead I have to calculate the new size. Not sure if this counts as an KDE issue or third party applications though :)
My wifi does not reconnect on reboot for me, but I believe this is a postmarketOS bug.
The large "peek at desktop" button in the taskbar is pretty useless and way too big for its function. Windows at least made that only 5px wide.
Performance of Firefox on Plasma seems a bit worse than when running it on Gnome (with wayland)
The screenshot utility is quite clunky compared to what I'm used to in Gnome. I can only select what to screenshot after the fact it seems. It's quite good as far as screenshot UIs go but not perfect yet.
The single-click-to-activate in Dolpin, I'd rather double click on items
Also the active control blue border around the file list in Dolphin. It does not really need to be there and it makes the interface so much more messy. It shows there's focus on the filebrowser widget, but what does that mean? You either have focus on a file in it and that should have the highlight or you don't and then the widget should not have focus.

The Dolpin File manager with the blue border around the files

After some more use

One of the things I had not used yet is Discover, I'm used to installing packages using the command line. I barely use Gnome Software, when I do it's mostly because the flatpak commandline utility is a mess to use. I tried some packagemanagement using Discover and ran into a few pros and cons. This is not only on the KDE side ofcourse because this heavily integrates with postmarketOS specific things.

The main thing I noticed is that Discover is faster than Gnome Software, it's not even close. The speed at which Discover search works makes it actually a realistic option compared to launching a terminal. There's a few paper cuts too ofcourse. If the internet is down and you launch Discover it will show a notification for every repository configured in apk, which is 3 notifications on stock postmarketOS edge.

This notification is almost completely meaningless, It says transferring not what it's transferring and shows no actual progress.

If the internet is up it works pretty smoothly though. The only thing I miss in the rest of the UI is the "Launch" button that's present in Gnome Software to start the software you just installed instead of going through the app launcher.

Customization

Since Plasma is supposedly very customizable, I've been going through the settings to make it work closer to the way I want.

First thing is fixing up some hardware configuration. The touchpad on the Pinebook Pro does not have separate buttons so tap-to-click needs to be enabled, and more importantly, two-finger-tap to right click. The touchpad configuration is nicely set-up in the settings utility in Plasma, this was an easy change to do.

Changing the window management behavior for the meta+up shortcut was also pretty simple, I only was slightly confused by the shortcuts settings a bit because unchecking the checkbox below "Default shortcut" makes it not respond to my custom shortcut. But leaving the default enabled is fine sincec I would never hit meta+PgUp on a laptop keyboard by accident.

Changing up the cursor theme has been proven to be the most annoying. The options for this by default is Breeze and Breeze-light. Which is not what I want, and not what I want in white. This means it's down to the button I always dislike in these environments, the user content browser.

The one in this case is no exception. User content is a huge mess. Names mean nothing, screenshots are mostly useless, it's hard to find some "normal" things beside all the really weird options. Usually the service that hosts the data is also pretty unreliable. After 3 search queries I started to get this error window that immediately reappears when I close it with the button while also showing it has sucessfully fetched the results of the query behind it anyway.

After installing some cursor themes I found that the rendering of anything that's not Breeze is just bad. The preview in the settings window look fine but when I actually apply my configuration the cursor is way smaller. It also only applies when hovering over the window decoration, I still get the old cursor theme on the window itself any any newly launched windows. Deleting the currently active cursor theme also crashes the settings application.

I gave up on this when I remembered that icon themes are standardized. So I installed the adwaita-icon-theme package from Alpine Linux and then I got the option for the adwaita cursor theme. After a reboot the new cursor was consistently applied.

Other customisation things I wanted to try is the general system theming. There's quite a bit of different theming settings available. The first one I messed with is the "Application style"

The preinstalled application styles

By default the theme is set to Breeze, I switched it to Fusion because it has actual old school 3D controls like GTK used to have until they decided that custom options are forbidden on libadwaita-enabled applications. Maybe I like this theme because it's closest to old-school clearlooks :)

Changing this theme also fixes the blue border focus weirdness in Dolphin.

For the GTK apps I use there's a seperate settings page.

Gtk theme settings

For GTK there's no preview thumbnails and the settings are very incomplete. With the default Breeze settings the GTK apps integrate more into the general theming of Plasma, with Default it will let the apps use Adwaita. With the Emacs theme... it also just uses Adwaita. ~~The option I'm missing here though is the dark variant of Adwaita~~. After installing the gnome-themes-extra package I have the Adwaita Dark theme available that I wanted.

I think making GTK look like Breeze is nice from a technical standpoint but by doing that the app will not look like a proper Plasma application, it will just look like it has the same colors and widgets but it will still follow the Gnome HIG, not the Plasma one. Also changing the widget theme to Fusion like I did will not change that for the Breeze GTK theme ofcourse so that introduces more inconsistency. I'd rather have the GTK apps look the way it was intended instead of half-following the Plasma styles.

Finding more workarounds

After some use I have found workarounds for some of my bigger issues. The wifi connection issue can be solved by setting the connection profile to "All users" makes auto-connecting on startup work again.

The annoying widgets I did not want in various places could be removed with the customisation features of Plasma, so not really a workaround but really intended functionality :D

The only large issue I have left is the performance of the application launcher, I'm pretty sure that won't be an easy fix.

Conclusion

I've been running Plasma for a bit now on the Pinebook Pro and I think I will leave it on. It fulfills it's purpose of being the frame around my terminal windows and the browser and hits most of the performance goals I have for this hardware.

Green accent color! without hacking up the theme!