BrixIT BlogRandom blog posts from Martijn Braamhttps://blog.brixit.nl/https://blog.brixit.nl/image/w300//static/files/blog.brixit.nl/1669734482/favicon.pngBrixIT Bloghttps://blog.brixit.nl/Thu, 25 Jan 2024 22:45:44 -000060Fixing the Megapixels sensor linearizationhttps://blog.brixit.nl/fixing-the-megapixels-sensor-linearization/95MegapixelsMartijn BraamThu, 25 Jan 2024 22:45:44 -0000<p>Making a piece of software that dumps camera frames from V4L2 into a file is not very difficult to do, that's only a few hundred lines for C code. Figuring out why the pictures look cheap is a way harder challenge.</p> <p>For a long time Megapixels had some simple calibrations for blacklevel (to make the shadows a bit darker) and whitelevel (to make the light parts not grey) and later after a bit of documentation studying I had added calibration matrices all the way back in <a href="https://blog.brixit.nl/pinephone-camera-pt4/">part 4</a> of the Megapixels blog series.</p> <p>The color matrix that was added in Megapixels is a simple 3x3 matrix that converts the color sensitivity of the sensor in the PinePhone to calibrated values for the rest of the pipeline. Just a simple 3x3 matrix is not enough to do a more detailed correction though. Luckily the calibration software I used produces calibration files that contain several correction curves for the camera. For example the HSV curve that changes the hue, saturation and brightness of specific colors.</p> <p>Even though this calibration data is added by Megapixels I still had issues with color casts. Occasionally someone mentions to be how "filmic" or "vintage" the PinePhone pictures look. This is the opposite of what I'm trying to do with the picture processing. The vintage look is because color casts that are not linear to brightness are very similar on how cheap or expired analog film rolls reproduce colors. So where is this issue coming from?</p> <p>I've taken a closer look to the .dcp files produced by the calibration software. With a bit of python code I extracted the linearization curve from this file and plotted it. It turns out that the curve generated after calibration was perfectly linear. It makes a bit of sense since this calibration software was never made to create profiles for completely raw sensor data. It was made to create small corrections for professional cameras that already produce nice looking pictures. Looks like I have to produce this curve myself</p> <h2>Getting a sensor linearization curve</h2> <p>As my first target I looked into the Librem 5. Mainly because that's the phone that currently has the most battery charge. I had hoped there was some documentation about the sensor response curves in the datasheet for the sensor. It turns out that even getting a datasheet for this sensor is problematic. So the solution is to measure the sensor instead.</p> <p>Measuring this pretty hard though, the most important part is having calibrated reference for most solutions. I've thought about figuring out how to calibrate a light to produce precise brightness dimming steps and measuring the curve of the light with a colorimeter to fix any color casts of the lights. Another idea was taking pictures of a printed grayscale curve but that has the issue that the light on the grayscale curve needs to be perfectly flat.</p> <p>But after thinking about this in the background for some weeks I had a thought: instead of producing a perfect reference grayscale gradient it's way easier to point the camera at a constant light source and then adjust the shutter speed of the camera to produce the various light levels. Instead of a lot of external factors with calibrated lights which can throw off measurements massively I assume that the shutter speed setting in the sensor is accurate.</p> <p>The reason I can assume this is accurate is because the shutter speed setting in these phone sensors is in "lines". These cameras don't have shutters, it's all electronic shutter in the sensor. This means that if the shutter is set to 2 lines it means that the line being read by the sensor at that moment is cleared only 2 scanlines before. This is the "rolling shutter" effect. If the shutter is set to 4 lines instead every line has exactly twice the amount of time to collect light after resetting. This should result in a pretty much perfectly linear way to control the amount of light to calibrate the response with.</p> <p>In the case of the Librem 5 this value can be set from 2 lines to 3118 lines where the maximum value means that all the lines of the sensor have been reset by the time the first line is read out giving the maximum amount of light gathering time.</p> <p>With libmegapixels I have enough control over the camera to make a small C application that runs this calibration. It goes through these steps:</p> <ol><li>Open the specified sensor and set the shutter to the maximum value</li> <li>Start measuring the brightness of the 3 color channels and adjust the sensor gain so that with the current lighting the sensor will be close to clipping. If on the lowest gain setting the light source is still too bright the tool will ask to lower the lamp brightness.</li> <li>Once the target maximum brightness has been hit the tool will start lowering the shutter speed in regular steps and then saving the brightness for the color channels at that point.</li> <li>The calibration data is then written to a csv file</li> </ol> <p>The process looks something like this:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1706114157/image.png" class="kg-image"></figure> <p>This is a short run for testing where only 30 equally spaced points are measured. I did a longer run for calibration with it set to 500 points instead which takes about 8 minutes. This is a plot of the resulting data after scaling the curves to hit 1.0 at the max gain:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1706114296/image.png" class="kg-image"></figure> <p>The response of the sensor is not very linear at all... This means that if a picture is whitebalanced on the midtones the shadows will have a teal color cast due to the red channel having lower values. If the picture would be corrected with whitebalance to correct the darker colors it would result in the brighter colors to turn magenta.</p> <p>The nice thing is that I don't have to deal with actually correcting this. This curve can just be loaded into the .dng file metadata and the processing software will apply this correction at the right step in the pipeline.</p> <h2>Oops</h2> <p>It is at this point that I figured out that the LinearizationTable DNG tag is a grayscale correction table so it can't fix the color cast. At least it will improve the brightness inconsistencies between the various cameras.</p> <p>With some scripting I've converted the measured response curve into a correction curve for the LinearizationTable and then wrote that table into some of my test pictures with <code>exiftool</code>. </p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1706221766/compare-linearizationtable.jpg" class="kg-image"></figure> <p>This is the result. The left image is a raw sensor dump from the Librem 5 rear camera that does not have any corrections at all applied except the initial whitebalance pass. On the left is the exact image pipeline but with the LinearizationTable tag set in the dng before feeding it to <code>dcraw</code>.</p> <p>The annoying thing here is that both pictures don't look correct. The first one has the extreme gamma curve that is applied by the sensor so everything looks very bright. The processed picture is a bit on the dark side but that might be because the auto-exposure was run on the first picture causing underexposure on the corrected data.</p> <p>The issue with that though is that some parts of the image data are already clipping while they shouldn't be and exposing the picture brighter would only make that worse.</p> <p>Maybe I have something very wrong here but at this point I'm also just guessing how this stuff is supposed to work. Documentation for this doesn't really exist. This is all the official documentation:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1706222220/image.png" class="kg-image"><figcaption>No, chapter 5 not helpful</figcaption></figure> <p>Maybe it all works slightly better if the input raw data is not 8-bits but that's a bunch more of kernel issues to fix on the Librem 5 side.</p> <h2>Conclusion</h2> <p>So not that much progress on this at all as I hoped. I made some nice tools to produce data that makes pictures worse. Once the clipping in the highlights is fixed this might be very useful though since practically everything in the DNG pipeline expects the input raw data to be linear and it just isn't.</p> <p>The <a href="https://gitlab.com/megapixels-org/libmegapixels/-/commit/f6686d7a5a176384da3b5a1eaf93985aeb29d7be">sensor measuring tool</a> is included in the libmegapixels codebase now though.</p> <p>To fix auto-exposure I also need to figure out a way to apply this correction curve before running the AE algorithms on the live view. More engineering challenges as always :)</p> <hr> <div style="width: 75%; margin: 0 auto; background: rgba(128,128,128,0.2); padding: 10px;"> <h4>Development Funding</h4> <p>The current developments of Megapixels are funded by... You! The end-users. It takes a lot of time and a lot of weird expertice to make Linux cameras work and I've not been able to do it without your support.</p> <p>The donations are being used for the occasional hardware required for Megapixels development (Like a nice Standard Illuminant A for calibration) and the various other FOSS applications I develop for the Linux ecosystem. Every single bit helps to not do all this work entirely for free.</p> <a href="https://blog.brixit.nl/donations/">Donations</a> </div> <p></p> The dilemma of tagging library releaseshttps://blog.brixit.nl/the-dilemma-of-tagging-library-releases/94MegapixelsMartijn BraamSun, 14 Jan 2024 16:11:17 -0000<p>I've been working on the libmegapixels library for quite a bit now. The base of the library is pretty solid which is configuring a V4L2 pipeline so you can get camera frames on modern ARM platforms. Most of the work on the library side is figuring the AWB/AE/AF code and how that will fit together with applications.</p> <p>Due to the AAA code not working yet and the API not being being fully defined on how those parts will fit together I've been holding of on tagging an actual release on the libmegapixels library.</p> <p>A lot of my projects, especially libraries, are written in Python so I've long enjoyed the luxury of APIs being duck-typed and having the possibility of adding optional arguments to methods in the future. Sadly in C libraries I can't get away with never defining the types for arguments that might change in the future or adding optional arguments.</p> <p>My original plan was to tag a release on libmegapixels together with the first 2.x release of Megapixels since these pieces of software are intended to fit together but after thinking about it some more (and some convincing from other people interested in the libmegapixels release) I've decided to tag a 0.1 release.</p> <p>In an ideal world I can just release code when it's fully done and tested. In this case the long time it takes to get everything ready for use will mean that potential contributors to the code will also be held back from experimenting with the codebase. Especially since a large part of libmegapixels is the config files it ships for specific hardware configurations. If I wouldn't make any releases then at some point users/developers will be forced to just ship random git commits which is a way worse situation to be in for bug tracking.</p> <p>With this 0.1 release I want to make it possible to start writing config files for various phones and platforms to test camera pipelines. Hopefully this will also mean any issues with the configuration file format that people might hit will be figured out before I have to tag a "final" 1.x release.</p> <h2>The release</h2> <p>So the initial tagged release of <code>libmegapixels</code>:</p> <ul><li>located at <a href="https://gitlab.com/megapixels-org/libmegapixels/-/tags/0.1.0">https://gitlab.com/megapixels-org/libmegapixels/-/tags/0.1.0</a></li> <li>Build instructions at <a href="https://libme.gapixels.me/building.html">https://libme.gapixels.me/building.html</a></li> <li>Comes with absolutely no guarantee of stability for the C api of the library</li> <li>Most likely the config file format is stable but might have small tweaks before the 1.x release</li> </ul> <p>Hopefully this will allow people to start experimenting with the codebase and generate some feedback on it so I'm not just developing this for months and completely overfitting it to the three devices I'm testing on.</p> <p>I'm planning to make a similar release for <code>libdng</code> soon. That library is also mostly stable but I need to fix up the last parts of the API to allow reading and writing all the required metadata.</p> Megapixels 2.0: DNG loading and Autowhitebalancehttps://blog.brixit.nl/megapixels-2-0-dng-loading-and-whitebalancing/93MegapixelsMartijn BraamFri, 22 Dec 2023 01:25:46 -0000<p>After getting some nice DNG exporting code to work with libdng in the <a href="https://blog.brixit.nl/megapixels-2-0-dng-exporting/">last post</a> I decided to go mess with auto white-balancing again on the Librem 5.</p> <p>I got the Megapixels 2.x codebase to the point where it smoothly displays the camera feed on the Librem 5 and the PinePhone Pro. One of the things that Just Worked(tm) on the original PinePhone is the auto white-balance correction of the rear camera. This has also not worked on the front camera on that device and the results of lacking AWB code is very obvious: the pictures are very green.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703201479/gc2145.jpg" class="kg-image"><figcaption>Great example of lack of AWB on the PinePhone front camera</figcaption></figure> <p>This was very easy on the rear camera of the PinePhone. The camera module inside the phone can automatically do white-balance corrections by having an AWB algorithm running on the 8051 core inside the sensor which adjusts the analog gain of the camera. The only thing that Megapixels does on the PinePhone is turning that feature on and it just works. The front camera of the PinePhone should have a similar feature but it does not work due to the state of the Linux driver for that sensor.</p> <p>For the PinePhone Pro and the Librem 5 (and most other devices) the white-balancing is a lot harder to deal with. The sensor does not have any automatic way of dealing with this and it has to be done on the CPU side. For this there's also two options:</p> <ul><li>Get the unbalanced camera feed and correct those frames in software while displaying them and storing the correction factors in the DNG files for the pictures that have been taken.</li> <li>Send the corrections back to the sensor instead so the camera feed is already balanced. This should lead to higher quality pictures because the use of the analog to digital converter in the sensor is more optimal. It&#x27;s also harder because now there&#x27;s latency between changing the gain and receiving the corrected data.</li> </ul> <p>But the nice thing of doing hardware support on multiple platforms is that I have to support both these cases :(</p> <p>In case of the Librem 5 I'm implementing the first option since the sensor driver for the rear camera on that device does not implement the necessary controls to do the second option, it's also a bit easier to get working right.</p> <h2>The Algorithm™</h2> <p>There are many ways to actually implement a white-balance algorithm. I'll be going with the most simple one. The gray-world algorithm.</p> <p>This algorithm works on the assumption that if you average out all the colors in your picture you'd get something that's roughly grey.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703202886/grey-world.jpg" class="kg-image"></figure> <p>Intuitively you would think that nothing is ever that nicely balanced or that colorful walls for example would skew the results massively. But as you can see in the demonstration above the more you start to blur the picture the less saturated it will become. This works for a surprising amount of pictures.</p> <p>To calculate the white-balance correction for a picture it's not blurring the picture though as in the demonstration above. The way it's calculated is by taking the average of all the pixels in the picture and then using the inverse of that result to set the gain.</p> <p>Another thing that's different for white-balancing is that it's the first step in the color processing pipeline, it should not be ran on the final picture like this example but on the raw color values from the sensor since there it would be the closest in the pipeline to the ADC gain applied in the sensor.</p> <h2>The Megapixels implementation</h2> <p>taking the average of a full raw frame is not very fast. It also complicates the internal Megapixels code a lot to do extra image processing on the raw frame stage of the code. That's why in Megapixels I'm <i>not</i> running the white-balance code on the raw frames. It's instead being run on the preview feed shown to the user before taking the picture since that way it takes advantage of the GPU debayering and scaling of the input data.</p> <p>To make this work the libmegapixels code does the average of an entire processed RAW frame to get my average R, G and B values. After running this average the code is no longer dealing with a lot of data anymore so it's a lot easier to write quick code. To fix the code the inverse of the color matrix is ran to get a value that's close to what the average would've been of the raw data before scaling it and doing the preview color corrections.</p> <p>The result of that code gives a new R, G and B value that represent the balance of the color of the picture, the new gain for the color channels is then calculated as <code>1/R</code> and then normalized so the gain for the green channel is always 1.0. This is because on the sensors there's only a control for the red and blue gains.</p> <p>Except that on the Librem 5 there's no controls for the red and blue gains at all, so in that case the new gains are fed into the GPU shader that calculates the preview again where it will be applied as gains right after the debayering step.</p> <h2>The white-balance in the DNG output</h2> <p>With the two scenarios above there's also two cases for the DNG exporting. Either the RAW data in the DNG is already balanced or it's completely unbalanced. Luckily the DNG specification has me covered!</p> <p>When the raw image data is completely unbalanced like it is on most professional cameras the gains for balancing the picture are stored in the <code>AsShotNeutral</code> tag. This tells the DNG developing software the gains the camera used to display the preview and it will be available in the white-balance section of the developing software as "As Shot" or "Camera" white-balance.</p> <p>In the case where the ADC gains are manipulated to apply the white-balance this doesn't work since the gains written to the <code>AsShotNeutral</code> tag would be 1.0 for all channels. This <i>does</i> produce the correct picture for simple cases except that the whitebalance shown for the image in the editing software would always be 5612K.</p> <p>Having the wrong whitebalance is not just an issue of metadata neatness though. Practically all the color pipeline calculations after loading the DNG file and applying the RAW white-balance are dependent on the color temperature. The metadata in the DNG stores two color matrices and two correction LUTs. The guidelines for this calibration data is that one of the sets of calibration data is for D65 lighting which is basically outdoors on a cloudy day; pretty blue-ish lighting around 6500k. The second one is for "Standard Illuminant A" which is a reference tungsten light around 2856k. The developing software takes the data for both color temperatures and interpolates between the two to produce the matrices and curves for the color temperature of the white-balanced picture.</p> <p>To deal with the case where the the sensor already produced white-balanced raw data using the ADC gains the white-balance gains can be written to the <code>AnalogBalance</code> tag. This will be used to invert the white-balance gains in the sensor again before running the rest of the processing pipeline which means the correct color temperature will be used.</p> <h2>So does it work?</h2> <p>Yeah, mostly. It could use a bit of tweaking and the calibration I'm using for the sensor is just wrong.</p> <video src="https://brixitcdn.net/white-balance-encoded.mp4" controls style="max-width: 50%"></video> <p>The video here is extremely janky, most likely due to the auto-gain in this test build being completely broken and my code being sloppy. There's a few things that need to be fixed here aside from figuring out more preformance regressions:</p> <ul><li>The whitebalance code stops working when there&#x27;s not enough light and it jumps to the full green picture. At this point it should keep holding the old white-balance to make it less jarring.</li> <li>There needs to be smoothing applied to the whitebalance changes. It&#x27;s mostly pretty solid since this doesn&#x27;t have any latency with sensor adjustments but when the camera moves to the pumpkins you can see it being unstable.</li> </ul> <p>Overall it mostly works though. The performance is a bit more stable when there's daylight. Cameras simply work better when there's more light available. The various sources of artificial light here is also throwing off the camera a lot with a lot of light coming from my monitor and some very poor quality light coming from the room lighting.</p> <h2>The SEGFAULT button</h2> <p>So Megapixels somewhat balances the pictures but the second half of the process is not something I've been able to test yet: storing DNG files with this whitebalance metadata. The Megapixels 1.x codebase had the code for saving the <code>AsShotNeutral</code> and <code>AnalogBalance</code> tags and I've re-implemented that in libdng. The issue is that in the current state of the Megapixels code pressing the shutter button just causes the whole application to segfault.</p> <p>This segfault occurs somewhere in the interaction with the color profile curves loaded from the calibration .dcp file with the libtiff library when saving though libdng. This being 3 threads deep into the Megapixels codebase makes this a bit annoying to debug so I decided I needed to yak-shave this a bit further and add more tooling to the libdng codebase...</p> <h2>The mergedng tool</h2> <p>My solution for making this easier to debug is adding an utility in libdng that actually uses the feature to load a calibration file to append the curves to the final picture. Due to me just not stopping to write code I've implemented basic DNG reading support for this in libdng and as frontend the <code>mergedng</code> utility.</p> <p>The functionality for this tool is pretty simple. It reads an input DNG file and takes the picture data and metadata from that. It then takes a .dcp file as second argument which provides the calibration curves for the camera and it then merges those TIFF tags and writes out a new DNG file. This is an utility I needed anyway since I've been searching for it, it makes it easy to "upgrade" pictures taken with earlier versions of Megapixels with new calibration data from more recent .dcp files.</p> <p>Writing the code for this functionality was pretty straightforward. The .dcp loading and appending code already existed in the libdng codebase since that's the code which already causes the SEGFAULT in Megapixels when taking a picture. The extra added code in libdng is the new functions for reading a DNG file and taking that image metadata for writing a new picture.</p> <p>After implementing all this and adding some unit tests for the DCP loading code I've come to the realization that... it just works...</p> <p>In this simplified codebase everything touching the data just simply works so my original crashing issue in Megapixels is somewhere unrelated. This is where I'm at now and where I've decided to write a blog post instead of diving deep into the Megapixels codebase again :)</p> <hr> <div style="width: 75%; margin: 0 auto; background: rgba(128,128,128,0.2); padding: 10px;"> <h4>Development Funding</h4> <p>The current developments of Megapixels are funded by... You! The end-users. It takes a lot of time and a lot of weird expertice to make Linux cameras work and I've not been able to do it without your support.</p> <p>The donations are being used for the occasional hardware required for Megapixels development (Like a nice Standard Illuminant A for calibration) and the various other FOSS applications I develop for the Linux ecosystem. Every single bit helps to not do all this work entirely for free.</p> <a href="https://blog.brixit.nl/donations/">Donations</a> </div> <p></p> The MNT keyboard reviewedhttps://blog.brixit.nl/the-mnt-keyboard-reviewed/92LinuxMartijn BraamTue, 19 Dec 2023 23:59:01 -0000<p>MNT Research is one of those few companies that actually releases open source hardware. Instead of just getting a <a href="https://mntre.com/documentation/reform-keyboard-v3-manual.pdf">schematic</a> with your hardware (which is great even by itself) there's the full <a href="https://source.mnt.re/reform/reform/-/tree/master/reform2-keyboard3-pcb">sources for that schematic</a>, the Kicad parts libraries, the sources for the firmware and even documentation how to use that code.</p> <p>I received my <a href="https://shop.mntre.com/products/mnt-reform-keyboard-30">MNT Standalone Keyboard V3</a> a few days ago so I've been typing on it now for a bit. This is all happening while I'm recovering from covid so I hope if I read back this post in a few days it is actually somewhat coherent :)</p> <p>This being a more niche product sadly does make it a bit on the expensive side. But I must say this is by far the most solid keyboard I've owned. My main keyboard on my desktop is an Das Keyboard 4 ultimate. It's a nice keyboard but it doesn't compare to the full machined aluminium frame on the MNT keyboard.</p> <p>The whole keyboard is mounted on what's basically a 4mm slab of aluminium which has a nice MNT logo machined on it on the bottom</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703022648/20231219_0011.jpg" class="kg-image"></figure> <p>This makes the keyboard feel incredibly solid, even with the rest of the frame taken off it's practically impossible to even bend the keyboard. The second half of the frame is the top edge that screws on the base plate with 8 screws.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703022719/20231219_0012.jpg" class="kg-image"></figure> <p>This is another very carefully designed aluminium part In the close-up above you can see the opening for the USB-C connection for the keyboard and the internal cutouts for the display daughterboard with the screw mounting.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703027123/20231219_0021.jpg" class="kg-image"></figure> <h2>The electronics</h2> <p>This keyboard is based around an Atmega32U4 microcontroller. This is the same keyboard PCB as what's shipped in the MNT Reform laptop so there are two connectors on this board. The USB-C connector is what's exposed on the standalone keyboard and the laptop presumably uses the USB header that's beside it.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703023510/20231219_0015.jpg" class="kg-image"></figure> <p>Beside the USB header is one of the dip switches. SW36 is labeled "STANDALONE" here. This switches the board to use USB power instead of the 3.3V supplied by the laptop mainboard. The ribbon connector is the connection to the OLED display board.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703026316/20231219_0016.jpg" class="kg-image"></figure> <p>On the left side of the display board there is an empty footprint for the standard Atmega programming header and a serial port that's used to connect to the laptop mainboard. Additionally there's a reset button and SW84 which has the confusing label "RG".</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703026421/image.png" class="kg-image"></figure> <p>Thanks to the schematics being available in the manual it's easy to find that this is the switch to enable programming. The rest of the interesting parts is hidden somewhere below the display board or on the bottom side of the PCB possibly. I have not taken the keyboard further apart for this review since all the information I'd ever want is already available in the schematics. The keyboard matrix itself is read out by the Atmega directly which provides the full keyboard functionality and the OLED display is on a small daughterboard to slightly rise it towards the front bezel.</p> <h2>Firmware</h2> <p>Since this is one of the 8-bit Atmel parts it's very easy to build firmware using the gcc-avr compiler packaged in various distributions. All the source files are stored in the <a href="https://source.mnt.re/reform/reform/-/tree/master/reform2-keyboard-fw">firmware repository</a> for the various MNT products.</p> <p>Checking the version of the firmware is pretty easy. With the circle key on the top-right corner of the keyboard the menu on the display opens. You can use the arrow keys to browse to the "System Status" option or just press the "s" key on the keyboard.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703027504/20231220_0008.jpg" class="kg-image"></figure> <p>Which shows the hardware revision this firmware was build for and the version that was specified when building:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703027858/20231220_0018.jpg" class="kg-image"></figure> <p>It seems like the "g" at the start of the commit has was accidental here and it refers to commit <code>7e73483</code> in the firmware repository. This seems to be the newest tag when the keyboard was shipped so that makes sense.</p> <p>So lets change something! The key in the bottom left corner of the keyboard is the Hyper key instead of Ctrl as you'd expect from most keyboards. The Ctrl key is moved in place of the Caps Lock button on normal keyboard layouts which is great for a lot of uses. I never use Hyper though so I want to change that key to be my second Ctrl key.</p> <p>The readme specifies that the keyboard layout is defined in the various <code>matrix_*</code> files so after reading around a bit it seems like I have to edit <code>matrix_3.h</code> for my keyboard.</p> <p>Reading the manual again I realized that doing this makes me lose access to the media keys since those are defined as "Hyper+F*" for the various media actions. To fix that I changed the right control button into the Hyper key, this is the button with the three dots on it. My resulting code change:</p> <pre><code>diff --git a/reform2-keyboard-fw/matrix_3.h b/reform2-keyboard-fw/matrix_3.h index bb72f6d..f9db133 100644 --- a/reform2-keyboard-fw/matrix_3.h +++ b/reform2-keyboard-fw/matrix_3.h @@ -25,7 +25,7 @@ // Sixth row #define MATRIX3_DEFAULT_ROW_6 \ - HID_KEYBOARD_SC_EXECUTE,\ + HID_KEYBOARD_SC_LEFT_CONTROL,\ HID_KEYBOARD_SC_LEFT_GUI,\ HID_KEYBOARD_SC_LEFT_ALT,\ KEY_SPACE,\ @@ -33,7 +33,7 @@ KEY_SPACE,\ KEY_SPACE,\ HID_KEYBOARD_SC_RIGHT_ALT,\ - HID_KEYBOARD_SC_RIGHT_CONTROL,\ + HID_KEYBOARD_SC_EXECUTE,\ HID_KEYBOARD_SC_LEFT_ARROW,\ HID_KEYBOARD_SC_DOWN_ARROW,\ HID_KEYBOARD_SC_RIGHT_ARROW </code></pre> <p>Now to build this there's a simple Makefile. Since I've already programmed Atmega parts on this machine I already have the compiler installed making this very quick and easy.</p> <p>I ended up compiling with the following command:</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>make <span class="nv">REFORM_KBD_OPTIONS</span><span class="o">=</span><span class="s2">&quot;-DKBD_VARIANT_3 -DKBD_MODE_STANDALONE -DKBD_FW_VERSION=\\\&quot;Martijn\\\&quot;&quot;</span> </pre></div> <p>This is straight from the readme with an additional define to set the firmware version to "Martijn". After building this I got the <code>keyboard.hex</code> file that can be flashed.</p> <p>The flashing is as simple as running the <code>flash.sh</code> script. This will instruct you to press "Circle + X" to enter flashing mode and then run the neccesary commands to flash the keyboard. After running this I noticed that the delete key on the keyboard was no longer a delete key. It turns out I don't have <code>VARIANT_3</code> but instead <code>VARIANT_3_US</code>. A quick rebuild and reflash also fixes that.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1703029671/20231220_0021.jpg" class="kg-image"><figcaption>The brightness differences on the display are a camera artifact</figcaption></figure> <p>Tadaa! My own name in the firmware version field. It's super easy to mess with this firmware.</p> <h2>The keyboard itself</h2> <p>Well the keyboard works just fine as a keyboard. Typing on this keyboard takes a few minutes to get used to compared to my normal keyboard since all the keys are slightly closer together. The split spacebar is also annoying me a bit. It turns out that the left split in the spacebar is <i>exactly</i> the spot where I normally hit the spacebar with my thumb.</p> <p>The switches are nice and clicky (but silent, I have the version with brown switches in them). Overall the keyboard just does what it needs to. The standard layout is quite unusual but everything can be changed with open firmware so I'm confident I can get to a layout I'm 100% happy with.</p> <h2>Conclusion</h2> <p>This is an extremely solid and very compact keyboard I can easily throw in my backpack. It being an USB-C keyboard makes it fit neatly with all my other random cables I usually take with me.</p> <p>It might be slightly more expensive than similar keyboards, but I don't know of similar keyboards with a case this rugged and the display functionality (I forgot to mention you can use HID reports from the host to write custom content to the display from your computer). The openness of this product makes the extra cost certainly worth it for me.</p> <p>I'll probably be messing with the firmware for this keyboard a bit more while I use it. There's some small things to fix like the device reporting the name "LUFA Keyboard Demo Application" in Linux instead of a neater "MNT Keyboard" or something.</p> Looking closer at the sysloghttps://blog.brixit.nl/looking-closer-at-the-syslog/91LinuxMartijn BraamMon, 11 Dec 2023 15:43:17 -0000<p>The syslog protocol, it's one of the ancient protocols in the Unix world. For a long time the logging was handled by daemons like syslog-ng and rsyslog, this has now been taken over by journald on a lot of systems. But have you ever wondered how your log messages even end up in <code>/var/log</code> in the first place?</p> <p>I've started looking into syslog implementations when building a replacement for the use of busybox syslogd in postmarketOS. In postmarketOS this daemon is configured to just send syslog messages to a in-memory buffer for logging and never store anything on disk in <code>/var/log</code>. This is mainly to make sure there's no unneeded writes to the flash storage in a lot of the old phones that are supported by postmarketOS. There's a few downsides to this logging implementation though:</p> <ul><li>No persistent logging of system messages across reboots. This would be easy to check if certain log messages were present on earlier boots when debugging.</li> <li>Completely unstructured logs while people are pretty much used to journald logging with nice filters</li> </ul> <p>As a replacement I wrote <code>logbookd</code>. It's a tiny syslog daemon that supports disk and memory logging and provides some nice filtering options to be closer to journald. The bulk of this work is handled by doing structured logging into SQLite.</p> <h2>So how does the syslog work</h2> <p>The way the syslog works is incredibly simple. The syslog daemon opens an unix domain socket at <code>/dev/log</code>. Applications connect to this socket and write log messages in the syslog format and the syslog daemon takes care of filtering those out and putting it in the various files in /var/log.</p> <p>The complication of this is that there is no real syslog protocol. There are two standards for it though. There is <a href="https://datatracker.ietf.org/doc/html/rfc3164">RFC 3164</a> and <a href="https://datatracker.ietf.org/doc/html/rfc5424">RFC 5424</a> which both describe the syslog protocol. The 3164 document was only created in 2001 and describes what various implementations are doing in the wild. It's RFC 5424 that actually nails down a specific format.</p> <p>I wrote parser for the 5424 format initially since that's the newest standard and it's by far the easiest to parse. An RFC 5424 log message looks like this:</p> <pre><code>&lt;13&gt;1 2023-12-11T14:56:59.0189+01:00 laptop test - - [timeQuality tzKnown=&quot;1&quot;] Hi</code></pre> <p>The first part here between the angular brackets is the PRI value. It encodes the logging facility and severity as one number. The least significant 3 bits encode the severity on a scale of 1-8 and the other bits encode one of the 23 facilities that are defined. Some examples of the facilities are:</p> <ul><li><code>0</code> for kernel messages</li> <li><code>1</code> for generic userspace messages</li> <li><code>2</code> for the mail system</li> </ul> <p>Most of the other numbers are for more old services like UUCP and FTP and for some numbered user-defined codes. In the example above the 13 means facility 1 (user) and severity 5 (notice).</p> <p>The other parts of this message are in order:</p> <ul><li>The protocol version number which is set to <code>1</code> here.</li> <li>A timestamp with timezone for the log message</li> <li>The <code>laptop</code> is the hostname for the message, this will be set to <code>-</code> when NULL</li> <li>The <code>test</code> part is the application name. This can also be <code>-</code> for NULL</li> <li>The next field is called PROCID and is set to <code>-</code> for NULL in my case. According to the standard it might be used for the pid but is mostly implementation defined.</li> <li>The second null is the MSGID, it can define a message type from the specific service, it will also be null in most cases.</li> <li>The next part is <code>[timeQuality tzKnown=&quot;1&quot;]</code> which is the STRUCTURED-DATA field. It can contain any implementation defined data. This is a subset of the structured data created by the <code>logger</code> tool used to create test messages. This field can also be just <code>-</code> for no structured data.</li> <li>Finally the actual log message. That&#x27;s just <code>Hi</code> in this case.</li> </ul> <p>Writing a parser for this format is relatively straightforward. In the logbookd implementation there's a row for every one of these fields in the logging table and the message is split up according to these rules.</p> <p>There is a fatal flaw in the RFC 5424 specification though: nobody is using it. None of the log messages on my running systems are actually in this format. It looks like practically all software uses RFC 3164, which is a fancy way of saying they do whatever they want.</p> <h2>RFC 3164</h2> <p>So this is actually the true specification for syslog messages being used in the wild. Let's look at one of these messages:</p> <pre><code>&lt;13&gt;Dec 11 15:21:50 laptop test: Hi</code></pre> <p>It's a lot simpler! But not actually. This is a pretty minimal message. The initial part is the same as the RFC 5424 message, the PRI is luckily parsed the exact same way. There is no version indicator though and it does not use an ISO timestamp format.</p> <p>The more problematic issues with this format though is that it does support a lot more data but it's pretty badly defined. Even all the parts shown in the example above are optional. The most minimal syslog message that is still up to this spec is <code>Hi</code>.</p> <p>It's also somewhat valid to send messages with a badly formatted timestamp and it's up to the syslogd to fix up the timestamp in the message. This also makes it very easy to make it actually parse parts of the timestamp as the hostname since this is all badly defined and space separated.</p> <p>Since there is no official field for the pid of a process this is usually appended to the application name in square brackets.</p> <p>The logbookd implementation is mostly based on the way these old messages are parsed in rsyslog and tries to not guess parts. This means only the timestamp, app, hostname and message fields are filled in.</p> <h2>Kernel logging</h2> <p>Not all logging in the system comes from userspace. On Linux there's also the kernel log ringbuffer that can be read from <code>/dev/kmsg</code>. Reading from this file will return all the log messages in the kernel ringbuffer and also makes it possible to stream new log messages with further reads. The log messages from the kernel are in a similar but different format than the syslog socket:</p> <pre><code>6,1004,5150172365,-;hid-generic 0003: hiddev96,hidraw2: USB HID device on usb-0000:00:14 SUBSYSTEM=hid DEVICE=+hid:0003</code></pre> <p>The first field in the kernel message is again the PRI. This follows the same numbering as the syslog RFCs but it's not in angular brackets this time. In this case it's facility 0 (the kernel) and severity 6 (info). The second field is the KSEQ. This is a number that counts up for every log message since boot. The logbookd implementation uses this to de-duplicate the kernel log messages after opening the file since it will always return the old kernel log messages first.</p> <p>After that comes the timestamp. Instead of string parsing this is a straight up unix timestamp so it's way easier to deal with. The field after the timestamp is <code>-</code> indicating NULL, this is the flags field.</p> <p>After the semicolon the actual kernel log message starts. This is the message as is rendered in the <code>dmesg</code> utility. After the log message there's a newline but the log line doesn't end there! The structured data is defined as indented continuation lines after the message itself and this contains some easier machine-parsable data that is usually hidden in <code>dmesg</code>. </p> <h2>Systemd journald</h2> <p>So everything changed when journald was introduced. Figuring out how this all works involves diving into the systemd source code. Systemd provides several unix sockets related to logging in <code>/var/run/systemd/journal</code>:</p> <ul><li><code>dev-log</code> this is symlinked to <code>/dev/log</code> and receives syslog formatted lines and writes it to the journal</li> <li><code>stdout</code> is a socket that receives logs from systemd units. This is what the <code>systemd-cat</code> command connects to. It writes a header on connection to give the application metadata and then the stdout or stderr is just connected straight to this socket.</li> <li><code>socket</code> receives the log messages in the binary journald format</li> </ul> <p>There is a few other fancy things that journald does. It is possible to filter your log messages with the <code>--boot</code> argument. If no argument is supplied it will only show messages from the current boot. If you specify a negative number it's possible to get only log messages from a specific previous bootup.</p> <p>The way this is done is by reading from <code>/proc/sys/kernel/random/boot_id</code>. This is a value generated by the kernel on bootup. It is a UUID generated from random data. These are also the values you see when you run <code>journalctl --list-boots</code>. The BOOT_ID value shown there is this UUID with the dashes removed.</p> <p>My logbookd implementation also reads the boot_id on startup and stores it with the logs, this allows filtering in the exact same way with the logread <code>-b</code> parameter.</p> <h2>Logging to a database</h2> <p>So the main departure journald and logbookd do from the older syslog daemons is that they don't log to plain-text files. Journald has a custom database format the messages are stored in and logbookd stores messages in an SQLite database.</p> <p>Structured logging to a database has a few nice upsides. The main one is being able to do way more detailed filtering than what is reasonably possible with grep. It's a lot easier to filter on a specific date and time range in a database and due to database indexes this is still fast.</p> <p>One of the other main reasons for using SQLite in logbookd is that the implementation in postmarketOS was configured to only log to memory. Using SQLite as logging back-end meant that it's easily possible to replicate this by writing to an in-memory database which is already supported by SQLite.</p> <p>The final thing added to logbookd is the middle ground between in-memory and on-disk logging: the reduced writes mode. In this mode the syslog is written to an in-memory database but when receiving a SIGINT, SIGTERM or SIGUSR1 signal the logbookd daemon will open the on-disk database and lets SQLite do a database migration. This means that SQLite will append the write the new loglines to the disk without rewriting all the existing logs there. On bootup this database is restored again so the logging system behaves as-if it's configured to do normal on-disk logging.</p> <h2>You can use this now</h2> <p>If you're running postmarketOS edge and you have updated to the latest version your installation should've migrated to the logbookd logging daemon. The <code>logread</code> utility implements the common options the busybox logread command already had. For normal use this means that there's not much difference except that the log output from the logread utility is now colored and contains kernel logs.</p> <p>Some examples of the new things that are now possible:</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>logread -b list <span class="go">ID BOOT ID FIRST ENTRY LAST ENTRY</span> <span class="go"> 0 05c3f283-3bae-4b2a-8431-210dd63310e0 Dec 11 16:33:59 Dec 11 16:34:05</span> <span class="go">-1 f3ea2fa1-6f9e-4e82-bd0b-201091fcb5b4 Dec 07 18:21:06 Dec 07 18:25:50</span> <span class="gp">$ </span>logread -b <span class="m">0</span> -n <span class="m">2</span> <span class="go">[Dec 11 16:35:09] daemon dleyna-renderer-service[18060]: Client :1.166 lost</span> <span class="go">[Dec 11 16:35:10] daemon dleyna-renderer-service[18060]: dLeyna: Exit</span> <span class="gp">$ </span>logread -b <span class="m">1</span> -n <span class="m">1</span> <span class="go">[Dec 07 18:25:18] kern kernel: perf: interrupt took too long (2531 &gt; 2500), lowering kernel.perf_event_max_sample_rate to 79000</span> <span class="gp">$ </span>logread -b -1 -u logbookd -n <span class="m">1</span> <span class="go">[Dec 07 18:25:37] syslog logbookd: Ready to process log messages</span> </pre></div> <p>Being able to see interleaved kernel and userspace messages also makes certain scenarios a lot easier to debug.</p> <p>Hopefully this makes a few things easier to debug. There's a bunch of software that also logs directly into <code>/var/log</code> in seperate files, this has not been replaced by logbookd and is also not directly query-able by this new system. For the rest of the log messages enjoy the new colors :)</p> Megapixels 2.0: DNG exportinghttps://blog.brixit.nl/megapixels-2-0-dng-exporting/89MegapixelsMartijn BraamSat, 18 Nov 2023 14:17:38 -0000<p>It seems overkill to make a whole seperate library dedicated to replacing 177 lines of code in Megapixels that touches libtiff, but this small section of code causes significant issues for distribution packaging and compatability with external photo editing software. Most importantly the adjusted version in Millipixels for the Librem 5 does not output DNG files that are close enough to the Adobe specifications to be loaded into the calibration software.</p> <p>Making this a seperate library would make it easier to test. In the Adobe DNG SDK there is a test utility that can verify if a TIFF file is up to DNG spec and it can (with a lot of complications) be build for Linux.</p> <h2>The spec</h2> <p>The first thing after copying over the code block from Megapixels to a seperate project is reading the Adobe DNG specification.</p> <p>When I wrote the original export code in Megapixels it was based around some example code I found on Github for using Libtiff that I can no longer find and it results in something that's close enough to a valid DNG file for the <code>dcraw</code> utility. This is also a DNG 1.0 file that is generated.</p> <p>I have spend the next day reading the <a href="https://www.kronometric.org/phot/processing/DNG/dng_spec_1.4.0.0.pdf">DNG 1.4 specification</a> from Adobe to understand what a valid DNG file is absolutely minimally required to have. These are my notes from that:</p> <div class="highlight"><pre><span></span><span class="gu">## Inside a DNG file</span> <span class="k">*</span> SubIFDType 0 is the original raw data <span class="k">*</span> SubIFDType 1 is the thumbnail data <span class="k">*</span> The recommendation is to store the thumbnail as the first IFD <span class="k">*</span> TIFF metdata goes in the first IFD <span class="k">*</span> EXIF tags are preferred <span class="k">*</span> Camera profiles are stored in the first IFD <span class="gu">## Required tags</span> <span class="k">*</span> DNGVersion <span class="k">*</span> UniqueCameraModel </pre></div> <h2>Validation</h2> <p>I also spend a long time to build the official Adobe DNG SDK. This is mostly useless for developing any open source software due to licensing but it does provide a nice <code>dng_validate</code> utility that can be used to actually test the DNG files. Building this utility is pretty horrifying since it requires some specific versions of dependencies and some patches to work on modern compilers.</p> <p>The libdng codebase now has the <a href="https://gitlab.com/megapixels-org/libdng/-/blob/master/adobe_dng_sdk.sh">adobe_dng_sdk.sh</a> script that will build the required libraries and the validation binary.</p> <p>with the Megapixels code adjusted with the info from the documentation above I fed some random noise as data to the library to generate a DNG file and run it through the validator.</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>dng_validate out.dng <span class="go">Validating &quot;out.dng&quot;...</span> <span class="go">*** Warning: This file has Chained IFDs, which will be ignored by DNG readers ***</span> <span class="go">*** Error: Unable to find main image IFD ***</span> </pre></div> <p>Well that's not a great start... There's also a <code>-v</code> option to get some more verbose info</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>dng_validate -v out.dng <span class="go">Validating &quot;out.dng&quot;...</span> <span class="go">Uses little-endian byte order</span> <span class="go">Magic number = 42</span> <span class="go">IFD 0: Offset = 308, Entries = 10</span> <span class="go">NewSubFileType: Preview Image</span> <span class="go">ImageWidth: 20</span> <span class="go">ImageLength: 15</span> <span class="go">BitsPerSample: 8</span> <span class="go">Compression: Uncompressed</span> <span class="go">PhotometricInterpretation: RGB</span> <span class="go">StripOffsets: Offset = 8</span> <span class="go">StripByteCounts: Count = 300</span> <span class="go">DNGVersion: 1.4.0.0</span> <span class="go">UniqueCameraModel: &quot;LibDNG&quot;</span> <span class="go">NextIFD = 10042</span> <span class="go">Chained IFD 1: Offset = 10042, Entries = 6</span> <span class="go">NewSubFileType: Main Image</span> <span class="go">ImageWidth: 320</span> <span class="go">ImageLength: 240</span> <span class="go">Compression: Uncompressed</span> <span class="go">StripOffsets: Offset = 441</span> <span class="go">StripByteCounts: Count = 9600</span> <span class="go">NextIFD = 0</span> <span class="go">*** Warning: This file has Chained IFDs, which will be ignored by DNG readers ***</span> <span class="go">*** Error: Unable to find main image IFD ***</span> </pre></div> <p>Let's have a look at what the DNG spec says about this:</p> <blockquote>DNG recommends the use of SubIFD trees, as described in the TIFF-EP specification. SubIFD chains are not supported.<br><br>The highest-resolution and quality IFD should use NewSubFileType equal to 0. Reduced resolution (or quality) thumbnails or previews, if any, should use NewSubFileType equal to 1 (for a primary preview) or 10001.H (for an alternate preview). <br><br>DNG recommends, but does not require, that the first IFD contain a low-resolution thumbnail, as described in the TIFF-EP specification.</blockquote> <p>So I have the right tags and the right IFDs but I need to make an IFD tree instead of chain in libtiff. I have no idea how IFD trees work so up to the next specification!</p> <p>It seems like TIFF trees are defined in the Adobe PageMaker 6 tech notes from 1995. That document describes that the NextIFD tag that libtiff used for me is used primarily for defining multi-page documents, not multiple encodings of the same document like what happens here with a thumbnail and the raw data. You know this is a 1995 spec because it gives a Fax as example of a multi-page document.</p> <p>In the examples provided in that specification the first image is the main image and the NextIFD tag is just replaced by a subIFD tag. In case of DNG the main image is the thumbnail for compatibility with software that can't read the raw camera data.</p> <p>Switching over to a SubIFD tag is suprisingly simple, just badly documented. Libtiff will create the NextIFD tag automatically for you but if you create an empty SubIFD tag then libtiff will fill in the offset for the next IFD for you when closing the file:</p> <div class="highlight"><pre><span></span><span class="n">TIFF</span><span class="w"> </span><span class="o">*</span><span class="n">tif</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">TIFFOpen</span><span class="p">(</span><span class="n">path</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;w&quot;</span><span class="p">);</span><span class="w"></span> <span class="c1">// Set the tags for IFD 0 like normal here</span> <span class="n">TIFFSetField</span><span class="p">(</span><span class="n">tif</span><span class="p">,</span><span class="w"> </span><span class="n">TIGTAG_SUBFILETYPE</span><span class="p">,</span><span class="w"> </span><span class="n">DNG_SUBFILETYPE_THUMBNAIL</span><span class="p">);</span><span class="w"></span> <span class="c1">// Create a NULL reference for one SubIFD</span> <span class="kt">uint64_t</span><span class="w"> </span><span class="n">offsets</span><span class="p">[]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="mf">0L</span><span class="w"> </span><span class="p">};</span><span class="w"></span> <span class="n">TIFFSetField</span><span class="p">(</span><span class="n">tif</span><span class="p">,</span><span class="w"> </span><span class="n">TIFFTAG_SUBIFD</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="o">&amp;</span><span class="n">offsets</span><span class="p">);</span><span class="w"></span> <span class="c1">// Write the thumbnail image data here</span> <span class="c1">// Close the first IFD</span> <span class="n">TIFFWriteDirectory</span><span class="p">(</span><span class="n">tif</span><span class="p">);</span><span class="w"></span> <span class="c1">// Start IFD1 describing the raw data</span> <span class="n">TIFFSetField</span><span class="p">(</span><span class="n">tif</span><span class="p">,</span><span class="w"> </span><span class="n">TIFFTAG_SUBFILETYPE</span><span class="p">,</span><span class="w"> </span><span class="n">DNG_SUBFILETYPE_ORIGINAL</span><span class="p">);</span><span class="w"></span> <span class="c1">// write raw data and close the directory again</span> <span class="n">TIFFWriteDirectory</span><span class="p">(</span><span class="n">tif</span><span class="p">);</span><span class="w"></span> <span class="c1">// Close the tiff, this will cause libtiff to patch up the references</span> <span class="n">TIFFCLose</span><span class="p">(</span><span class="n">tif</span><span class="p">);</span><span class="w"></span> </pre></div> <p>So with the code updated the validation tool neatly shows the new SubIFD tags and finds actual errors in my DNG file data now</p> <pre><code>Uses little-endian byte order Magic number = 42 IFD 0: Offset = 308, Entries = 11 NewSubFileType: Preview Image ImageWidth: 20 ImageLength: 15 BitsPerSample: 8 Compression: Uncompressed PhotometricInterpretation: RGB StripOffsets: Offset = 8 StripByteCounts: Count = 300 SubIFDs: IFD = 10054 DNGVersion: 1.4.0.0 UniqueCameraModel: &quot;LibDNG&quot; NextIFD = 0 SubIFD 1: Offset = 10054, Entries = 6 NewSubFileType: Main Image ImageWidth: 320 ImageLength: 240 Compression: Uncompressed StripOffsets: Offset = 453 StripByteCounts: Count = 9600 NextIFD = 0 *** Error: Missing or invalid SamplesPerPixel (IFD 0) *** *** Error: Missing or invalid PhotometricInterpretation (SubIFD 1) ***</code></pre> <p>Ah, so these two tags are actually required but not described as such in the DNG specification since these are TIFF tags instead of DNG tags (while it does explicitly tells other TIFF required data).</p> <p>Patching up these errors is easy, just slightly annoying since the validation tool seemingly gives only a single error per IFD requiring to iterate on the code a bit more. After a whole lot of iterating on the exporting code I managed to get the first valid DNG file:</p> <pre><code>Raw image read time: 0.000 sec Linearization time: 0.002 sec Interpolate time: 0.006 sec Validation complete</code></pre> <p>Now the next step is adding all the plumbing to make this usable as library and making an actually nice command line utility.</p> <h2>First actual test</h2> <p>Now I have written the first iterations of libmegapixels and libdng it should be possible to actually load a picture in some editing software. So let's try some end-to-end testing with this.</p> <p>With the <code>megapixels-getframe</code> utility from libmegapixels I can get a frame from the sensor (In this case the rear camera of the Librem 5) and then feed that raw data to the <code>makedng</code> utility from libdng.</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>getframe -o test.raw <span class="go">Using config: /usr/share/megapixels/config/purism,librem5.conf</span> <span class="go">received frame</span> <span class="go">received frame</span> <span class="go">received frame</span> <span class="go">received frame</span> <span class="go">received frame</span> <span class="go">Stored frame to: test.raw</span> <span class="go">Format: 4208x3120</span> <span class="go">Pixfmt: GRBG</span> <span class="gp">$ </span>makedng -w <span class="m">4208</span> -h <span class="m">3120</span> -p GRBG test.raw test.dng <span class="go">Reading test.raw...</span> <span class="go">Writing test.dng...</span> </pre></div> <p>No errors and the file passes the DNG validation, let's load it into RawTherapee :)</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1700184535/image.png" class="kg-image"><figcaption>The first frame loaded into RawTherapee</figcaption></figure> <p>I had to boost the exposure a bit since the <code>megapixels-getframe</code> tool does not actually control any of the sensor parameters like the exposure time so the resulting picture is very dark. There's also no whitebalance or autofocus happening so the colors look horrible. </p> <p>But... </p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1700184873/compare-checker.jpg" class="kg-image"></figure> <p>The colors are correct! The interpetation of the CFA pattern of the sensor and the orientation of the data is all correct.</p> <h2>Integration testing</h2> <p>The nice thing about having the seperate library is that testing it becomes a lot easier than testing a GTK4 application. I have added the first simple end-to-end test to the codebase now that feeds some data to makedng and checks if the result is a valid DNG file using the official Adobe tool.</p> <div class="highlight"><pre><span></span><span class="ch">#!/bin/bash</span> <span class="nb">set</span> -e <span class="k">if</span> <span class="o">[</span> <span class="nv">$#</span> -ne <span class="m">1</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> <span class="nb">echo</span> <span class="s2">&quot;Missing tool argument&quot;</span> <span class="nb">exit</span> <span class="m">1</span> <span class="k">fi</span> <span class="nv">makedng</span><span class="o">=</span><span class="s2">&quot;</span><span class="nv">$1</span><span class="s2">&quot;</span> <span class="nb">echo</span> <span class="s2">&quot;Running tests with &#39;</span><span class="nv">$makedng</span><span class="s2">&#39;&quot;</span> <span class="c1"># This testsuite runs raw data through the makedng utility and validates the</span> <span class="c1"># result using the dng_validate tool from the Adobe DNG SDK. This tool needs</span> <span class="c1"># to be manually installed for these tests to run.</span> <span class="c1"># Create test raw data</span> mkdir -p scratch magick -size 1280x720 gradient: -colorspace RGB scratch/data.rgb <span class="c1"># Generate DNG</span> <span class="nv">$makedng</span> -w <span class="m">1280</span> -h <span class="m">720</span> -p RG10 scratch/data.rgb scratch/RG10.dng <span class="c1"># Validate DNG</span> dng_validate scratch/RG10.dng </pre></div> <p>This is launched from ctest in my cmake files for now since I'm developing most of this stuff using CLion which only properly supports cmake projects. This is why a lot of my C projects have both meson and cmake files to build them but only the meson project file has install commands in it.</p> <p>For more advanced testing it would be neat to have raw sensor dumps of several sensors in different formats which are all pictures of a colorchecker like the picture above. Then have some (probably opencv) utility that can validate that a colorchecker is present in the picture with the right colors.</p> <p>There also needs to be a non-adobe-propriatary validation tool that can be easily run as testsuite for distribution packaging so at build time it's possible to validate that the combination of libdng and the distribution version of libtiff can produce sane output. This has caused several issues in Megapixels before after all.</p> <h2>Overall architecture</h2> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1700232871/path4862-1-4.png" class="kg-image"><figcaption>I&#x27;ve spent too much time drawing this</figcaption></figure> <p>With the addition of libdng the architecture for Megapixels 2.0 starts to look like this. Megapixels no longer has any pipeline manipulation code, that is all handled by the library which after configuration just passes the file descriptor for the sensor node to Megapixels to handle the realtime control of the sensor parameters.</p> <p>The libdng code replaces the plain libtiff exporting done in Megapixels and generate the DNG files that will be read by postprocessd. Postprocessd reads the dng files with the help of the dcraw library which already has custom DNG reading code that does not use libtiff.</p> <p>The next steps now is to flesh out the library public interface for libdng so it can do all the DNG metadata that Megapixels requires and then hooking it up to Megapixels to actually use it.</p> <hr> <h3>Funding update</h3> <p>Since my <a href="https://blog.brixit.nl/adding-hardware-to-libmegapixels/">previous post</a> about the libmegapixels developments and the <a href="https://blog.brixit.nl/megapixels-2-0/">Megapixels 2.0 post</a> I wrote before that I've almost doubled the funding for actually working on all the FOSS contributions. I'm immensely thankful for all the new patrons and it also made me notice that the <a href="https://blog.brixit.nl/donations/">donations</a> page on this site was no longer being regenerated. That is fixed now.</p> <p>I'm also still trying to figure out if I can add some perks for patrons to all of this but practically all options just amount to making things slightly worse for non-patrons. I hope just making the FOSS ecosystem better one of code line at a time is enough :)</p> Adding hardware to libmegapixelshttps://blog.brixit.nl/adding-hardware-to-libmegapixels/88MegapixelsMartijn BraamMon, 13 Nov 2023 17:59:48 -0000<p>Since in the last post I only showed off the libmegapixels config format and made some claims about configurablility without demonstrating it. I thought that it might be a good idea to actually demonstrate and document it.</p> <p>As example device I will use my Xiaomi Mi Note 2 with a broken display, shown above. Also known in PostmarketOS under the codename <a href="https://wiki.postmarketos.org/wiki/Xiaomi_Mi_Note_2_(xiaomi-scorpio)">xiaomi-scorpio</a>. I picked this device as demo since I have already used this hardware in Megapixels 1.x so I know the kernel side of it is functional. I have not run any libmegapixels code on this device before writing this blogpost so I'm writing it as a I go along debugging it. Hopefully this device does not require any ioctl that has not been needed by the existing supported devices.</p> <p>What makes it possible to get camera output from this phone is two things:</p> <ul><li>The camera subsystem in this device is supported pretty well in the kernel, in this case it&#x27;s a Qualcomm device which has a somewhat universal driver for this</li> <li>The sensor in this phone has a proper driver</li> </ul> <p>The existing devices that I used to develop libmegapixels are based around the Rockchip, NXP and Allwinner platforms so this will be an interesting test if my theory works.</p> <h2>The config file name</h2> <p>Just like Megapixels 1.x the config file is based around the "compatible" name of the device. This is defined in the device tree passed to Linux by the bootloader. Since this is a nice mainline Linux device this info can be found in the kernel source: <a href="https://github.com/torvalds/linux/blob/b85ea95d086471afb4ad062012a4d73cd328fa86/arch/arm64/boot/dts/qcom/msm8996pro-xiaomi-scorpio.dts#L17">https://github.com/torvalds/linux/blob/b85ea95d086471afb4ad062012a4d73cd328fa86/arch/arm64/boot/dts/qcom/msm8996pro-xiaomi-scorpio.dts#L17</a></p> <pre><code>compatible = &quot;xiaomi,scorpio&quot;, &quot;qcom,msm8996pro&quot;, &quot;qcom,msm8996&quot;;</code></pre> <p>This device tree specifies three names for this device ranking from more specific to less specific. <code>xiaomi,scorpio</code> is the exact hardware name, <code>qcom,msm8996pro</code> is the variant of the SoC and the <code>qcom,msm8996</code> name is the inexact name of the SoC. Since this configuration defined both the SoC pipeline and the configuration for the specific sensor module the only sane option here is <code>xiaomi,scorpio</code> since that describes that exact hardware configuration. Other <code>msm8996</code> devices might be using a completely different sensor.</p> <p>The most specific option is not always the best option, in the case of the PinePhone for example the compatible is:</p> <pre><code>&quot;pine64,pinephone-1.1&quot;, &quot;pine64,pinephone&quot;, &quot;allwinner,sun50i-a64&quot;;</code></pre> <p>In this hardware the camer system for the 1.0, 1.1 and 1.2 revision is identical so the config file just uses the <code>pine64,pinephone</code> name.</p> <p>Knowing this the config file name will be <code>xiaomi,scorpio.conf</code> and can be placed in three locations. <code>/usr/share/megapixels/config</code>, <code>/etc/megapixels/config</code> and just the plain filename in your current working directory.</p> <p>Now we know what the config path is the hard part starts, figuring out what to put in this config file.</p> <h2>The media pipeline</h2> <p>The next step is figuring out the media pipeline for this device. If the kernel has support for the hardware in the device it should create one or more <code>/dev/media</code> files. In the case of the Scorpio there's only a single one for the camera pipeline but there might be additional ones for stuff like hardware accelerated video encoding or decoding. </p> <p>You can get the contents of the media pipelines with the <code>media-ctl</code> utility from <code>v4l-utils</code>. Use <code>media-ctl -p</code> to print the pipeline and you can use the <code>-d</code> option to choose another file than <code>/dev/media0</code> if needed. For the Scorpio the pipeline contents are:</p> <pre><code>Media controller API version 6.1.14 Media device information ------------------------ driver qcom-camss model Qualcomm Camera Subsystem serial bus info platform:a34000.camss hw revision 0x0 driver version 6.1.14 Device topology - entity 1: msm_csiphy0 (2 pads, 5 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev0 pad0: Sink [fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb] &lt;- &quot;imx318 3-001a&quot;:0 [ENABLED,IMMUTABLE] pad1: Source [fmt:UYVY8_2X8/1920x1080 field:none colorspace:srgb] -&gt; &quot;msm_csid0&quot;:0 [] -&gt; &quot;msm_csid1&quot;:0 [] -&gt; &quot;msm_csid2&quot;:0 [] -&gt; &quot;msm_csid3&quot;:0 [] [ Removed A LOT of entities here for brevity ] - entity 226: imx318 3-001a (1 pad, 1 link) type V4L2 subdev subtype Sensor flags 0 device node name /dev/v4l-subdev19 pad0: Source [fmt:SRGGB10_1X10/5488x4112@1/30 field:none colorspace:raw xfer:none] -&gt; &quot;msm_csiphy0&quot;:0 [ENABLED,IMMUTABLE] - entity 228: ak7375 3-000c (0 pad, 0 link) type V4L2 subdev subtype Lens flags 0 device node name /dev/v4l-subdev20 </code></pre> <p>The header shows that this is a media device for the <code>qcom-camss</code> system, which handles cameras on Qualcomm devices. There is also a node for the <code>imx318</code> sensor which further confirms that this is the right media pipeline.</p> <p>Analyzing the pipeline in this format is pretty hard when there's more than two nodes though, that's why there is a neat option in media-ctl to output the mediagraph as an actual graph using Graphviz.</p> <pre><code>$ apk add graphviz $ media-ctl -d 0 --print-dot | dot -Tpng &gt; pipeline.png</code></pre> <p>Which produces this image:</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1699888898/pipeline.png" class="kg-image"></figure> <p>In a bunch of cases you can copy most of the configuration of this graph from another device that uses the same SoC but since this is the first Qualcomm device I'm adding I have to figure out the whole pipeline.</p> <p>The only part that's really specific to the Xiaomi Scorpio is the top two nodes. The <code>imx318</code> is the actual camera module in the phone connected with mipi to the SoC. The <code>ak7375</code> is listed as a "Motor driver". This means that it is the chip handeling the lens movements for autofocus. There are no connections to this node since this device does not handle any graphical data, the entity only exists so you can set v4l control values on it to move the focus manually.</p> <p>All the boxes in the graph are called entities and correspond with the <code>Entity</code> blocks in the <code>media-ctl -p</code> output. The boxes are yellow if they are entities with the type <code>V4L</code>, these are the nodes that will show up als <code>/dev/video</code> nodes to actually get the image data out of this pipeline.</p> <p>The lines between the boxes are called links, the dotted lines are disabled links and solid lines are enabled links. On this hardware a lot of the links are created by the kernel driver and are hardcoded. These links show up in the text output as <code>IMMUTABLE</code> and mostly describe fixed hardware paths for the image data.</p> <p>The goal of configuring this pipeline is to get the image data from the IMX sensor all the way down to one of the /dev/video nodes and figuring out the purpose of the entities in between. If you are lucky there is actual documentation for this. In this case I have found documentation at <a href="https://www.kernel.org/doc/html/v4.14/media/v4l-drivers/qcom_camss.html">https://www.kernel.org/doc/html/v4.14/media/v4l-drivers/qcom_camss.html</a> which is for the v4.14 kernel but for some reason is removed on later releases.</p> <p>This documentation has neat explanations for these entities:</p> <ul><li>2 CSIPHY modules. They handle the Physical layer of the CSI2 receivers. A separate camera sensor can be connected to each of the CSIPHY module;</li> <li>2 CSID (CSI Decoder) modules. They handle the Protocol and Application layer of the CSI2 receivers. A CSID can decode data stream from any of the CSIPHY. Each CSID also contains a TG (Test Generator) block which can generate artificial input data for test purposes;</li> <li>ISPIF (ISP Interface) module. Handles the routing of the data streams from the CSIDs to the inputs of the VFE;</li> <li>VFE (Video Front End) module. Contains a pipeline of image processing hardware blocks. The VFE has different input interfaces. The PIX (Pixel) input interface feeds the input data to the image processing pipeline. The image processing pipeline contains also a scale and crop module at the end. Three RDI (Raw Dump Interface) input interfaces bypass the image processing pipeline. The VFE also contains the AXI bus interface which writes the output data to memory.</li> </ul> <p>This documentation is not for this exact SoC so the amount of entities of each type is different.</p> <p>Configuring the pipeline and connecting it all up is now just a lot of trial and error, in the case of the Scorpio it has already been trial-and-error'd so there is an existing config file for the old Megapixels at <a href="https://gitlab.com/postmarketOS/megapixels/-/blob/master/config/xiaomi,scorpio.ini?ref_type=heads">https://gitlab.com/postmarketOS/megapixels/-/blob/master/config/xiaomi,scorpio.ini</a></p> <p>In this old pipeline description format the path is just enabling the links between the first <code>csiphy</code>, <code>csid</code>, <code>ispif</code> and <code>vfe</code> entity. Since this release of Megapixels did not really support further configuration it just tried to then set the resolution and pixel format for the sensors on all entities after it and hoped it worked. On an unknown platform just picking the left-most path will pretty likely bring up a valid pipeline, the duplicated entities are mostly useful for cases where you are using multiple cameras at once.</p> <h2>Initial config file</h2> <p>The first thing I did is creating a minimal config file for the scorpio that had the minimal pipeline to stream unmodified data from the sensor to userspace.</p> <pre><code>Version = 1; Make: &quot;Xiaomi&quot;; Model: &quot;Scorpio&quot;; Rear: { SensorDriver: &quot;imx318&quot;; BridgeDriver: &quot;qcom-camss&quot;; Modes: ( { Width: 3840; Height: 2160; Rate: 30; Format: &quot;RGGB10&quot;; Rotate: 90; Pipeline: ( {Type: &quot;Link&quot;, From: &quot;imx318&quot;, FromPad: 0, To: &quot;msm_csiphy0&quot;, ToPad: 0}, {Type: &quot;Link&quot;, From: &quot;msm_csiphy0&quot;, FromPad: 1, To: &quot;msm_csid0&quot;, ToPad: 0}, {Type: &quot;Link&quot;, From: &quot;msm_csid0&quot;, FromPad: 1, To: &quot;msm_ispif0&quot;, ToPad: 0}, {Type: &quot;Link&quot;, From: &quot;msm_ispif0&quot;, FromPad: 1, To: &quot;msm_vfe0_rdi0&quot;, ToPad: 0}, {Type: &quot;Mode&quot;, Entity: &quot;imx318&quot;}, {Type: &quot;Mode&quot;, Entity: &quot;msm_csiphy0&quot;}, {Type: &quot;Mode&quot;, Entity: &quot;msm_csid0&quot;}, {Type: &quot;Mode&quot;, Entity: &quot;msm_ispif0&quot;}, ); }, ); }; </code></pre> <p>This can be tested with the <code>megapixels-getframe</code> command.</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>./megapixels-getframe <span class="go">Using config: /etc/megapixels/config/xiaomi,scorpio.conf</span> <span class="go">[libmegapixels] Could not link 226 -&gt; 1 [imx318 -&gt; msm_csiphy0] </span> <span class="go">[libmegapixels] Capture driver changed pixfmt to UYVY</span> <span class="go">Could not select mode</span> </pre></div> <p>This command tries to output as much debugging info as possible, but the reality is that you'll most likely need to look at the kernel source to figure out what is happening and what arbitrary constraints exist.</p> <p>So the iterating and figuring out errors starts. First the most problematic line is the <code>UYVY</code> format one. This most likely means that the pipeline pixelformat I selected was not correct and to fix that the kernel helpfully selects a completely different one. <code>getframe</code> will detect this and show this happening. In this case the RGGB10 format is wrong and it should have been RGGB10p. The kernel implementation is a bit inconsistent about which format it actually is while MIPI only allows one of these two in the spec. Changing that removes that error.</p> <p>The other interesting error is the link that could not be created. If you look closely at the Graphviz output you'll see that this link is already enabled by the kernel and in the text output it is also <code>IMMUTABLE</code>. This config line can be dropped because this is not configurable.</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>./megapixels-getframe <span class="go">Using config: /etc/megapixels/config/xiaomi,scorpio.conf</span> <span class="go">VIDIOC_STREAMON failed: Broken pipe</span> </pre></div> <p>Progress! At least somewhat. The mode setting commands succeed but now the pipeline can not actually be started. This is because some drivers only validate options when starting the pipeline instead of when you're actually setting modes. This is one of the most annoying errors to fix because there's no feedback whatsoever on <i>what</i> or <i>where</i> the config issue is.</p> <p>My suggestion for this is to first run <code>media-ctl -p</code> again and see the current state of the pipeline. This output shows the format for the pads of the pipeline so you can find a connection that might be invalid by comparing those. My pipeline state at this point is:</p> <ul><li><code>imx318</code>: <code>SRGGB10_1X10/3840x2160@1/30</code></li> <li><code>csiphy0</code>: <code>SRGGB10_1X10/3840x2160</code></li> <li><code>csid0</code>: <code>SRGGB10_1X10/3840x2160</code></li> <li><code>ispif0</code>: <code>SRGGB10_1X10/3840x2160</code></li> <li><code>vfe0_rdi0</code>: <code>UYVY8_2X8/1920x1080</code></li> </ul> <p>AHA! the last node is not configured correctly. It's always the last one you look at. It turns out the issue was that I'm simply missing a mode command in my config file that sets the mode on that entity so it's left at the pipeline defaults. Let's test the pipeline with that config added:</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>/megapixels-getframe <span class="go">Using config: /etc/megapixels/config/xiaomi,scorpio.conf</span> <span class="go">received frame</span> <span class="go">received frame</span> <span class="go">received frame</span> <span class="go">received frame</span> <span class="go">received frame</span> </pre></div> <p>The pipeline is streaming! This is the bare minimum configuration needed to make Megapixels 2.0 use this camera. For reference after all the changes above the config file is:</p> <pre><code>Version = 1; Make: &quot;Xiaomi&quot;; Model: &quot;Scorpio&quot;; Rear: { SensorDriver: &quot;imx318&quot;; BridgeDriver: &quot;qcom-camss&quot;; Modes: ( { Width: 3840; Height: 2160; Rate: 30; Format: &quot;RGGB10p&quot;; Rotate: 90; Pipeline: ( {Type: &quot;Link&quot;, From: &quot;msm_csiphy0&quot;, FromPad: 1, To: &quot;msm_csid0&quot;, ToPad: 0}, {Type: &quot;Link&quot;, From: &quot;msm_csid0&quot;, FromPad: 1, To: &quot;msm_ispif0&quot;, ToPad: 0}, {Type: &quot;Link&quot;, From: &quot;msm_ispif0&quot;, FromPad: 1, To: &quot;msm_vfe0_rdi0&quot;, ToPad: 0}, {Type: &quot;Mode&quot;, Entity: &quot;imx318&quot;}, {Type: &quot;Mode&quot;, Entity: &quot;msm_csiphy0&quot;}, {Type: &quot;Mode&quot;, Entity: &quot;msm_csid0&quot;}, {Type: &quot;Mode&quot;, Entity: &quot;msm_ispif0&quot;}, {Type: &quot;Mode&quot;, Entity: &quot;msm_vfe0_rdi0&quot;}, ); }, ); };</code></pre> <h2>Camera metadata</h2> <p>The config file not only stores information about the media pipeline but can also store information about the optical path. Every mode can define the focal length for example because changing the cropping on the sensor will give you digital zoom and thus a longer focal length. With modern phones with 10 cameras on the back it is also possible to define all of them as the "rear" camera and have multiple modes with multiple focal lengths so camera apps can switch the pipeline for zooming once zooming is implemented in the UI.</p> <p>Finding out the values for this optical path is basically just using search engines to find datasheets and specs. Sometimes the pictures generated by android have the correct information for this in the metadata as well.</p> <p>This information is also mostly absent from sensor datasheets since that only describe the sensor itself, you either need to find this info from the camera module itself (which is the sensor plus the lens) or the specifications for the phone.</p> <p>From spec listings and review sites I've found that the focal length for the rear camera is 4.06mm and the aperture is f/2.0. This can be added to the mode section:</p> <pre><code>Width: 3840; Height: 2160; Rate: 30; Format: &quot;RGGB10p&quot;; Rotate: 90; FocalLength: 4.06; FNumber: 2.0;</code></pre> <h2>Reference for pipeline commands</h2> <p>Since this is now practically the main reference for writing config files until I get documentation generation up and running for libmegapixels I will put the complete documentation for the various commands here.</p> <p>While parsing the config file there are four values stored as state : <code>width</code>, <code>height</code>, <code>format</code> and <code>rate</code>. The values for these default to the ones set in the mode and they are updated whenever you define one of these values explicitly in a command. This prevents having to write the same resolution values repeatedly on every line but it still allows having entities in the pipeline that scale the resolution.</p> <h3>Link</h3> <pre><code>{ Type: &quot;Link&quot;, From: &quot;msm_csiphy0&quot;, # Source entity name, required FromPad: 1, # Source pad, defaults to 0 To: &quot;msm_csid0&quot;, # Target entity name, required ToPad: 0 # Target pad, defaults to 0 }</code></pre> <p>Translates to an <code>MEDIA_IOC_SETUP_LINK</code> ioctl on the media device.</p> <h3>Mode</h3> <pre><code>{ Type: &quot;Mode&quot;, Entity: &quot;imx318&quot; # Entity name, required Width: 1280 # Horisontal resolution, defaults to previous in pipeline Height: 720 # Vertical resolution, defaults to previous in pipeline Pad: 0 # Pad to set the mode on, defaults to 0 Format: &quot;RGGB10p&quot; # Pixelformat for the mode, defaults to previous in pipeline }</code></pre> <p>Translates to an <code>VIDIOC_SUBDEV_S_FMT</code> ioctl on the entity.</p> <h3>Rate</h3> <pre><code>{ Type: &quot;Rate&quot;, Entity: &quot;imx318&quot;, # Entity name, required Rate: 30 # FPS, defaults to previous in pipeline }</code></pre> <p>Translates to an <code>VIDIOC_SUBDEV_S_FRAME_INTERVAL</code> ioctl on the entity.</p> <h3>Crop</h3> <pre><code>{ Type: &quot;Crop&quot;, Entity: &quot;imx318&quot;, # Entity name, required Width: 1280 # Area width, defaults to previous width in pipeline Height: 720 # Area height resolution, defaults to previous height in pipeline Top: 0 # The vertical offset, defaults to 0 Left: 0 # The horisontal offset, defaults to 0 Pad: 0 # Pad to set the crop on, defaults to 0 }</code></pre> <p>Translates to an <code>VIDIOC_SUBDEV_S_CROP</code> ioctl on the entity.</p> <h2>The future of libmegapixels</h2> <p>It has been quite a bit of work to create libmegapixels and it has been a mountain of work to rework Megapixels to integrate it. The first 90% of this is done but the trick is always in getting the second 90% finished. In the <a href="https://blog.brixit.nl/megapixels-2-0/">Megapixels 2.0</a> post I already mentioned this has burned me out. On the other hand it's a shame to let this work go to waste.</p> <p>There is a few parts of autofocus, autoexposure and autowhitebalance that are very complicated and math heavy to figure out, I can't figure it out. The loop between libmegapixels and Megapixels exists to pass around the values but I can't stop the system from oscillating and can't get it to settle on good values. There seems to be no good public information available on how to implement this in any case.</p> <p>Another difficult part is sensor calibration. I have the hardware and software to create calibration profiles but this system expects the input pictures to come from... working cameras. The system completely lacks proper sensor linearisation which makes setting a proper whitebalance not really possible. You might have noticed the specific teal tint that gives away that a picture is taken on a Librem 5 for example. If that teal tint is corrected for manually then the midtones will look correct but highlights will become too yellow. Maybe there's a way to calibrate this properly or maybe this just takes someone messing with the curves manually for a long while to get correct.</p> <p>There also needs to be an alternative to writing dng files with libtiff so for my own sanity it is required to write libdng. The last few minor releases of libtiff have all been messing with the tiff tags relating to DNG files which have caused taking pictures to not work for a lot of people. The only way around this seems to be stop using libtiff like all the Linux photography software has already done. This is not a terribly hard thing to implement, it just has been prioritized below getting color correct so far and I have not had the time to work on it.</p> <p>There is also still segfaults and crashes relating to the GPU debayer code in Megapixels for most of the pixel formats. This is very hard to debug due to the involvement of the GPU in the equation.</p> <h3>How can you help</h3> <p>If you know how to progress with any of this I gladly accept any patches for this to push it forward.</p> <p>The harder part of this section is... money. I love working on photography stuff, I can't believe the Megapixels implementation has even gotten this far but it basically takes me hyperfocusing for weeks for 12 hours per day on random camera code to get to this point, that is not really sustainable. It's great to work on this for some days and making progress, it's really painful to work for weeks on that one 30 line code block and making no progress whatsoever. At some point my dream is that I can actually live off doing open source work but so far that has still been a distant dream.</p> <p>I've had the <a href="https://blog.brixit.nl/donations/">donations</a> page now for some years and I'm incredibly happy that people are supporting me to work on this at all. It's just forever stuck on receiving enough money that you feel like a responsibility to produce progress but not nearly enough to actually fund that progress. So in practice only extra pressure.</p> <p>So I hate asking for money, but it would certainly help towards the dream of being an actual full time FOSS developer :)</p> Megapixels 2.0https://blog.brixit.nl/megapixels-2-0/87LinuxMartijn BraamThu, 09 Nov 2023 18:33:39 -0000<p>The Megapixels camera application has long been the most performant camera application on the original PinePhone. I have not gotten the Megapixels application to that point alone. There have been several other contributors that have helped slowly improving performance and features of this application. Especially Benjamin has leaped it forward massively with the threaded processing code and GPU accelerated preview.</p> <p>All this code has made Megapixels very fast on the PinePhone but also has made it quite a lot harder to port the application to other hardware. The code is very much overfitted for the PinePhone hardware.</p> <h2>Finding a better design</h2> <p>To address the elephant in the room, yes libcamera exists and promises to abstract this all away. I just disagree with the design tradeoffs taken with libcamera and I think that any competition would only improve the ecosystem. It can't be that libcamera got this exactly right on the first try right?</p> <p>Instead of the implementation that libcamera has made that makes abstraction code in c++ for every platform I have decided to pick the method that libalsa uses for the audio abstraction in userspace.</p> <p>Alsa UCM config files are selected by soundcard name and contain a set of instructions to bring the audio pipeline in the correct state for your current usecase. All the hardware specific things are not described in code but instead in plain text configuration files. I think this scales way better since it massively lowers the skill floor needed to actually mess with the system to get hardware working.</p> <p>The first iteration of Megapixels has already somewhat done this. There's a config file that is picked based on the hardware model that describes the names of the device nodes in /dev so those paths don't have to be hardcoded and it describes the resolution and mode to configure. It also describes a few details about the optical path to later produce correct EXIF info for the pictures.</p> <div class="highlight"><pre><span></span><span class="k">[device]</span><span class="w"></span> <span class="na">make</span><span class="o">=</span><span class="s">PINE64</span><span class="w"></span> <span class="na">model</span><span class="o">=</span><span class="s">PinePhone</span><span class="w"></span> <span class="k">[rear]</span><span class="w"></span> <span class="na">driver</span><span class="o">=</span><span class="s">ov5640</span><span class="w"></span> <span class="na">media-driver</span><span class="o">=</span><span class="s">sun6i-csi</span><span class="w"></span> <span class="na">capture-width</span><span class="o">=</span><span class="s">2592</span><span class="w"></span> <span class="na">capture-height</span><span class="o">=</span><span class="s">1944</span><span class="w"></span> <span class="na">capture-rate</span><span class="o">=</span><span class="s">15</span><span class="w"></span> <span class="na">capture-fmt</span><span class="o">=</span><span class="s">BGGR8</span><span class="w"></span> <span class="na">preview-width</span><span class="o">=</span><span class="s">1280</span><span class="w"></span> <span class="na">preview-height</span><span class="o">=</span><span class="s">720</span><span class="w"></span> <span class="na">preview-rate</span><span class="o">=</span><span class="s">30</span><span class="w"></span> <span class="na">preview-fmt</span><span class="o">=</span><span class="s">BGGR8</span><span class="w"></span> <span class="na">rotate</span><span class="o">=</span><span class="s">270</span><span class="w"></span> <span class="na">colormatrix</span><span class="o">=</span><span class="s">1.384,-0.3203,-0.0124,-0.2728,1.049,0.1556,-0.0506,0.2577,0.8050</span><span class="w"></span> <span class="na">forwardmatrix</span><span class="o">=</span><span class="s">0.7331,0.1294,0.1018,0.3039,0.6698,0.0263,0.0002,0.0556,0.7693</span><span class="w"></span> <span class="na">blacklevel</span><span class="o">=</span><span class="s">3</span><span class="w"></span> <span class="na">whitelevel</span><span class="o">=</span><span class="s">255</span><span class="w"></span> <span class="na">focallength</span><span class="o">=</span><span class="s">3.33</span><span class="w"></span> <span class="na">cropfactor</span><span class="o">=</span><span class="s">10.81</span><span class="w"></span> <span class="na">fnumber</span><span class="o">=</span><span class="s">3.0</span><span class="w"></span> <span class="na">iso-min</span><span class="o">=</span><span class="s">100</span><span class="w"></span> <span class="na">iso-max</span><span class="o">=</span><span class="s">64000</span><span class="w"></span> <span class="na">flash-path</span><span class="o">=</span><span class="s">/sys/class/leds/white:flash</span><span class="w"></span> <span class="k">[front]</span><span class="w"></span> <span class="na">...</span><span class="w"></span> </pre></div> <p>This works great for the PinePhone but it has a significant drawback. Most mobile cameras require an elaborate graph of media nodes to be configured before video works, the PinePhone is the exception in that the media graph only has an input and output node so Megapixels just hardcodes that part of the hardware setup. This makes the config file practically useless for all other phones and this is also one of the reason why different devices have different forks to make Megapixels work.</p> <p>So a config file that only works for a single configuration is pretty useless. Instead of making this an .ini file I've switched the design over to libconfig so I don't have to create a whole new parser and it allows for nested configuration blocks. The config file I have been using on the PinePhone with the new codebase is this:</p> <div class="highlight"><pre><span></span><span class="k">Version</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="m">1</span><span class="err">;</span><span class="w"></span> <span class="k">Make</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;PINE64&quot;</span><span class="err">;</span><span class="w"></span> <span class="k">Model</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;PinePhone&quot;</span><span class="err">;</span><span class="w"></span> <span class="k">Rear</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="k">SensorDriver</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;ov5640&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">BridgeDriver</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;sun6i-csi&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">FlashPath</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;/sys/class/leds/white:flash&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">IsoMin</span><span class="err">:</span><span class="w"> </span><span class="m">100</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">IsoMax</span><span class="err">:</span><span class="w"> </span><span class="m">64000</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Modes</span><span class="err">:</span><span class="w"> </span><span class="p">(</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="k">Width</span><span class="err">:</span><span class="w"> </span><span class="m">2592</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Height</span><span class="err">:</span><span class="w"> </span><span class="m">1944</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Rate</span><span class="err">:</span><span class="w"> </span><span class="m">15</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Format</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;BGGR8&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Rotate</span><span class="err">:</span><span class="w"> </span><span class="m">270</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">FocalLength</span><span class="err">:</span><span class="w"> </span><span class="m">3</span><span class="k">.</span><span class="m">33</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">FNumber</span><span class="err">:</span><span class="w"> </span><span class="m">3</span><span class="k">.</span><span class="m">0</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Pipeline</span><span class="err">:</span><span class="w"> </span><span class="p">(</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Link&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;ov5640&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">FromPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;sun6i-csi&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">ToPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;ov5640&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Width</span><span class="err">:</span><span class="w"> </span><span class="m">2592</span><span class="p">,</span><span class="w"> </span><span class="k">Height</span><span class="err">:</span><span class="w"> </span><span class="m">1944</span><span class="p">,</span><span class="w"> </span><span class="k">Format</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;BGGR8&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">)</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="k">Width</span><span class="err">:</span><span class="w"> </span><span class="m">1280</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Height</span><span class="err">:</span><span class="w"> </span><span class="m">720</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Rate</span><span class="err">:</span><span class="w"> </span><span class="m">30</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Format</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;BGGR8&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Rotate</span><span class="err">:</span><span class="w"> </span><span class="m">270</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">FocalLength</span><span class="err">:</span><span class="w"> </span><span class="m">3</span><span class="k">.</span><span class="m">33</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">FNumber</span><span class="err">:</span><span class="w"> </span><span class="m">3</span><span class="k">.</span><span class="m">0</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Pipeline</span><span class="err">:</span><span class="w"> </span><span class="p">(</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Link&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;ov5640&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">FromPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;sun6i-csi&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">ToPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;ov5640&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">)</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="p">}</span><span class="w"></span> <span class="w"> </span><span class="p">)</span><span class="err">;</span><span class="w"></span> <span class="p">}</span><span class="err">;</span><span class="w"></span> <span class="k">Front</span><span class="err">:</span><span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="k">SensorDriver</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;gc2145&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">BridgeDriver</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;sun6i-csi&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">FlashDisplay</span><span class="err">:</span><span class="w"> </span><span class="k">true</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Modes</span><span class="err">:</span><span class="w"> </span><span class="p">(</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="k">Width</span><span class="err">:</span><span class="w"> </span><span class="m">1280</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Height</span><span class="err">:</span><span class="w"> </span><span class="m">960</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Rate</span><span class="err">:</span><span class="w"> </span><span class="m">60</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Format</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;BGGR8&quot;</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Rotate</span><span class="err">:</span><span class="w"> </span><span class="m">90</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Mirror</span><span class="err">:</span><span class="w"> </span><span class="k">true</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="k">Pipeline</span><span class="err">:</span><span class="w"> </span><span class="p">(</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Link&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;gc2145&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">FromPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;sun6i-csi&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">ToPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;gc2145&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">)</span><span class="err">;</span><span class="w"></span> <span class="w"> </span><span class="p">}</span><span class="w"></span> <span class="w"> </span><span class="p">)</span><span class="err">;</span><span class="w"></span> </pre></div> <p>Instead of having a hardcoded preview mode and main mode for every sensor it's now possible to make many different resolution configs. This config recreates the 2 existing modes and Megapixels now picks faster mode for the preview automatically and use higher resolution modes for the actual picture. </p> <p>Every mode now also has a <code>Pipeline</code> block that describes the media graph as a series of commands, every line translates to one ioctl called on the right device node just like Alsa UCM files describe it as a series of amixer commands. Megapixels no longer has the implicit PinePhone pipeline so here it describes the one link it has to make between the sensor node and the csi node and it tells Megapixels to set the correct mode on the sensor node.</p> <p>This simple example of the PinePhone does not really show off most of the config options so lets look at a more complicated example:</p> <div class="highlight"><pre><span></span><span class="k">Pipeline</span><span class="err">:</span><span class="w"> </span><span class="p">(</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Link&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">From</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;imx258&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">FromPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">,</span><span class="w"> </span><span class="k">To</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_csi&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">ToPad</span><span class="err">:</span><span class="w"> </span><span class="m">0</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;imx258&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Format</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;RGGB10P&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Width</span><span class="err">:</span><span class="w"> </span><span class="m">1048</span><span class="p">,</span><span class="w"> </span><span class="k">Height</span><span class="err">:</span><span class="w"> </span><span class="m">780</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_csi&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_isp&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_isp&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Pad</span><span class="err">:</span><span class="w"> </span><span class="m">2</span><span class="p">,</span><span class="w"> </span><span class="k">Format</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;RGGB8&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Crop&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_isp&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Crop&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_isp&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Pad</span><span class="err">:</span><span class="w"> </span><span class="m">2</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_resizer_mainpath&quot;</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Mode&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_resizer_mainpath&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Pad</span><span class="err">:</span><span class="w"> </span><span class="m">1</span><span class="p">},</span><span class="w"></span> <span class="w"> </span><span class="p">{</span><span class="k">Type</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;Crop&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Entity</span><span class="err">:</span><span class="w"> </span><span class="s2">&quot;rkisp1_resizer_mainpath&quot;</span><span class="p">,</span><span class="w"> </span><span class="k">Width</span><span class="err">:</span><span class="w"> </span><span class="m">1048</span><span class="p">,</span><span class="w"> </span><span class="k">Height</span><span class="err">:</span><span class="w"> </span><span class="m">768</span><span class="p">},</span><span class="w"></span> <span class="p">)</span><span class="err">;</span><span class="w"></span> </pre></div> <p>This is the preview pipeline for the PinePhone Pro. Most of the Links are already hardcoded by the kernel itself so here it only creates the link from the rear camera sensor to the csi and all the other commands are for configuring the various entities in the graph.</p> <p>The <code>Mode</code> commands are basically doing the <code>VIDIOC_SUBDEV_S_FMT</code> ioctl on the device node found by the entity name. To make configuring modes on the pipeline not extremely verbose it implicitly takes the resolution, pixelformat and framerate from the main information set by the configuration block itself. Since several entities can convert the frames into another format or size it automatically cascades the new mode to the lines below it.</p> <p>In the example above the 5th command sets the format to <code>RGGB8</code> which means that the mode commands below it for <code>rkisp1_resizer_mainpath</code> also will use this mode but the <code>rkisp1_csi</code> mode command above it will still be operating in <code>RGGB10P</code> mode.</p> <h2>Splitting of device management code</h2> <p>Testing changes in Megapixels is pretty hard. To develop the Megapixels code I'm building it on the phone and launching it over SSH with a bunch of environment variables set so the GTK window shows up on the phone and I get realtime logs on my computer. If there's anything that's going on after the immediate setup code it is quite hard to debug because it's in one of the three threads that process the image data.</p> <p>To implement the new pipeline configuration I did that in a new empty project that builds a shared library and a few command line utilities that help test a few specific things. This codebase is <code>libmegapixels</code> and with it I have split off all hardware access from Megapixels itself making both these codebases a lot easier to understand.</p> <p>It has been a lot easier to debug complex camera pipelines using the commandline utilities and only working on the library code. It should also make it a lot easier to make Megapixels-like applications that are not GTK4 to make it integrate more with other environments. One of the test applications for libmegapixels is <code>getframe</code> which is now all you need to get a raw frame from the sensor.</p> <p>Since this codebase is now split into multiple parts I have put it into a seperate gitlab organisation at <a href="https://gitlab.com/megapixels-org">https://gitlab.com/megapixels-org</a> which hopefully keeps this a bit organized.</p> <p>This is also the codebase used for <a href="https://fosstodon.org/@martijnbraam/110775163438234897">https://fosstodon.org/@martijnbraam/110775163438234897</a> which shows off libmegapixels and megapixels 2.0 running on the Librem 5.</p> <h2>Burnout</h2> <p>So now the worse part of this blog post. No you can't use this stuff yet :(</p> <p>I've been working on this code for months, and now I've not been working on this code for months. I have completely burned out on all of this.</p> <p>The libmegapixels code is in pretty good state but the Megapixels rewrite is still a large mess:</p> <ul><li>Saving pictures doesn&#x27;t really work and I intended to split that off to create libdng</li> <li>The QR code support is not hooked up at all at the moment</li> <li>Several pixelformats don&#x27;t work correctly in the GPU decoder and I can&#x27;t find out why</li> <li>Librem 5 and PinePhone Pro really need auto-exposure, auto-focus and auto-whitebalance to produce anything remotely looking like a picture. I have ported the auto-exposure from Millipixels which works reasonably well for this but got stuck on AWB and have not attempted Autofocus yet.</li> </ul> <p>The mountain of work that's left to do to make this a superset of the functionality of Megapixels 1.x and the expectations surrounding it have made this pretty hard to work on. On the original Megapixel releases nothing mattered because any application that could show a single frame of the camera was already a 100% improvement over the current state.</p> <p>Another issue is that whatever I do or figure out it will always be instantly be put down with "Why are you not using libcamera" and "libcamera probably fixes this". </p> <p>Some things people really need to understand is that an application not using libcamera does <i>not</i> mean other software on the system can't support libcamera. If Firefox can use libcamera to do videocalls that's great, that's not the usecase Megapixels is going for anyway.</p> <p>What also doesn't help is receiving bugreports for the PinePhone Pro while Megapixels does not support the PinePhone Pro. There's a patchset added on top to make in launch on the PinePhone Pro but there's a reason this patchset is not in Megapixels. The product of the Megapixels source with the ppp.patch added on top probably shouldn't've been distributed as Megapixels...</p> <p>What also doesn't help is that if Megapixels 2.0 were finished and released it would also create a whole new wave of criticism and comparisons to libcamera. I would have to support Megapixels for the people complaining that it's not enough... You could've not had a camera application at all...</p> <p>It also doesn't help that the libcamerea developers are also the v4l2 subsystem maintainers in the kernel. I have during development of libmegapixels tried sending a simple patch for an issue I've noticed that would massively improve the ease of debugging PinePhone Pro cameras. I've sent this 3 character patch upstream to the v4l2 mailing lists and it got a Reviewed-by in a few days.</p> <p>Then after 2 whole months of radio silence it got rejected by the lead developer of libcamera on debatable grounds. Now this is only a very small patch so I'm merely dissapointed. If I had put more work into the kernel side improving some sensor drivers I might have been mad but at this point I'm just not feeling like contributing to the camera ecosystem anymore. </p> <hr> <p><b>Edit:</b> I've been convinced to actually try to do this full-time and push the codebase forward enough to make it usable. This is continued at <a href="https://blog.brixit.nl/adding-hardware-to-libmegapixels/">https://blog.brixit.nl/adding-hardware-to-libmegapixels/</a></p> Making an USB Ethernet adapter work [SR9700]https://blog.brixit.nl/making-a-usb-ethernet-adapter-work-sr9700/86LinuxMartijn BraamSat, 28 Oct 2023 17:41:51 -0000<p>I just needed a simple USB to Ethernet adapter for testing. It does not need to be fast, it does not need to reach USB 3.0 speeds or gigabit speeds. I own two other USB ethernet adapters that have various reliability issues so I got a random cheap one from ebay.</p> <p>The adapter I ended up getting was the SR-QF9700 that does not have an actual brand on it. When plugging it into my laptop there was a slight issue though. It shows up as a CD drive...</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1698514364/image.png" class="kg-image"></figure> <p>This is one of those annoying features where USB dongles will present a CD drive with drivers to windows to make them work, ignoring that this is just an rndis device that has worked without special drivers <i><i>for decades</i></i>. This wouldn't be problematic if it exposed both the ethernet interface and the driver CD at the same time so I could just ignore it.</p> <p>In theory this can be made working by hacks such as <code>usb_modeswitch</code> but I've seen several forum posts online that mention it not working on this hardware. Even if that did solve the issue, it still makes this device useless for me because I don't want to mess with special software on every device I plug it into.</p> <p>I was ready to add this adapter to the e-waste problem but then I thought about how this would be handled from the manufacturing side. The chip is generic but those driver installers are usually branded. There needs to be somewhere to actually store the drivers on the device...</p> <h2>Taking it apart</h2> <p>This thing is very easy to open. There are no screws or clips in it at all, it's all held together by the sticker with the model number.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1698514385/20231028_0014.jpg" class="kg-image"></figure> <p>Inside the adapter is a simple PCB that has a crystal and the USB ethernet module. This particular adapter contains a CoreChips SR9700 single-chip ethernet controller. Doing anything interesting with this would be hard, there's not really public documentation for this chip except for the pin descriptions. The SR9900 datasheet mentions some internal one-time programmable memory, but it being one-time programmable does not really help me.</p> <p>So lets look at the back of the PCB</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1698514403/20231028_0006.jpg" class="kg-image"></figure> <p>Another chip! This is an 4MBit SPI Nor flash chip. This must be the chip actually storing the windows drivers.</p> <p>So what is the behavior of the ethernet controller chip if this chip is absent...</p> <p>SPI flash pinouts are pretty standardized, but I looked up the datasheet for this specific chip to be sure:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1698514414/image.png" class="kg-image"></figure> <p>There's many ways to make a SPI flash chip temporarily not function. I decided to jam a screwdriver between the <code>CS</code> and <code>SO</code> pin to make the chip unable to respond to SPI communication. With the screwdriver in place I plugged in the USB cable and behold:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1698514424/image.png" class="kg-image"></figure> <p>Ethernet! Well that was easy. Now to make this permanent I shorted together the same two pins with my soldering iron since removing the whole chip is way harder.</p> <p>So instead of messing with any device/computer I want to plug this ethernet adapter into I now have a normal USB ethernet adapter with just one extra solder connection :)</p> <p><br></p> Building a digital audio mixerhttps://blog.brixit.nl/building-a-digital-audio-mixer/83ElectronicsMartijn BraamWed, 04 Oct 2023 17:41:04 -0000<p>Digital audio mixers have always been the magical tech from the future when I started messing with audio mixers. Back then the cost of those mixers was extremely far out of reach for home use, into the thousands of euros.</p> <p>Prices for digital mixing has come down a lot, it's now possible to use tablets as control interface instead of having physical controls which drives down the price a lot. I've been using a borrowed Behringer X-air 18 for a while which is priced around €650. That mixer had to go back into production so now I'm mixer-less again.</p> <h2>Looking at the options</h2> <p>There's an annoying issue with audio mixers. To get a mixer that has a lot of features you also have to get a lot of channels. I used a lot of the routing and processing features of the X-air 18 but I only used 4 of the channels. If I switch to a smaller/cheaper version in the X-air series like the X-air 12 I still have way too many channels but I lose the multichannel usb audio interface part of it. Even the smallest version still takes up 2 rack units with features I don't need. </p> <p>Going to the competition of the X-air series doesn't help either. There's only significantly more expensive and larger options or more limited hardware. This leaves me once again with the last option: build it myself.</p> <h2>Teensy Audio Library</h2> <p>While looking at options for hardware with audio I/O I came across the <a href="https://www.pjrc.com/store/teensy41.html">Teensy 4.1</a>. This is a microcontroller board that has built-in 10/100 ethernet and has several digital audio interfaces. More importantly it already has a very nice library called the <a href="https://www.pjrc.com/teensy/td_libs_Audio.html">Teensy Audio Library</a> for creating digital audio pipelines with it.</p> <p>The hardware has a native USB interface that already has the libraries available to make the Teensy act like an USB audio interface. It also has two i2s/tdm interfaces for hooking up DAC/ADC chips. It is probably possible to create a 16-in 16-out mixer with this chip.</p> <p>So I ordered the Teensy and started creating the software. The audio pipeline is easily designed using the web editor for the audio library to generate the pipeline code:</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1696435005/image.png" class="kg-image"></figure> <p>This creates a basic audio mixer with 6 inputs and 4 outputs. Since I don't have any extra hardware yet the development is done purely using the stereo USB input and output.</p> <p>Every input channel has a <code>biquad</code> block which implements a parametric 4-band EQ. Then an <code>amp</code> block to do dynamics together with the detected signal level from the <code>rms</code> blocks. This creates a very simple channel strips with a compressor and EQ. </p> <p>The second part of the pipeline is a fully connected 6x4 matrix of mixers. This allows routing any of the inputs to any of the outputs with signal levels controlled by the faders on the mixer.</p> <p>After the mixer matrix there's <code>amp</code> blocks again for the output volume faders and <code>rms</code> blocks for the output VU meters.</p> <h2>Network control</h2> <p>So suprisingly the full DSP audio pipeline part was the easy part of the project. Just loading the generated code into the Teensy was enough to get audio running through the hardware after setting some default values in the blocks and hardcoding some values for the routing matrix. But a mixer you can't control or monitor is not a very useful mixer.</p> <p>I want to control the mixer over a network connection because I got used to being able to open the mixer interface on any PC here to control my audio routing, in my case the audio output of multiple computers is routed to the mixer to get a mix to my headphones.</p> <p>I also don't want to hardcode the control application I'd have to write for this mixer for this specific hardware so I have designed a network protocol that on connection describes the functionality and routing matrix to the control application and the control application dynamically creates a visual interface for it.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1696436026/image_6.png" class="kg-image"><figcaption>Prototype control application</figcaption></figure> <p>The control application is a Python GTK3 application and the Teensy side has a c++ class that implements this network protocol on top of the Teensy native ethernet controller. The network control code is fully separated from the audio code and with a few <code>#ifdef</code>s can probably be made more universal than that.</p> <p>The network library (I've called it Mixolydian) broadcasts the existance of the mixer using mDNS so the clients can discover it and show a nice list of detected hardware in the UI. It has a TCP protocol for controlling the mixer that allows multiple clients to be connected at the same time and seeing the changes in real time. It also has a separate UDP protocol for sending over real time audio meters.</p> <p>With this together there's now the components to make a digital audio mixer on any platform and add in the networking class to have a control application for it.</p> <h2>So why flexibility</h2> <p>The issue with making a 4 channel digital mixer is not everyone needs the same 4 channels. I need USB signals and multiple outputs. You might need 5 analog inputs instead and AES/EBU out only. So instead of making the exact mixer I need I made the opensource base to make any digital mixer. The protocol accounts for mixers that are significantly larger than what I made on a Teensy, the hard limit is 65535 connections in total (inputs and outputs combined).</p> <p>Since the Teensy has multiple i2s interfaces which are easily broken out into ribbon connectors and it has some extra SPDIF hardware it is possible to make a somewhat modular tiny digital mixer from this. The only thing you need is plugging in modules and connecting it together in firmware.</p> <p>There's a lot of options with this hardware. I want to put an 8x8 mixer into a 1U rack case with some led bar-graphs in the front. You can also make a 2-channel microphone interface in a desktop case with all hardware controls.</p> <h2>S/PDIF connectivity</h2> <p>So after I got USB signals running as a proof of concept I wanted to get some more audio channels to make it an actually useful audio mixer. I have ordered the Teensy audio shield which provides an unbalanced stereo input and output but while that is shipping through Europe I want to get some more development in.</p> <p>Then I remembered I have an old Behringer Ultramatch Pro in the dusty racks of decommissioned hardware. This is a stereo input/output DAC/ADC that has S/PDIF and AES/EBU connectivity in a single rack unit.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1696439160/20231004_0013.jpg" class="kg-image"><figcaption>The Behringer Ultramatch Pro SRC2496</figcaption></figure> <p>This is a nice unit for debugging since it has many blinkenlights for showing the status of the connection. Luckily getting the audio output from the mixer working through this was very easy. I just added the <code>spdif</code> output block to the audio graph and soldered a cable to pin 14 of the Teensy wired directly to the RCA input of the Ultramatch. Immediately I had audio running to my headphones plugged into the Ultramatch.</p> <p>The second part was getting the analog inputs of the Ultramatch wired as two mono input busses on the mixer so I can plug in microphones. The audio library has a nice input block called <code>spdif_async</code> that takes in any spdif signal and resamples it to match the clock of the Teensy audio pipeline.</p> <p>Sadly this wasn't as simple as just soldering down a wire to pin 15 and getting it working. It turns out that this pin expects a TTL level S/PDIF signal while the Ultramatch outputs a coax S/PDIF signal that is 0.5v peak-to-peak. After trying to breadboard together some converter circuits I saw that the AES output is supposed to be 5V peak-to-peak. I've touched the hot wire from an XLR cable to pin 15 of the Teensy and suddenly I had an S/PDIF lock and input signal in the mixer. This is without connecting up ground or the cold signal from the connection.</p> <p>I have now soldered down this single wire and taped everything down, it's wildly out of spec but works for development :)</p> <h2>Next steps</h2> <p>So short term there's a few things to do. Once I get the Teensy audio shield I'll be able to get audio I/O running using one of the i2s ports on the Teensy and I can put this all in one of those 1U rack project cases with nice connectors.</p> <p>To make this a bit more nice and integrated and professional I'm working on a PCB design that carries a Teensy module and exposes the audio connections on pinheaders and seperate PCBs for connecting CODEC chips to that for audio input and output and probably some 48V phantom power. I'm not sure yet of the design but the nice thing about opensource designs is you can always modify it to what you need.</p> <p>In the long term this needs to switch away from the really nice audio library to go beyond the 16bit 44.1KHz limitation of the library. The sample rate is not that problematic but clipping signals in a 16-bit integer audio pipeline is just way too easy.</p> <p>Hopefully this is the start of a small open source digital audio mixer ecosystem and some more people will build digital audio mixers :D</p> Squirrel Stringshttps://blog.brixit.nl/squirrel-strings/81Martijn BraamTue, 05 Sep 2023 15:00:35 -0000<p>Don't you sometimes wonder why a whole class of security vulnerabilities exist in programming languages? Some languages try to fix issues like memory safety vulnerabilities, but one issue that's encountered on The Web™ more often is injection vulnerabilities.</p> <div class="highlight"><pre><span></span><span class="n">username</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;username&quot;</span><span class="p">)</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">execute</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;SELECT * FROM users WHERE username=</span><span class="si">{</span><span class="n">username</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">):</span> <span class="n">frubnicate</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> </pre></div> <p>The possibility for injection vulnerabilities is everywhere. Most famously SQL injections where string templating or concatenation is used to build SQL commands. It's still very easy to create one of these vulnerabilities because the APIs are perfect for injection.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1693922039/image.png" class="kg-image"><figcaption>The PHP documentation for MySQL</figcaption></figure> <p>The only mitigation for this is warnings in the documentation and lets be fair, a lot of people won't read documentation.</p> <p>This is not only an issue for SQL. There's a lot of possibilities for injection like HTML injection in template engines making it possibly to perform XSS attacks. Or shell injection vulnerabilities in embedded devices.</p> <div class="highlight"><pre><span></span><span class="n">server</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;dns&quot;</span><span class="p">)</span> <span class="n">os</span><span class="o">.</span><span class="n">system</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;echo &quot;</span><span class="si">{</span><span class="n">server</span><span class="si">}</span><span class="s1">&quot; &gt; /etc/resolv.conf&#39;</span><span class="p">)</span> </pre></div> <p>For running commands it's safer and recommended to use the APIs that expect a list of arguments instead of a string, bypassing string concatenation. For HTML injection template engines usually protect the application by escaping variables fed into it.</p> <p>For SQL the solution is parameterized queries to seperate the actual SQL from the data supplied by users. But what if this distinction was a programming language feature instead?</p> <h2>Introducing squirrel strings</h2> <p>So why call it squirrel strings? Because naming things is hard and I thought of this for fixing SQL.</p> <p>A squirrel string is a string that is made from a string template but not finalized into an actual string. In this example I'll show how something like this would work in Python since there it can fit in quite nicely.</p> <div class="highlight"><pre><span></span><span class="n">username</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;username&quot;</span><span class="p">)</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="s2">&quot;SELECT * FROM users WHERE username=</span><span class="si">{username}</span><span class="s2">&quot;</span><span class="p">):</span> <span class="n">frubnicate</span><span class="p">(</span><span class="n">row</span><span class="p">)</span> </pre></div> <p>This is the exact same example as the worst-case example above but with a single letter changed. The f-string has become an s-string. Now it seems to be doing the exact same thing, but in a secure way. The major difference is hidden here in the <code>execute</code> method.</p> <p>A hypothetical squirrel string language construct would function the same as a Python f-string but would not return a plain string, instead it returns an object containing the literal part of the template and the data injected.</p> <div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">foo</span> <span class="o">=</span> <span class="s2">&quot;World&quot;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">demo</span> <span class="o">=</span> <span class="n">s</span><span class="s2">&quot;Hello </span><span class="si">{foo}</span><span class="s2">!&quot;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">demo</span> <span class="go">&lt;Squirrel &quot;Hello ?!&quot; [&quot;World&quot;]&gt;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">demo</span><span class="o">.</span><span class="n">template</span> <span class="go">[&quot;Hello &quot;, &quot;!&quot;]</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">demo</span><span class="o">.</span><span class="n">data</span> <span class="go">[&quot;World&quot;]</span> </pre></div> <p>For this to not to introduce weird edge-cases where the squirrel-string is used after changing the data the squirrel string should evaluate the expressions in the template before storing it in the squirrel object.</p> <div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">foo</span> <span class="o">=</span> <span class="n">s</span><span class="s2">&quot;1+2={1+2}&quot;</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">foo</span><span class="o">.</span><span class="n">template</span> <span class="go">[&quot;1+2=&quot;, &quot;&quot;]</span> <span class="gp">&gt;&gt;&gt; </span><span class="n">foo</span><span class="o">.</span><span class="n">data</span> <span class="go">[3]</span> </pre></div> <h2>An SQL query</h2> <p>So with this language feature it would be way easier to make a safer database API that accepts SQL strings. To ensure injection doesn't happen through regular strings type-checks can be used to check at runtime and type annotations can be used at "compile" time.</p> <div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="n">query</span><span class="p">:</span> <span class="n">Squirrel</span><span class="p">):</span> <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">Squirrel</span><span class="p">):</span> <span class="k">raise</span> <span class="n">SecurityError</span><span class="p">()</span> <span class="n">sql</span> <span class="o">=</span> <span class="s2">&quot;?&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">query</span><span class="o">.</span><span class="n">template</span><span class="p">)</span> <span class="k">return</span> <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">query</span><span class="o">.</span><span class="n">data</span><span class="p">)</span> <span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="s2">&quot;UPDATE users SET password=</span><span class="si">{password}</span><span class="s2"> WHERE id=</span><span class="si">{userid}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="c1"># Becomes UPDATE users SET password=? WHERE id=?</span> </pre></div> <p>With this in place it would be safe to template SQL strings because the insecure user data never becomes actually part of the query, which I think is a way better solution than making developers use more verbose APIs to ensure the safety.</p> <p>Another side benefit is that the type of the data placed into the template is preserved so numbers would not be passed as strings into the query. The constant part of the query could also be cached internally by the database API as a prepared statement without the developer having to use another API.</p> <h2>In the real world</h2> <p>So this is all a random shower thought. None of this is actually implemented and I might be missing some obvious flaw here that causes this to not work.</p> <p>The API for consuming squirrel-strings is something I thought up while writing the post and for certain use-cases a completely different API might be more useful. Maybe have the literal string as <code>"Hello !"</code> and a dict with data and offsets into the template. Maybe the api should stringlify all incoming data anyway.</p> <p>The examples also use Python here since it's relatively easy to fit this into Python. I don't know of any other programming language that has flags for string literals embedded into the syntax.</p> <p>But even if this is a perfect idea with no downsides, it wouldn't fix injection vulnerabilities immediately. Languages would have to create some syntax for defining squirrel strings. Libraries need to start accepting squirrel strings and at some point compatability with existing code would need to be broken, otherwise the "legacy" codepath would still be the default insecure one.</p> <p>But in any case I wanted to put the idea out there.</p> <h3>Update</h3> <p>It was <a href="https://fosstodon.org/@flyingpimonster/111013365525779342">pointed out to me</a> that this feature is almost fully in javascript already. Javascript has <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#tagged_templates">Tagged Templates</a> which allow something similar but not as an actual object, the string is instead converted in the arguments for a function.</p> <p>The neat thing is that does make it possible to implement this in javascript using that language feature:</p> <div class="highlight"><pre><span></span><span class="kd">function</span><span class="w"> </span><span class="nx">squirrel</span><span class="p">(</span><span class="nx">template</span><span class="p">,</span><span class="w"> </span><span class="p">...</span><span class="nx">args</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="c1">// This should probably be something more proper but</span><span class="w"></span> <span class="w"> </span><span class="c1">// I&#39;m not a JS programmer</span><span class="w"></span> <span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="nx">result</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="ow">new</span><span class="w"> </span><span class="nb">Object</span><span class="p">();</span><span class="w"></span> <span class="w"> </span><span class="nx">result</span><span class="p">.</span><span class="nx">template</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">template</span><span class="p">;</span><span class="w"></span> <span class="w"> </span><span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">args</span><span class="p">;</span><span class="w"></span> <span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="nx">result</span><span class="p">;</span><span class="w"></span> <span class="p">}</span><span class="w"></span> <span class="kd">let</span><span class="w"> </span><span class="nx">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">&quot;unsafe data&quot;</span><span class="p">;</span><span class="w"></span> <span class="kd">let</span><span class="w"> </span><span class="nx">example</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nx">squirrel</span><span class="sb">`SELECT * FROM people WHERE name=</span><span class="si">${</span><span class="nx">name</span><span class="si">}</span><span class="sb">`</span><span class="p">;</span><span class="w"></span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">example</span><span class="p">.</span><span class="nx">template</span><span class="p">)</span><span class="w"> </span><span class="c1">// [&#39;SELECT * FROM people WHERE name=&#39;, &#39;&#39;]</span><span class="w"></span> <span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">example</span><span class="p">.</span><span class="nx">data</span><span class="p">)</span><span class="w"> </span><span class="c1">// [&#39;unsafe data&#39;]</span><span class="w"></span> </pre></div> <p>This functionality is in fact used by some <a href="https://www.npmjs.com/package/sql-template-tag">Javascript libraries</a> already to run SQL queries.</p> Expensive cameras are actually betterhttps://blog.brixit.nl/expensive-cameras-are-actually-better/80PhotographyMartijn BraamWed, 16 Aug 2023 00:01:48 -0000<p>It has been quite a while since I've bought a "new" camera. A lot of my photography has been done on a Nikon D3300. This is a 2014 entry-level DSLR. While this camera is marked as entry level it has practically all the features you'd ever need for photography and I never really have had any issues with picture quality of this camera.</p> <p>Now the world is moving on to mirrorless cameras it has become almost affordable to buy a 10 year old second hand professional camera, so I snatched up the D750 right after Nikon introduced a new batch of Z-series mirrorless cameras. I've always wanted to use some full-frame cameras but the price difference has been absurd.</p> <p>When I bought the D3300 somewhere in ~2015 If I remember correctly I did a comparison between specs of a bunch of the current Nikon cameras and came to the conclusion that getting something more expensive as the entry level just wasn't worth it. A comparison of the models at the time:</p> <ul><li>The D3300 &quot;entry level&quot; for ~300 euro at the end of 2015.</li> <li>The D5300 &quot;upper entry&quot; for ~500 euro. This adds a tilting screen mostly.</li> <li>The D7200 &quot;enthusiast&quot; for ~1000 euro. Higher max ISO and faster shutter.</li> <li>The D610 &quot;high end&quot; for ~1500 euro. This one is the cheapest full frame model but was old-gen by that time already.</li> <li>The D750 &quot;high end&quot; for ~2000 euro. The same generation but full frame.</li> <li>The D810 &quot;professional&quot; for ~3000 euro. Takes both SD cards and Compact Flash</li> <li>The D4s &quot;flagship&quot; for ~6000 euro. Still only takes Compact Flash cards.</li> </ul> <p>So the most obvious spec in speclistings but the least important for 99% of the work, sensor resolution. Almost all these cameras are the same 24.7 megapixels resolution except for the D810 that is slightly newer and has 36 megapixels and the D4s that is a bit older and has 16 megapixels. </p> <p>A lot of specifications in webshop spec lists for these models are almost or exactly the same between these models, mainly because these are mainly based around the Expeed 4 SoC. When I chose the D3300 I was upgrading from a second hand D70 and my budget allowed only choosing between the two cheapest options. The main differentiator betwen those two models are that the screen tilts and it has a lot more autofocus points, I deemed the tilting feature completely useless on a DSLR because the main selling point for these cameras is that you have an optical viewfinder. Since I didn't need more autofocus points than the D70 I just got the cheapest one.</p> <p>The D3300 is a great camera, it has significantly better noise performance than the D70 which I practically had to keep below ISO 800 to don't get super noisy results.</p> <h2>Looking at some benchmarks</h2> <p>I'm not sure if this data was available yet when I was deciding between models almost a decade back, but there is an amazing camera comparison website called DXOMARK. It has a <a href="https://www.dxomark.com/Cameras/Compare/Side-by-side/Nikon-D750-versus-Nikon-D3300___975_928">spec comparison between the D3300 and the D750</a>. This gives a better picture of camera differences through actual measurements of the sensor.</p> <p>For some reason it's really hard or impossible to find which sensor is in which model camera which seems like an obvious spec to list since it's the most important component of the camera. Some of the major sensor differences between these two cameras:</p> <ul><li>The old one is a DX size sensor and the D750 is an FX camera. This has some upsides and downsides I&#x27;ll get at later.</li> <li>The bit depth is 12 vs 14 bit. And since since that&#x27;s not super intuitive, this means it captures 4x more levels, not 1.16x the amount.</li> <li>11 vs 51 autofocus points.</li> <li>Up to 5dB improvement in noise level at high-iso use. This is almost half the noise on max ISO.</li> <li>12.7dB vs 14.5dB of dynamic range at the minimum ISO. This means I can have a lot brighter highlights in my shot without the sensor clipping.</li> </ul> <p>After using the D750 for some weeks these differences turned out to make quite a bit of difference in which shot I could still make without things going noisy or blurry.</p> <h2>User experience differences</h2> <p> Another difference you won't easily see from spec sheets is the massive difference in UX.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1692136664/PXL_20230815_215628746.jpg" class="kg-image"><figcaption>The D750 on the left and the D3300 on the right</figcaption></figure> <p>On first glance the UI on the devices is pretty similar and on photos from camera reviews you'd think these behave pretty similar. It could not be more different.</p> <p>The D3300 turns on the screen for practically any action you do that doesn't require looking through the viewfinder. The bottom two rows are the quick-access settings for changing some of the main behaviors of the camera like autofocus/manual focus. The exposure and focus areas and the ISO. Some of the settings can be changed with holding a combination of hardware buttons and scrolling the thumbwheel but most of the configuration happens through this UI. The way you use those software quick-settings is with the d-pad to navigate through it.</p> <p>The D750 does seem to have roughly the same buttons on the back but the UI is missing the two bottom rows of settings. It's not only missing those two rows but it's actually not possible at all to change many of these settings using the d-pad and the screen. This pretty quickly forced me to learn all the hardware buttons which is probably a good thing since this is a lot faster to use.</p> <p>A few examples of the usage differences:</p> <ul><li>To change ISO on the D3300 I have to first move the camera out of the automatic modes using the mode dial on the top and then use the &quot;i&quot; button beside the screen to enter the actual quick settings menu. Then With the d-pad I move to the ISO option and with the &quot;ok&quot; button it opens a dialog to select the ISO with the d-pad buttons again. This does not allow changing between auto/manual ISO, that happens a few layers deep in the main menu.For the D750 I hold the button on the back that has &quot;iso&quot; above it and then I use the thumbwheel to scroll through the options. I for this action the back screen lights up quickly to show what I&#x27;m doing but there&#x27;s also a tiny ISO indicator in the viewfinder to see the options while I&#x27;m scrolling through them. The list of options include &quot;auto&quot; so no seperate menu path is needed.</li> <li>To change between autofocus and manual focus on the D3300 there&#x27;s another quick-setting menu that showing &quot;MF&quot; in the picture. Using the same procedure as the ISO option this can be navigated to switch between &quot;MF&quot;, &quot;AF-S&quot; and &quot;AF-C&quot;. The availability of these options is dependent on which picture mode the camera is in.On the D750 there&#x27;s an actual physical toggle on the side to switch between MF and AF very quickly. There&#x27;s a bunch more AF variants in this camera and also more focus point modes. These extra options can be accessed by pressing the button in the center of the toggle and using the thumbwheel the AF mode can be changed between continuous, single and auto. The second wheel on the front can change the focus point mode at the same time between single-point, all points, groups and 3d-tracking.</li> </ul> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1692137711/PXL_20230815_221346196.jpg" class="kg-image"></figure> <ul><li>Changing the shutter speed or aperture on the D3300 depends on the mode. Since there&#x27;s a single thumbwheel on this camera it will pick the &quot;main&quot; setting on the mode you&#x27;re in. On shutter-priority mode it will control the shutter and on apererture-priority mode it will control the aperture. In program or manual mode the wheel controls the shutter speed. To access the aperture setting the exposure compensation button has to be held downOn the D750 there&#x27;s two wheels so the rear one is always shutter speed and the front one is always aperture control.</li> <li>To change between release modes like single-photo, burst, timer and quiet mode the D3300 has the release mode button below the d-pad that pops up another menu to navigate.The D750 has a second ring below the mode selector dial that you can rotate to select these modes between Single, Continuous low-speed, Continuous high-speed, Quiet, Quiet-continuous, Timer and mirror-lock-up.</li> </ul> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1692138374/PXL_20230815_222446696.jpg" class="kg-image"></figure> <ul><li>The mode dial on the D750 has more &quot;professional&quot; options than the D3300. The D3300 mode dial is filled with magic useless modes like portrait, sport, landscape, baby?, macro and night-mode. These options are basically very slight variations on the auto program. It also has a dedicated position to open the in-camera manual for some reason.The D750 mode dial has all the special-auto modes under a single &quot;scene&quot; position so you can easily skip them and it adds the U1 and U2 positions instead. These are fully programmable positions that store the entire camera configuration. I have one of these programmed for focus trapping and one fixed ISO 100 and 1/160 shutter. Another small detail is that the mode dial has a locking button that prevents accidentally changing the mode.</li> </ul> <p>There is a lot more of these differences, practically all the settings you'd ever need are accessible using some button combination and the wheels and you can see the settings you're adjusting through the viewfinder to never lose sight of the subject of the picture. Some of these settings are especially nicely done and I really like how it spells out "3D" using the focus points when selecting the 3D focus tracking mode.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1692139159/PXL_20230815_223600021.jpg" class="kg-image"><figcaption>The D750 viewfinder when selecting the 3d focus mode</figcaption></figure> <p>Another difference isn't the added physical buttons but there's a whole second display, I was used to having this display on the D70 but the smaller entry cameras are sadly missing this.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1692139260/PXL_20230815_223453152.jpg" class="kg-image"><figcaption>The top LCD on the D750</figcaption></figure> <p>This top display is an old-school LCD with an optional backlight that can be triggered to read it in the dark. This picture shows basically all the settings you can read on this display but one of the nice features is that this is an always-on display. When the camera is off the amount of remaining shots is still visible (which is the 1.2k above) and when the SD card is removed it shows [-E-] instead. This makes it really quick to see if the camera is ready while it's still in my case.</p> <h2>Using the camera</h2> <p>There's quite a bit of things I noticed while using this camera the first few days. One of the first things that was obvious is how much better the autofocus system on this camera is. Not only is there a bit more granularity with the extra autofocus points the lenses also get focus a lot quicker.</p> <p>On the D3300 I mainly kept the camera in single-point single-push focus mode when possible because it relatively often picks a completely wrong autofocus point in the full-sensor mode. I also barely use the continuous focus modes because it does not super reliably pick the right moments to keep focussing and stopping it again when reframing shots.</p> <p>Another point where the D3300 focus is lacking quite a bit is subjects that are moving quickly. The 3d tracking autofocus on the D750 can actually keep up with a puppy charging towards you at full speed.</p> <p>Another big change is mainly due to switching to a much larger sensor. This changes the lens selection I use quite a bit. Since I've never planned to get a full-frame camera I never considered that as an option while buying lenses. My most used primes on the D3300 are the Nikor 35mm and 50mm and of those I mainly use the 50mm one. Due to the smaller sensor on the old camera it means that most of my photos are effectively shot on 75mm.</p> <p>At first using the lenses was quite weird. The 50mm lens on the D750 was behaving more like the 35mm lens on the D3300 which would have a 52mm effective focal length in that setup. But when putting the 35mm lens on the D750 I got... also 52mm. This is because my 50mm prime is a full frame lens and the 35mm one is a DX lens. The D750 detects this and automatically crops down the pictures in this case.</p> <p>This annoyed me for quite a bit and I considered getting a new prime in the 75-85mm range to get back my favorite focal length but then I realized that I could just use my 70-300mm telephoto lens as my "standard'" lens instead. While this would be quite unwieldy to use on the D3300 it works great on daily use of the D750 and since this is a lens with optical stabilization it helps even more with low light usage.</p> <p>I'm also quite happy that my most expensive lens, the 150-600mm happened to also be a full-frame lens. Dragging both the camera and this lens around is quite heavy but the results are amazing. With the improved autofocus it becomes possible to snap pictures of birds in flight even!</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1692140619/darktable.0PPA61_1.jpg" class="kg-image"><figcaption>This picture would&#x27;ve never been in focus on the D3300</figcaption></figure> <p>The lower effective focal length using the FX sensor on the lenses, combined with a lower noise floor has helped a massive amount with getting a lot of pictures successfully I would've never even attempted on the D3300 because I would have to push the ISO so high the results become useless or I would not be able to focus quickly and accurately enough to get the shot.</p> <h2>Astrophotography</h2> <p>I find astrophotography interesting, it's incredible what you're able to see using a camera pointed at the sky that you could never see with the naked eye. While I find it interesting I never gotten into it enough to actually pick gear specifically for astrophotography qualities. A few days ago I tried the D750 for astrophotography for the first time and the results have blown me away.</p> <p>One difference here is that the D3300 has a pentamirror and the D750 has a pentaprism. This is a part that sits between the lens and the eyepiece of the camera and is responsible for reflecting the image through the lens a few times so it lines up with your eye while looking through the eyepiece and the picture isn't mirrored.</p> <p>The pentamirror is the cheaper option which consists of a few mirrors glued together. This means that the light has to go through a few more glass to air transitions before it reaches your eyes which results in the image through the viewfinder being slightly darker. I had not noticed any difference really when using the D750 for normal photography but for astrophotography it made seeing stars through the viewfinder a lot easier.</p> <p>Another difference is that on the D750 the autofocus system is actually good enough to autofocus on stars which makes the whole process a lot less painful. </p> <p>And finally due to better low-light performance I was able to get way better results than my previous tries with the D3300. </p> <figure class="kg-card kg-image-card kg-width-wide"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1692143163/stack4_03.jpg" class="kg-image"><figcaption>The Andromeda Galaxy with the D750</figcaption></figure> <p>Here's an example of a stack of 30 pictures with an ISO of 12800 showing the Andromeda Galaxy. I would put a D3300 picture for comparison here but at this focal length and shutter duration it just doesn't get enough light to extract the data from.</p> <h2>Conclusion</h2> <p>I'm glad I've picked up this camera. I've long thought the pro cameras were just a few thousands of euro added price to get a second SD slot and a few more hardware buttons. The reviews of these cameras seems like a lot of cork sniffing but if you compare the entry and pro cameras there's actually some significant improvements. A lot of these improvements also are most likely artifical market segmentation since this has the same CPU in both models and a lot of user experience are just missing software features.</p> <p>There's some things that are suprisingly similar. One example is the video recording feature. The recording quality is practically the same on both cameras and the experience with anything relating to the live view is equaly horrible. These cameras are not for recording video. This also makes the tilting screen a useless feature.</p> <p>For most pictures the D3300 is absolutely fine and as bonus it's also a lot lighter and smaller. </p> Developers are lazy, thus Flatpakhttps://blog.brixit.nl/developers-are-lazy-thus-flatpak/76LinuxMartijn BraamSat, 03 Jun 2023 15:58:47 -0000<p>In the last decade I have seen a very slow but steady shift to solutions for packaging software that try to isolate the software from host systems to supposedly make things easier. My first experience with this was Docker, now Flatpak is the thing for desktop applications.</p> <h2>The promise of Flatpak</h2> <p>So the thing Flatpak is supposed to fix for me as developer is that I don't need to care about distributions anymore. I can bolt on whatever dependencies I want to my app and it's dealt with. I also don't need to worry about having software in distributions, if it's in Flatpak it's everywhere. Flatpak gives me that unified base to work on and everything will be perfect. World hunger will be solved. Finally peace on earth.</p> <p>Sadly there's reality. The reality is to get away from the evil distributions the Flatpak creators have made... another distribution. It is not a particularly good distribution, it doesn't have a decent package manager. It doesn't have a system that makes it easy to do packaging. The developer interface is painfully shoehorned into Github workflows and it adds all the downsides of containerisation.</p> <h3>Flatpak is a distribution</h3> <p>While the developers like to pretend real hard that Flatpak is not a distribution, it's still suspiciously close to one. It lacks a kernel and a few services and it lacks the standard Linux base directory specification but it's still a distribution you need to target. Instead of providing seperate packages with a package manager it provides a runtime that comes with a bunch of dependencies. Conveniently it also provides multiple runtimes to make sure there's not actually a single base to work on. Because sometimes you need Gnome libraries, sometimes you need KDE libraries. Since there's no package manager those will be in seperate runtimes.</p> <p>While Flatpak breaks most expectations of a distribution it's still a collection of software and libraries build together to make a system to run software in, thus it's a distribution. A really weird one.</p> <h3>No built in package manager</h3> <p>If you need a dependency that's not in the runtime there's no package manager to pull in that dependency. The solution is to also package the dependencies you need yourself and let the flatpak tooling build this into the flatpak of your application. So now instead of being the developer for your application you're also the maintainer of all the dependencies in this semi-distribution you're shipping under the disguise of an application. And one thing is for sure, I don't trust application developers to maintain dependencies.</p> <p>This gets really nuts by looking at some software that deals with multimedia. Lets look at the Audacity flatpak. It builds as dependency:</p> <ul><li>wxwidgets</li> <li>ffmpeg</li> <li>sqlite</li> <li>chrpath</li> <li>portaudio</li> <li>portmidi</li> </ul> <p>So lets look at how well dependencies are managed here. Since we're now almost exactly half a year into 2023 I'll look at the updates for the last 6 months and compare it to the same dependencies in Alpine Linux.</p> <ul><li>audacity has been updated 4 times in the flatpak. It has been updated 5 times on Alpine.</li> <li>ffmpeg has been updated to 6.0 in both the flatpak and Alpine, but the ffmpeg package has had 9 updates because if codecs that have been updated.</li> <li>sqlite hasn&#x27;t been updated in the flatpak and has been updated 4 times in Alpine</li> <li>wxwidgets hasn&#x27;t been updated in the flatpak and has been updated 2 times in Alpine</li> <li>chrpath hasn&#x27;t had updates</li> <li>portaudio hasn&#x27;t had updates in flatpak and Alpine.</li> <li>portmidi hasn&#x27;t had updates</li> </ul> <p>This is just a random package I picked and it already had a lot more maintainance of the dependencies than the flatpak has. It most likely doesn't scale to have all developers keep track of all the dependencies of all their software.</p> <h3>The idea of isolation</h3> <p>One of the big pros that's always mentioned with Flatpak is that the applications run in a sandbox. The idea is that this sandbox will shield you from all the evil applications can do so it's totally safe to trust random developers to push random Flatpaks. First of all this sandbox has the same issue any permission system that exists also has. It needs to tell the user about the specific holes that have been poked in the sandbox to make the application work in a way that end users <i>understand</i> what the security implications of those permissions are.</p> <p>For example here's Gnome Software ready to install the flatpak for Edge:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1685803130/image.png" class="kg-image"></figure> <p>I find the permission handleing implemented here very interesting. There's absolutely no warning whatsoever about the bypassed security in this Flatpak untill you scroll down. The install button will immediately install it without warning about all the bypassed sandboxing features.</p> <p>So if you <i>do scroll down there's more details right? Sure there is!</i></p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1685803278/image.png" class="kg-image"></figure> <p>There's a nice red triangle with the words Unsafe! pfew, everyone is fine now. So this uses a legacy windowing system which probably means it uses X11 which is not secure and breaks the sandbox. Well if that's the only security issue then it <i>might</i> be acceptable? Let's click that button.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1685803447/image.png" class="kg-image"></figure> <p>Well yeah... let's hide that from users. Of course the browser needs to write to /etc. This is all unimportant to end users.</p> <p>The even worse news is that since this is proprietary software it's not really possible to audit what this would do, and even if it's audited it's ridiculously easy to push a new more evil version to Flathub since practically only the first version of the app you push is thorougly looked at by the Flathub maintainers.</p> <p>Even if there weren't so many holes in the sandbox. This does not stop applications from doing more evil things that are not directly related to filesystem and daemon access. You want analytics on your users? Just requirest the internet permission and send off all the tracking data you want.</p> <h2>So what about traditional distributions</h2> <p>I've heard many argument for Flatpaks by users and developers but in the end I can't really say the pros outweigh the cons.</p> <p>I think it's very important that developers do not have the permissions to push whatever code they want to everyone under the disguise of a secure system. And that's <i>my opinion as a software developer</i>.</p> <p>Software packaged by distributions has at least some degree of scrutiny and it often results in at least making sure build flags are set to disable user tracking and such features.</p> <p>I also believe software in general is better if it's made with the expectation that it will run outside of Flatpak. It's not that hard to make sure you don't depend on bleeding edge versions of libraries while that's not needed. It's not that hard to have optional dependencies in software. It's not that hard to actually follow XDG specifications instead of hardcoding paths.</p> <h2>But packaging for distributions is hard</h2> <p>That's the best thing! Developers are not supposed to be the ones packaging software so it's not hard at all. It's not your task to get your software in all the distributions, if your software is useful to people it tends to get pulled in. I have software that's packaged in Alpine Linux, ALT Linux, Archlinux AUR, Debian, Devuan, Fedora, Gentoo, Kali, LiGurOS, Nix, OpenMandriva, postmarketOS, Raspbian, Rosa, Trisquel, Ubuntu and Void. I did not have to package most of this.</p> <p>The most I notice from other distributions packaging my software is patches from maintainers that improve the software, usually in dealing with some edge case I forgot with a hardcoded path somewhere.</p> <p>The most time I've ever spent on distribution packaging is actually the few pieces of software I've managed to push to Flathub. Dealing with differences between distributions is easy, dealing with differences between runing inside and outside Flatpak is hard.</p> <h2>But Flatpaks are easier for end users</h2> <p>I've ran into enough issues as end user of flatpaks. A package being on Flathub does not mean that it will be installable for an end user. I've ran into this by installing packages on the PineBook Pro which generated some rather confusing error messages about the repositories missing. It turns out that the Aarch64 architecture was missing for those flatpaks so the software was just not available. Linux distributions generally try to enable as much architectures as possible when packaging, not just x86_64.</p> <p>A second issue I've had on my Pinebook Pro is that it has a 64GB rootfs. Using too many flatpaks is just very wasteful of space. In theory you have a runtime that has your major dependencies and then a few Megabytes of stuff in your application flatpak. In practice I nearly have an unique platform per flatpak installed because the flatpaks depend on different versions of that platform or just on different platforms.</p> <p>Another issue is with end users of some of my Flatpaks. Flatpak does not deal well with software that communicates with actual hardware. A bunch of my software uses libusb to communicate with sepecific devices as a replacement for some Windows applications and Android apps I would otherwise need. The issue end users will run in to is that they first need to install the udev rules in their distribution to make sure Flatpak can access those USB devices. For the distribution packaged version of my software it Just Works(tm)</p> <h2>Flatpak does have it's uses</h2> <p>I wouldn't say Flatpak is completely useless. For certain usecases it is great to have available. It think Flatpak makes most sense for when closed source software would need to be distributed.</p> <p>I would like to see this be more strict though. I wouldn't want to have flatpaks with holes in the sandbox with a proprietary license for example. Which is exactly what the Edge flatpak is.</p> <p>It's quite sad that Flatpak madness has gone so deep into the Gnome ecosystem that it's now impossible to run the nice Gnome Builder IDE without having your application in a flatpak. (EDIT: Turns out that using Builder without Flatpak is possible again)</p> <p>I don't think having every app on a Linux machine being Flatpak is anything I'd want, If I wanted to give developers that much power to push updates to anywhere in my system without accountability I'd just go run Windows.</p> Can't get blinky on the BL602https://blog.brixit.nl/blinky-on-the-bl602/74ElectronicsMartijn BraamMon, 01 May 2023 16:46:41 -0000<p>One of the boards I have in my parts box is the PINE64 BL602 evaluation board. The "PineCone". It's a competitor to WiFi enabled microcontrollers like the Espressif products.</p> <p>I put it aside since when I received it there was not really any software to go along with it and I don't have anywhere near the experience to bring up a board like this.</p> <h2>Getting blinky running 3 years laters</h2> <p>This product has been out for 3 years now so lets see if it has formed a community.</p> <p>So if you are not familiar with blinky, it's the Hello World! of the microcontroller world. The goal is have a led connected to one pin of the board and toggle that pin on and off every second to make a blinking led.</p> <p>Let's check the website first. The official page for this from PINE64 itself is <a href="https://pine64.com/product/pinecone-bl602-evaluation-board/">https://pine64.com/product/pinecone-bl602-evaluation-board/</a>. Since the only official thing is the webshop which doesn't list any documentation or links to progress further on how to use this thing I consider it a dead end. Let's move on to the pine64.org side.</p> <p>The community side is the wiki page at <a href="https://wiki.pine64.org/wiki/PineCone">https://wiki.pine64.org/wiki/PineCone</a>. This has the specs again and a block diagram. It also has the datasheets for the components on the board and the schematics. Not anything I actually need to get started using this product.</p> <p>Let's compare to the Raspberry Pi Pico, a competitor that's doing quite well. When you search for this you land on the official product page <a href="https://www.raspberrypi.com/products/raspberry-pi-pico/">https://www.raspberrypi.com/products/raspberry-pi-pico/</a> that has the shop link, the specifications but more importantly it's a hub linking to the "Getting started" guide and the documentation for users.</p> <p>So lets scroll down on the PineCone wiki page since it's my only hope for docs. Thee quarters down this long wiki page is a list of flashing utilities. I don't have anything to flash yet but I look forward to figuring which of the 6 listed flashers fits my personality best /s.</p> <p>The rest of this page is links to random github repositories and blog articles. The top link being the link to the SDK for this board. It links to <a href="https://github.com/pine64/bl_iot_sdk">https://github.com/pine64/bl_iot_sdk</a> and is labeled as "compilers, linkers, and all the code to build on Windows, Linux (x86_64), and MacOS". The last activity here was 2 years ago. This seems mostly focussed on the reverse engineering effort but lets have a look.</p> <h2>bl_iot_sdk</h2> <p>This repository has a README with again more links to a lot of different places. One of these is linking to <a href="https://pine64.github.io/bl602-docs/">https://pine64.github.io/bl602-docs/</a> which is a sphinx generated documentation site. This is several pages too deep from the official pine64 website but I forgive that since it has the magical words I've been looking for:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682954202/image.png" class="kg-image"></figure> <p>So I'm going down this path, it almost seems familiar to the way the Pi Pico sdk is used. Clone the repository and create an environment variable pointing to the checkout folder.</p> <p>Next it tells me how to compile the example apps, but I don't want to compile the example apps, those probably work and they are in-tree so those probaby Just Work(tm) without setting up the scaffolding you should do for your own projects. I just want to build a main.c that does blinky. This part of the documentation sadly stops after compiling and flashing the built-in apps.</p> <p>I have continued browsing this documentation for a bit but it does not show at all how to make a minimal blinky example and build it. The rest of the documentation is mainly using the advanced features, various flashing systems and debugging systems and image formats.</p> <p>So to compare this with the Pi Pico documentation: The first thing I find when I read the documentation of the SDK is:</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682954861/image.png" class="kg-image"><figcaption>Blinky in the Pi Pico docs</figcaption></figure> <p>Chapter 1.2 is the first chapter with actual documentation and it starts with all the things I want for getting started with yet another microcontroller board:</p> <ul><li>A very simple .c file that implements blinky without using any advanced features of the hardware.</li> <li>The output of <code>tree</code> showing exactly which folder structure I should have and where the files go</li> <li>The CMakeLists.txt that builds this application</li> </ul> <p>The BL602 documentation has shown me none of the above yet. At this point I give up and just open 20 tabs in my browser with all the linked pages from the wiki.</p> <p>All these links are random flashing utilities, porting new programming languages and porting new frameworks. Also a lot of them showing how to flash the example apps to the thing.</p> <p>PEOPLE DON'T BUY MICROCONTROLLERS TO FLASH EXISTING EXAMPLES</p> <h2>Getting ANYTHING to run</h2> <p>So there is no documentation on how to make blinky for this platform whatsoever. I give up and figure it out myself from the sdk example apps.</p> <p>There's no blinky in the examples, the simplest application I could find was the helloworld one that outputs some text over uart. This uses FreeRTOS to get that working so there's a complete lack of basic single .c file demo applications for this.</p> <h2>More issues</h2> <p>This SDK has multiple things I don't like. The most important one is that it ships prebuilt compilers in the source tree. This is not a proper SDK, this is some Android-level build mess.</p> <p>The whole point of using nice RISC-V boards is that this is a standard architecture, all Linux distributions already ship a maintained RISC-V toolchain so why would I ever use some random precompiled binaries shipped by a Chinese vendor.</p> <p>Why is this using Makefiles to build the thing instead of cmake which the Raspberry Pi foundation have already proven you can integrate way neater. This is just throwing hardware over the wall and linking to the chip vendor documentation, hoping the community can match a proper made tutorial, proper documentation and make a new SDK that's not a mess.</p> <p>I guess I just dump this thing back in the parts bin and wait some years. Quite sad since this seems to be some really nice hardware and a great alternative to the existing microcontroller platforms.</p> Digital Aerochromehttps://blog.brixit.nl/digital-aerochrome/73Martijn BraamSun, 30 Apr 2023 11:11:55 -0000<p>A long long time ago in the 70s there was a lot of interesting film being made. At this time film was used for everything, professional and consumer cameras, movies, aerial photography. A lot of time has gone into recreating film looks for digital cameras now to reproduce the color response of the old film cameras.</p> <p>One of the more popular examples of this is the Fujifilm X-series digital cameras that have film simulations built in. I've heard they are quite good but those cameras are quite expensive and I find it a bit useless to get a seperate camera for just that trick.</p> <p>One legendary film stock you won't see simulated on anything like this is Kodak Aerochrome III Color Infrared. This film was originally produced for aerial photography. Most infrared film is black and white film that's also sensitive to infrared. What makes Aerochrome so special is that it's a color film that is infrared sensitive and uses false color to visualize the infrared spectrum.</p> <p>While it's not possible to get real Aerochrome anymore it is possible to get the datasheet for it. This old datasheet has a great explanation of why this film exists, how it works and has color spectrum sensitivity data.</p> <p>Color film is made from multiple layers of photosensitive material. Each layer has a different sensitivity spectrum and will produce a specific color when developed. For example one layer will be sensitive to red light and when developed will produce a cyan dye. This will create the negative image on the color negative. When this negative is inverted it the cyan will become the red part of the image again.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682776691/image.png" class="kg-image"><figcaption>Sensitivity spectrum of Kodak Ektar 100 color negative film</figcaption></figure> <p>But the color produced on the negative doesn't have to correlate with the original color that was recorded. When the spectrum of the sensitive layer doesn't correlate with the produced color dye it's called false color film. </p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682776972/image.png" class="kg-image"><figcaption>Sensitivity spectrum for Kodak Aerochrome III Infrared color negative film</figcaption></figure> <p>The scale for this datasheet is quite different because it has to include the infrared spectrum. One important thing to note is that the Aerochrome film is normally used in combination with a yellow filter on the lens, this will remove most of the blue light from the photograph. Then the false color is mapped so that infrared light will create cyan which inverts back to red. The red sensitive layer will produce magenta and develop to green. The green sensitive layer will produce yellow dye which produces blue on the final image. All the three layers are also sensitive to blue light which is why the filter is used together with this film.</p> <h2>Reproducing on a digital camera</h2> <p>The difficulty with reproducing this film is that cameras are designed to be not sensitive to infrared light. Even if the infrared light filter is removed from a camera the infrared light will just be detected as regular red light. This is not very useful because for the proper false color the visible red light should be shifted to green while the infrared should stay red. My solution to this is taking two pictures. One regular picture to capture the normal color spectrum and one with a R72 filter. This filter in front of the lens will let only light above 720nm pass which is exactly the part of the infrared spectrum I need.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682791970/filter.jpg" class="kg-image"></figure> <p>To the naked eye this filter just looks black since human eyes aren't sensitive to infrared light. On my camera it produces a way more interesting result though.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682792304/image.png" class="kg-image"></figure> <p>This is the infrared light stored into the red channel of the picture. Now the trick to make this looking like aerochrome is combining the R and G channels from the normal picture and the R channel from the picture with the IR filter. This is annoyingly hard to do correctly with most software, especially if you need to align the images.</p> <h2>Custom software</h2> <p>So aligning two images and some very easy color manipulation is something I had already written before in postprocessd. I made a copy of this software and changed the pipeline to align the IR image on top of the RGB image. Since this is using libraw you can just feed it any supported camera raw file.</p> <p>After stacking the images are combined into a single 4-channel RGBI image. Then a 4x3 matrix is run over this image to get the final 3-channel RGB output.</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682801215/image.png" class="kg-image"></figure> <p>Yep, doing it this way is probably overkill compared to scripting a series of imagemagick commands. But it does allow the flexibilty of doing more complicated transforms than just moving the color channels around. The datasheet says to use a yellow filter because all the layers are sensitive to blue light, so what happens if I modify this matrix to leak blue light into it?</p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682801557/compare.jpg" class="kg-image"><figcaption>Yellow filter on the left, No yellow filter on the right</figcaption></figure> <p>Mixing in the blue channel into the three other channels will lower the contrast of the image a bit, it also makes the sky a lot lighter again since... that was quite blue.</p> <p>Since I have this option I added the <code>-y</code> argument to the tool to disable the yellow filter.</p> <p>My software for this is available on <a href="https://git.sr.ht/~martijnbraam/aerochrome">https://git.sr.ht/~martijnbraam/aerochrome</a></p> <figure class="kg-card kg-image-card"><img src="https://blog.brixit.nl/image/w1000//static/files/blog.brixit.nl/1682802039/image.png" class="kg-image"></figure> <figure class="kg-card kg-gallery-card"><div class="kg-gallery-container"><div class="kg-gallery-row"><div class="kg-gallery-image"><img src="https://blog.brixit.nl/image/w600//static/files/blog.brixit.nl/1682853043/5.jpg" class="kg-image" width="600" height="451"></div><div class="kg-gallery-image"><img src="https://blog.brixit.nl/image/w600//static/files/blog.brixit.nl/1682853049/1.jpg" class="kg-image" width="600" height="451"></div><div class="kg-gallery-image"><img src="https://blog.brixit.nl/image/w600//static/files/blog.brixit.nl/1682853055/2y.jpg" class="kg-image" width="600" height="451"></div></div><div class="kg-gallery-row"><div class="kg-gallery-image"><img src="https://blog.brixit.nl/image/w600//static/files/blog.brixit.nl/1682853067/6.jpg" class="kg-image" width="600" height="451"></div><div class="kg-gallery-image"><img src="https://blog.brixit.nl/image/w600//static/files/blog.brixit.nl/1682853072/8.jpg" class="kg-image" width="600" height="451"></div><div class="kg-gallery-image"><img src="https://blog.brixit.nl/image/w600//static/files/blog.brixit.nl/1682853079/10.jpg" class="kg-image" width="600" height="451"></div></div></div></figure> <h2>Can this be done realtime?</h2> <p>One neat feature of the Fujifilm X-series cameras is that it gives a live preview of the film look while taking the picture. Since this process needs two pictures that would be impossible unless Fujifilm develops a sensor where IR cells are added to the bayer matrix.</p> <p>Another option is hacking up somthing yourself. This is probably doable with a raspberry pi 4 compute module and two sensors, one regular IMX219 sensor and one NOIR option with the visible light filter added. To make it always align this can be shot through a beamsplitter prism which can be very small with these sensor sizes.</p> <p></p>