BrixIT BlogRandom blog posts from Martijn Braam Blog, 01 Feb 2023 15:44:57 -000060Alpine Linux is pretty neat BraamWed, 01 Feb 2023 15:44:57 -0000<p>I've used various Linux distributions in the past, starting with a Knoppix live CD a long time ago. For a long time I was an Ubuntu user (with compiz-fusion ofcourse), then I used Arch Linux for years thinking it was the perfect distribution. Due to postmarketOS I found out about Alpine Linux and now after using that for some years I think I should write a post about it.</p> <h2>Installing</h2> <p>Ubuntu has the easy graphical installer of course. Installing Arch Linux the first time is quite an experience the first time. I believe Arch since has added a setup wizard now but I have not tried it.</p> <p>Installing Alpine Linux is done by booting a live cd into a shell and installing from there just like Arch but it provides the <code>setup-alpine</code> shell script that runs you through the installation steps. It's about as easy as using the Ubuntu installer if you can look past the fact that it's text on a black screen.</p> <p>A minimal Alpine installation is quite small, that combined with the fast package manager makes the install process really really quick.</p> <h2>Package management</h2> <p>The package management is always one of the big differentiators between distributions. Alpine has it's own package manager called APK, the Alpine Package Keeper. While it's now confused with the android .apk format it predates Android by two years.</p> <p>The package management is pretty similar to Archlinux in some aspects. The APKBUILD package format is very similar to the pkgbuild files in Arch and the packages support similar features. The larger difference is the packaging mentality: Archlinux prefers to never split packages, just one .pkg.tar.zst file that contains all the features of the application and all the docs and development headers. Alpine splits out all these things to subpackages and the build system warns when the main package contains any documentation or development files.</p> <p>For a minimal example of this let's compare the tiff library. In Alpine Linux this is split into 5 packages:</p> <ul><li><code>tiff</code>, the main package that contains [460 kB]</li> <li><code>tiff-dev</code>, the development headers [144 kB]</li> <li><code>libtiffxx</code>, the c++ bindings [28 kB]</li> <li><code>tiff-doc</code>, the documentation files [5.21 MB]</li> <li><code>tiff-tools</code>, command line tools like ppm2tiff [544 kB]</li> </ul> <p>In Arch Linux this is a single package called <code>libtiff</code> that's 6.2 MB. For most Linux users you'd never need the library documentation which takes the most space in this example.</p> <p>The end result is that my Archlinux installations are using around 10x the disk space my Alpine installations use if I ignore the home directories.</p> <p>Some more differences are that Alpine provides stable releases on top of the rolling <code>edge</code> release branch. This improves reliablity a lot for my machines. You wouldn't normally put Arch Linux on a production server but I found Alpine to be almost perfect for that usecase. Things like the <code>/etc/apk/world</code> file makes management machines easier. It's basically the <code>requirements.txt</code> file for your Linux installation and you don't even need to use any extra configuration management tools to get that functionality.</p> <p>There's also some downsides to <code>apk</code> though. Things I'm missing is optional packages and when things go wrong it has some of the most useless error messages I've encountered in software: <code><code>temporary error (try again later)</code></code> . Throwing away the original error and showing "user friendly" messages usually does not improve the situation.</p> <h2>Glibc or not to glibc</h2> <p>One of the main "issues" that get raised with Alpine is that it does not use glibc. Alpine Linux is a musl-libc based distribution. In practice I don't have many problems with this since most my software is just packaged by in the distribution so I wouldn't ever see that it's a musl distribution. </p> <p>Issues appear mostly when trying to run proprietary software on top of Alpine or software that's so hard to build that you're in practice just getting the prebuilds. The solution to proprietary software is... don't use proprietary software :)</p> <p>For the cases where that's not possible there's always either Flatpak or making a chroot with a glibc distribution in it.</p> <h2>Systemd</h2> <p>Beside not using glibc there's also no systemd in Alpine. This is one of the things I miss the most actually. I don't enjoy the enormous amount of different "best practices" for specifying init scripts and the bad documentation surrounding it. So far by best solution for creating idiomatic init scripts for alpine is just submitting something to the repository and wait until someone complains about style issues.</p> <p>Beside that I'm pretty happy with the tools openrc provides for manageing services. The <code>rc-update</code> tool gives a nice consise overview of enabled boot services and the <code>service</code> tool just does what I expect. It seems like software is starting to depend on systemd restarting it instead of fixing memory leaks which causes me some issues sometimes.</p> <h2>Conclusion</h2> <p>Alpine Linux is neato. I try to use it everywhere I can.</p> Rewriting my blog again BraamMon, 05 Dec 2022 18:17:29 -0000<p>For quite a while my blog has run on top of Ghost. Ghost is a blog hosting system built on top of nodejs. It provides a nice admin backend for editing your posts and it comes with a default theme that's good enough to not care about theming.</p> <p>My blog has been through some rewrites, from Drupal to Jekyll to various PHP based CMS platforms and finally it ended up on Ghost. I like Ghost quite a lot but it has a few flaws that made me decide to replace it.</p> <ul><li>It&#x27;s nodejs. I know a lot of people like javascript for some reason but it&#x27;s quite horrific for system administration. I do not like having to add third party repositories to my system to get a recent enough nodejs build to host a simple website. I&#x27;m pretty sure this dependency on the latest and greatest is the main reason why everyone wants to stuff their project in docker and not care about the dependencies.</li> <li>It only officially supports Ubuntu to install on (aside from Docker). It was already annoying to get it installed on Debian but now I&#x27;ve moved over most of my servers to Alpine this has become even more of a pain. There&#x27;s no reason this has to integrate so deeply with the distribution that it matters what I&#x27;m running.</li> <li>It depends on systemd, this is largely part of the Ubuntu dependency above and not needing to implement something else. But from running Ghost in openrc I&#x27;ve noticed that it depends on the automatic restarts done by systemd when the nodejs process crashes.</li> <li>I&#x27;m just running a simple blog which is fully static content. this should not require running a permanent application server for availability of the blog and the dependency on Mysql is also very much overkill for something that&#x27;s so read-heavy.</li> <li>I don&#x27;t particularly like the templating system for adjusting the theme. I&#x27;ve been running a fork of the casper default theme for a while that just rips out the javascript files. This breaks almost nothing on the default theme except the infinite scroll on the homepage.</li> </ul> <h2>Switching to something different</h2> <p>I could try to switch to yet another blogging platform. The issue is that I really really like writing posts in the Ghost admin backend. With a casual look at the browser dev console another solution became obvious...</p> <p>The whole Ghost admin backend is neatly packaged into four files:</p> <ul><li><code>ghost.min.js</code></li> <li><code>vendor.min.js</code></li> <li><code>ghost.min.css</code></li> <li><code>vendor.min.css</code></li> </ul> <p>Since this is a fully standalone client side webapp thing I can just host those four files and then write a Python Flask application that implements the same REST api that the nodejs backend provided.</p> <p>This is exactly what I did, the whole backend is now a small Flask application that implements the Ghost API in ~500 lines of Python. It's called Spook (since this is dutch for ghost).</p> <p>This also uses an SQLite database as storage backend for the whole system. This Flask application also does not implement any of the website rendering. It just stores the state of the Ghost backend and does media uploads. To actually get a website from this it implements a hook script that gets called whenever anything on the website changes. With that it's possible to use any static site generator to generate a static website from this dataset.</p> <h2>Reimplementing casper</h2> <p>As static site generator I used a second Flask webapplication. This one uses the Flask-Frozen module to generate a static website, this is why this module has the name <code>frozen_casper</code>. This generator reads the SQLite database from Spook and generates static HTML compatible with the stylesheet from the stock casper theme from Ghost. It also generates an identical RSS feed so the RSS subscribers can be migrated over with identical post unique ids.</p> <p>I did make some modifications to the generated html like implementing very basic paging on the posts list pages instead of relying on javascript infinite scrolling.</p> <p>The <code>spook</code> and <code>frozen_casper</code> module together replaces a complete Ubuntu Server install, a mysql instance I had to keep giving RAM and a crashy nodejs webapplication. Since today this new system is hosting this blog :)</p> Taking a good picture of a PCB BraamSun, 27 Nov 2022 10:49:01 -0000<p>Pictures of boards are everywhere when you work in IT. Lots of computer components come as (partially) bare PCBs and single board computers are also a popular target. Taking a clear picture of a PCB is not trivial though. I suspect a lot of these product pictures are actually 3d renders.</p> <p>While updating <a href="">hackerboards</a> I noticed not all boards have great pictures available. I have some of them laying around and I have a camera... how hard could it be?</p> <p>Definitely the worst picture is the one in the header above. Taken with a phone at an angle with the flash in bad lighting conditions. I've taken quite a bunch of pictures of PINE64 boards, like some of the pictures in the header of the pine64 subreddit and the picture in the sidebar. I've had mixed results with taking the pictures but the best results I've had with taking board pictures is using an external flash unit. </p> <h2>The ideal setup</h2> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>So to create a great picture I've decided to make a better setup. I've used several components for this. The most important one is two external flashes controlled with a wireless transmitter. I've added softboxes to the flashers to minimize the sharp shadows usually created when using a flash. This produces quite nice board pictures with even lighting. </p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>All dust looks 1000x worse on pictures :(</figcaption></figure> <p>For all the pictures from this setup I've used a 50mm macro lens. Not only is it great for getting detail pictures of the small components, it's also the lens with the least distortion I have. Having less distortion in the lens is required to have a sharp picture all the way to the edges of the board and not have the edges of the board look curved. The curvature can be fixed in software but the focal plane not being straight to the corners can't be fixed.</p> <p>It's possible to get even less shadows on the board by using something like a ring light, while this gets slightly more clarity I also find this just makes the pictures less aesthetically pleasing.</p> <p>So how to deal with the edges of the board? For a lot of website pictures you'd want a white background. I have done this by just using a sheet of paper and cleaning up the background using photo editing software. This is quite time consuming though. The usual issues with this is that the background is white but not perfectly clipped to 100% pure white in the resulting picture. There's also the issue of the board itself casting a slight shadow.</p> <p>I took my solution for this from my 3D rendering knowledge (which is not much). You can't have a shadow on an emitter. To do this in the real world I used a lightbox.</p> <p>Lightboxes are normally for tracing pictures and are quite easy to get. It doesn't give me a perfectly white background but it gets rid of the shadows at least.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>To get this from good to perfect there's another trick though. If I take a picture without the flashes turned on but everything else on the same settings I get a terribly underexposed picture... except for the background.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>You never notice how many holes there are in a PCB until you put it on a lightbox</figcaption></figure> <p>All I need to do to get a clean background is increasing the contrast of this picture to get a perfect mask. Then in gimp I can just overlay this on the picture with the layer mode set to lighten only.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>The final composite</figcaption></figure> <p>It's also possible to use the mask picture as the alpha channel for the color picture instead. This works great if there's a light background on the website, it shows the flaws though when the website has a dark background.</p> <p>Let's create the worst-case scenario and use a pure black background:</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>Now edges are visible on the cutouts. Due to the mismatch in light color temperature with the light box the edges are also blue here. A lot of edges can be fixed by running the dilate filter in gimp on the mask layer to make the mask crop into the board by one pixel. it makes the holes in the board too large though. To get this perfect manual touchup is still required.</p> <h2>Automating it further</h2> <p>Now the input data is perfect enough that I can make the cutout with a few steps in gimp it's also possible to automate this further with the magic of ImageMagic.</p> <pre><code>$ convert board.jpg \( mask.jpg \ -colorspace gray \ -negate \ -brightness-contrast 0x20 \ \) \ -compose copy-opacity \ -composite board.png</code></pre> <p>This loads the normal picture from board.jpg and the backlit picture as mask.jpg and composites them together into a .png with transparency.</p> <p>But it can be automated even further! I still have a bit of camera shake from manually touching the shutter button on the camera and I need to remember to take both pictures every time I slightly nudge the device I'm taking a picture of.</p> <p>The camera I'm using here is the Panasonic Lumix GX7. One of the features of this camera is the built-in wifi. Using this wifi connection it's possible to use the atrocious Android application to take pictures and change a few settings.</p> <p>After a bit of reverse engineering I managed to create a <a href="">Python module</a> for communicating with this camera. Now I can just script these actions:</p> <pre><code>import time from remotecamera.lumix import Lumix # My camera has a static DHCP lease camera = Lumix(&quot;;) camera.init() camera.change_setting(&#x27;flash&#x27;, &#x27;forcedflashon&#x27;) camera.capture() time.sleep(1) camera.change_setting(&#x27;flash&#x27;, &#x27;forcedflashoff&#x27;) camera.capture()</code></pre> <p>Now I can just run the script and it will take the two pictures I need. It's probably also possible to fetch the images over wifi and automatically trigger the compositing but it sadly requires changing the wifi mode on the camera itself between remote control and file transfer.</p> <h2>Not just for PCBs</h2> <p>This setup is not only useful for PCB pictures. This is pretty great for any product picture where the product fits on the lightbox.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>Here's the composite of a SHIFT 6mq. Pictures of the screen itself are still difficult due to the pixel pattern interfering with the pixels of the camera sensor and the display reflecting the light of the flashes. This can probably be partially fixed once I get a polarizing filter that fits this lens.</p> Finding an SBC BraamTue, 15 Nov 2022 23:24:56 -0000<p>A long long time ago on a server far away there was a website called Board-DB. This website made in 2014 was a list of single board computers that became popular after the whole Raspberry Pi thing happened.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Board-DB in 2014</figcaption></figure> <p>The website itself was pretty simple, just a list of devices and a simple search box with a few filters. The magic was in the well maintained dataset behind it though. After some time this website seemed to have gone away and I had not thought about it for quite a bit until I happened to learn that the owner of <a href=""></a> originally ran the Board-DB website.</p> <h2>Rewriting</h2> <p>So a plan formed together with the old owner of the website. A good board search website is still after all these years a gap in the market. There's some websites that have these features but I keep running into missing filters. One of the things that annoy me the most about webshops and comparison sites is a lack of good filters and then including the most useless ones like a what color the product is.</p> <p>To redesign this I wanted to go to the other extreme. If you compare two boards and there's not a single different listed spec, then something is missing. The schema for the rewritten database has 165 columns now and I still found some things that are missing in this dataset like detailed PCI-e data.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>The 2022 version of the Board-DB website</figcaption></figure> <p>This new website was written in a timespan of 10 months in some spare time together with Raffaele. At the time of writing it contains data about 466 boards. Not all data is perfect yet since this is a huge research and cleanup job but the website is certainly useful already. As the data in the backend gets more cleaned up more filters in the board list can be added to drill down on the specs you need.</p> <h2>Architecture</h2> <p>The website is a Python Flask webapplication that uses SQLite as storage backend. SQLite is perfect in this case since it's mostly a read-heavy website. The website is designed from the start to run behind a caching reverse proxy like Varnish and makes sure that all the URLs in the site generate the exact same content unless boards are changed.</p> <p>The site is also designed to work without having javascript enabled but does use javascript to improve the experience. It even works in text-mode browsers. Due to it not using any javascript or css frameworks the website is also very light.</p> <h2>The dataset</h2> <p>At the start of the project I thought implementing the facetted search would be a large technical problem to solve but it that turned out to be relatively easy. By far hardest part of this system is adding and cleaning up the data. The original dataset is imported from a huge spreadsheet that with a bunch of heuristics gets converted into more specific data columns in the SQLite database that runs the website. The original dataset did not separate out specific details like which cores a SoC has and how many and what speed the ethernet ports were.</p> <p>Parts of this data was recoverable by searching for keywords in the original description fields in the spreadsheet and other data is manually fixed up with some bulk editing tools.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>One of the hacks in the CSV importer</figcaption></figure> <p>This all ends up getting rendered nicely again on the board detail page</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>I'm quite happy with how the web frontend turned out. It has a decently detailed filter list, an usable spec listing page and a board comparison feature. Getting the project this far would've been impossible without the help of Raffaele.</p> <p>The website is now online on <a href=""></a> </p> Automated Phone Testing pt.5 setupMartijn BraamTue, 25 Oct 2022 00:55:14 -0000<p>Now I've written all the parts for the Phone Test Setup in the previous parts, it's time to make the first deployment.</p> <p>For the postmarketOS deployment I'm running the central controller and mosquitto in a container on the postmarketOS server. This will communicate with a low-power server in a test rack in my office. The controller hardware is an old passively cooled AMD board in a 1U rack case. I sadly don't have any part numbers for the rack case since I got a few of these cases second hand with VIA EPIA boards in them.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>Ignore the Arduino on the left, that&#x27;s for the extra blinkenlights in the case that are not needed for this setup</figcaption></figure> <p>The specs for my controller machine are:</p> <ul><li>AMD A4-5000 APU</li> <li>4GB DDR3-1600 memory</li> <li>250GB SSD</li> <li>PicoPSU</li> </ul> <p>The specifications for the controller machine are not super important, the controller software does not do any cpu-heavy or memory-heavy tasks. The important thing is that it has some cache space for unpacking downloaded OS images and it needs to have reliable USB and ethernet. For a small setup a Raspberry Pi would be enough for example.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Ignore the Aperture sticker, this case is from a previous project</figcaption></figure> <p>This server now sits in my temporary test rack for development. This rack will hold the controller PC and one case of test devices. This rack case can hold 8 phones in total by using 4 rack units.</p> <h2>Deploying the software</h2> <p>After running all the software for the test setup on my laptop for months I now started installing the components on the final hardware. This is also a great moment to fix up all the installation documentation.</p> <p>I spend about a day dealing with new bugs I found while deploying the software. I found a few hardcoded values that had to be replaced with actual configuration and found a few places where error logging needed to be improved a lot. One thing that also took a bit of time is setting up Mosquitto behind an Nginx reverse proxy.</p> <p>The MQTT protocol normally runs on plain TCP on port 1883 but since this involves sending login credentials its better to use TLS instead. The Mosquitto daemon can handle TLS itself and with some extra certificates will run on port 8883. This has the downside that Mosquitto needs to have access to the certificates for the domain and it needs to be restarted after certbot does its thing.</p> <p>Since The TLS for the webapplication is already handled by Nginx running in reverse proxy mode it's easier to just set up Nginx to do reverse proxying for a plain TCP connection. This is the config that I ended up with:</p> <div class="highlight"><pre><span></span><span class="k">stream</span><span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="kn">upstream</span><span class="w"> </span><span class="s">mqtt_servers</span><span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="kn">server</span><span class="w"> </span><span class="n"></span><span class="p">:</span><span class="mi">1883</span><span class="p">;</span><span class="w"></span> <span class="w"> </span><span class="p">}</span><span class="w"></span> <span class="w"> </span><span class="kn">server</span><span class="w"> </span><span class="p">{</span><span class="w"></span> <span class="w"> </span><span class="kn">listen</span><span class="w"> </span><span class="mi">8883</span><span class="w"> </span><span class="s">ssl</span><span class="p">;</span><span class="w"></span> <span class="w"> </span><span class="kn">proxy_pass</span><span class="w"> </span><span class="s">mqtt_servers</span><span class="p">;</span><span class="w"></span> <span class="w"> </span><span class="kn">proxy_connect_timeout</span><span class="w"> </span><span class="s">1s</span><span class="p">;</span><span class="w"></span> <span class="w"> </span> <span class="w"> </span><span class="kn">ssl_certificate</span><span class="w"> </span><span class="s">/etc/letsencrypt/.../fullchain.pem</span><span class="p">;</span><span class="w"> </span> <span class="w"> </span><span class="kn">ssl_certificate_key</span><span class="w"> </span><span class="s">/etc/letsencrypt/.../privkey.pem</span><span class="p">;</span><span class="w"></span> <span class="w"> </span><span class="kn">ssl_dhparam</span><span class="w"> </span><span class="s">/etc/letsencrypt/ssl-dhparams.pem</span><span class="p">;</span><span class="w"></span> <span class="w"> </span><span class="p">}</span><span class="w"></span> <span class="p">}</span><span class="w"></span> </pre></div> <p>Another thing that had to be done for the deployment is writing actual service files for the components. The init files now sets up openrc in my Alpine installations to supervise the components, deal with logging and make sure the database schema migrations are run on restart.</p> <h2>Putting together the phone case</h2> <p>To neatly store the phones for the test setup I decided to use a 2U rack case since that's just high enough to store modern phones sideways. For this I'm using a generic 2U rackmount project box with the very easy to remember product code G17082UBK. This is a very cheap plastic case with an actual datasheet.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>All the internal and external dimensions for this case are documented</figcaption></figure> <p>I used this documentation to design a tray that fits between the screw posts in this case. The tray is printed in three identical parts and each tray has three slots. I use dovetail mounts to have the phone holders slide on this tray.</p> <p>All this is designed using OpenSCAD. I never liked 3D modelling software but with this it's more like programming the shapes you want. This appeals a lot to me since... I'm a software developer.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The tray design in OpenSCAD</figcaption></figure> <p>From this design I can generate an .stl file and send it to the 3D printer. The tray can print without any supports and takes about an hour on my printer. So 3 hours later I have the full base of the phone holder in my rack case.</p> <p>To actually mount the phones there can be phone-specific models that grip into this baseplate and hold the phone and extra electronics. I made a somewhat generic phone mount that just needs two measurements entered to get the correct device thickness at two points. This is the holder I'm currently using for the PinePhone and the Oneplus 6.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>The baseplates here are in blue and the phone holder is the green print. This version of the phone holder is designed to hold the Raspberry Pi Pico and has a ring for managing the soldered cables to the device. The size of the PinePhone is about the largest this case can hold. It will fill up the full depth of the case when the USB-C cable is inserted and it also almost hits the top of the case.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>The PinePhone and Oneplus 6 in the test rack case</figcaption></figure> <p>In this case I can hold 8 phones on the trays and have one of the slots on the tray left over to hold a future USB hub board that will have the 16 required USB ports to use all the devices in the case.</p> <p>For small desk setups the a single tray is also pretty nice to hold devices and the phone holder itself will also stand on its own. This is great if you have one to three devices you want to hook up to your laptop for local development.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>The tray is a bit flimsy without being screwed in the rack case but holds up phones great</figcaption></figure> <h2>Running the first test job</h2> <p>So now the case is together and the controller is deployed it's time to run an actual test. For this first test I'll be using Jumpdrive as the test image. This is by far the smallest image available for the PinePhone which makes testing a lot easier. It just boots a minimal kernel and initramfs and the greatest feature for this test: it spawns a shell on the serial console.</p> <p>Since GitHub is not a great hosting platform and the bandwidth limit for the <a href=""></a> repository has been reached it's not possible to fetch the raw file from github without being logged in so this script uses my own mirror of the latest PinePhone jumpdrive image.</p> <pre><code>devicetype: pine64-pinephone boot { rootfs: } shell { prompt: / # success: Linux script { ip a uname -a } }</code></pre> <p>This will boot Jumpdrive and then on the root console that's available over the serial port it will run <code>ip a</code> to show the networking info and then <code>uname -a</code> to get the kernel info. Because the success condition is <code>Linux</code> it will mark the job successful if the uname output is printed.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>The serial output also shows another feature of the controller: it will set the environment before executing the script, which in this case is just the <code>CI</code> variable and it will add <code>|| echo "PTS-FAIL"</code> to the commands so non-zero exit codes of the commands can be detected. When <code>PTS-FAIL</code> is in the serial output the task will be marked as failed. Using the <code>success:</code> and <code>fail:</code> variables in the script the success and failure text can be set.</p> <p>With these building blocks for test scripts it's now possible to implement a wide variety of test cases. Due to Jumpdrive not having enough networking features enabled in the kernel it's not yet possible to upgrade from the serial connection to a telnet connection in the test setup, this would make the test cases a bit more reliable since there's a very real possibility that a few bits are flipped right in the success or failure string for the testjob marking a successful job as failed due to timeouts.</p> <p>Getting a generic test initramfs to combine with a kernel to do basic testing will be a good thing to figure out for part 6, Jumpdrive only has busybox utilities available and only has very limited platform support in the first place. </p> Trying Plasma Desktop again BraamTue, 18 Oct 2022 23:51:04 -0000<p>So I'm trying KDE Plasma again, I hear 5.26 has many great improvements and the last time I ran KDE for more than a day was in 2014. I mainly run Gnome and Phosh and Sway on my devices and I feel like I don't use KDE enough for the amount of times I complain about it.</p> <p>So I decided to put postmarketOS Plasma Desktop on my PineBook Pro. Mainly because one of my issues with KDE has been performance. I know that Gnome runs on the edge of smooth on this hardware so it's easy to compare if KDE will be slower or faster for me there. Testing on faster hardware would only hide performance issues.</p> <h2>Installation</h2> <p>So I installed Plasma with the postmarketOS installer on the Pinebook Pro, I don't have an nvme drive in this device, just running it from the stock 64GB eMMC module. </p> <p>The only installation issue I had was the disk not resizing on first boot to fill the full eMMMC size, but that's a postmarketOS bug I need to fix, not a KDE issue.</p> <hr> <p>So that continues me writing the rest of this blog post from my Plasma installation :)</p> <h2>First use</h2> <p>My issue with KDE Mostly is that it has many papercuts that I just don't have to deal with in the Gnome ecosystem. I don't want to make this sound like a big hate post on KDE so here's some good things about it first:</p> <ul><li>I like how the default theme looks in light and dark mode, I&#x27;ll probably will swap out the cursor theme since it doesn&#x27;t really fit but the rest looks nice and consistent</li> <li>The notification system is quite an improvement. Gnome has the annoying tendency to pop up the notification right at the place where I&#x27;m working: the top center of the screen. If I get a few chat messages I have to manually click a few stacked notifications away unless I want to wait a few minutes for them to time out. The KDE notifications pop up in the bottom right corner and move out of the way, also the visual timeout indicator is great to have.</li> <li>I like the set of stock wallpapers in Plasma a lot.</li> <li>The default application launcher looks way more polished than I remember. Also I&#x27;m always happy when application launchers just pop up when you tap the meta key, which is still not a feature in all desktop environments.</li> <li>That lockscreen fade in animation is nice</li> <li>The performance has improved quite a lot. I feel like the performance of Plasma on this hardware was absolutely painful a year back when I ran it to get some photos and now it feels quite smooth in mosts places.</li> </ul> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Nice notification, ignore inconsistent padding</figcaption></figure> <p>Not everything is perfect but I'm quite pleased with how this is running. I know a lot of KDE is about everything being customizable, but the thing I like with Gnome is that it's already setup the way I want out of the box. Well mostly... For some reason on every desktop I install I have to re-add the ctrl+alt+t keybind to open a terminal. If I remember correctly this was a thing in older Ubuntu installations and nobody ships this keybind anymore.</p> <p>Now for the papercuts I hit in the first hour:</p> <ul><li>The application launcher is slow to pop up. Feels like this should just always be there. But I have the feeling that a lot of animations have a half second delay before they start animating.</li> <li>The terminal has way too many buttons and toolbars by default, luckily easy to remove again.</li> <li>meta+up doesn&#x27;t fullscreen the application, it moves it to the upper half of the screen instead. All the other environments I use fullscreen using this keyboard combination.</li> <li>partitionmanager doesn&#x27;t seem to have a function for &quot;extend partition in all the free space after it&quot;, instead I have to calculate the new size. Not sure if this counts as an KDE issue or third party applications though :)</li> <li>My wifi does not reconnect on reboot for me, but I believe this is a postmarketOS bug.</li> <li>The large &quot;peek at desktop&quot; button in the taskbar is pretty useless and way too big for its function. Windows at least made that only 5px wide.</li> <li>Performance of Firefox on Plasma seems a bit worse than when running it on Gnome (with wayland)</li> <li>The screenshot utility is quite clunky compared to what I&#x27;m used to in Gnome. I can only select what to screenshot after the fact it seems. It&#x27;s quite good as far as screenshot UIs go but not perfect yet.</li> <li>The single-click-to-activate in Dolpin, I&#x27;d rather double click on items</li> <li>Also the active control blue border around the file list in Dolphin. It does not really need to be there and it makes the interface so much more messy. It shows there&#x27;s focus on the filebrowser widget, but what does that mean? You either have focus on a file in it and that should have the highlight or you don&#x27;t and then the widget should not have focus.</li> </ul> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The Dolpin File manager with the blue border around the files</figcaption></figure> <h2>After some more use</h2> <p>One of the things I had not used yet is Discover, I'm used to installing packages using the command line. I barely use Gnome Software, when I do it's mostly because the flatpak commandline utility is a mess to use. I tried some packagemanagement using Discover and ran into a few pros and cons. This is not only on the KDE side ofcourse because this heavily integrates with postmarketOS specific things.</p> <p>The main thing I noticed is that Discover is faster than Gnome Software, it's not even close. The speed at which Discover search works makes it actually a realistic option compared to launching a terminal. There's a few paper cuts too ofcourse. If the internet is down and you launch Discover it will show a notification for every repository configured in apk, which is 3 notifications on stock postmarketOS edge. </p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>This notification is almost completely meaningless, It says transferring not what it's transferring and shows no actual progress.</p> <p>If the internet is up it works pretty smoothly though. The only thing I miss in the rest of the UI is the "Launch" button that's present in Gnome Software to start the software you just installed instead of going through the app launcher.</p> <h2>Customization</h2> <p>Since Plasma is supposedly very customizable, I've been going through the settings to make it work closer to the way I want.</p> <p>First thing is fixing up some hardware configuration. The touchpad on the Pinebook Pro does not have separate buttons so tap-to-click needs to be enabled, and more importantly, two-finger-tap to right click. The touchpad configuration is nicely set-up in the settings utility in Plasma, this was an easy change to do.</p> <p>Changing the window management behavior for the meta+up shortcut was also pretty simple, I only was slightly confused by the shortcuts settings a bit because unchecking the checkbox below "Default shortcut" makes it not respond to my custom shortcut. But leaving the default enabled is fine sincec I would never hit meta+PgUp on a laptop keyboard by accident.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>Changing up the cursor theme has been proven to be the most annoying. The options for this by default is Breeze and Breeze-light. Which is not what I want, and not what I want in white. This means it's down to the button I always dislike in these environments, the user content browser.</p> <p>The one in this case is no exception. User content is a huge mess. Names mean nothing, screenshots are mostly useless, it's hard to find some "normal" things beside all the really weird options. Usually the service that hosts the data is also pretty unreliable. After 3 search queries I started to get this error window that immediately reappears when I close it with the button while also showing it has sucessfully fetched the results of the query behind it anyway.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>After installing some cursor themes I found that the rendering of anything that's not Breeze is just bad. The preview in the settings window look fine but when I actually apply my configuration the cursor is way smaller. It also only applies when hovering over the window decoration, I still get the old cursor theme on the window itself any any newly launched windows. Deleting the currently active cursor theme also crashes the settings application.</p> <p>I gave up on this when I remembered that icon themes are standardized. So I installed the <code>adwaita-icon-theme</code> package from Alpine Linux and then I got the option for the adwaita cursor theme. After a reboot the new cursor was consistently applied.</p> <p>Other customisation things I wanted to try is the general system theming. There's quite a bit of different theming settings available. The first one I messed with is the "Application style"</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The preinstalled application styles</figcaption></figure> <p>By default the theme is set to Breeze, I switched it to Fusion because it has actual old school 3D controls like GTK used to have until they decided that custom options are forbidden on libadwaita-enabled applications. Maybe I like this theme because it's closest to old-school clearlooks :)</p> <p>Changing this theme also fixes the blue border focus weirdness in Dolphin.</p> <p>For the GTK apps I use there's a seperate settings page.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Gtk theme settings</figcaption></figure> <p>For GTK there's no preview thumbnails and the settings are very incomplete. With the default Breeze settings the GTK apps integrate more into the general theming of Plasma, with Default it will let the apps use Adwaita. With the Emacs theme... it also just uses Adwaita. <s>The option I'm missing here though is the dark variant of Adwaita</s>. After installing the gnome-themes-extra package I have the Adwaita Dark theme available that I wanted.</p> <p>I think making GTK look like Breeze is nice from a technical standpoint but by doing that the app will not look like a proper Plasma application, it will just look like it has the same colors and widgets but it will still follow the Gnome HIG, not the Plasma one. Also changing the widget theme to Fusion like I did will not change that for the Breeze GTK theme ofcourse so that introduces more inconsistency. I'd rather have the GTK apps look the way it was intended instead of half-following the Plasma styles.</p> <h2>Finding more workarounds</h2> <p>After some use I have found workarounds for some of my bigger issues. The wifi connection issue can be solved by setting the connection profile to "All users" makes auto-connecting on startup work again.</p> <p>The annoying widgets I did not want in various places could be removed with the customisation features of Plasma, so not really a workaround but really intended functionality :D</p> <p>The only large issue I have left is the performance of the application launcher, I'm pretty sure that won't be an easy fix.</p> <h2>Conclusion</h2> <p>I've been running Plasma for a bit now on the Pinebook Pro and I think I will leave it on. It fulfills it's purpose of being the frame around my terminal windows and the browser and hits most of the performance goals I have for this hardware.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>Green accent color! without hacking up the theme!</figcaption></figure> <p></p> Automated Phone Testing pt.4 setupMartijn BraamFri, 14 Oct 2022 17:20:51 -0000<p>To execute CI jobs on the hardware there needs to be a format to specify the commands to run. Every CI platform has its own custom format for this, most of them are based on YAML.</p> <p>My initial plan was to use YAML too for this since it's so common. YAML works <i>just</i> good enough to make it work on platforms like GitHub Actions and Gitlab CI. One thing that's quite apparent though is that YAML is just not a great language to put blocks of shell scripting into.</p> <div class="highlight"><pre><span></span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">busybox:latest</span><span class="w"></span> <span class="nt">before_script</span><span class="p">:</span><span class="w"></span> <span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo &quot;Before script section&quot;</span><span class="w"></span> <span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo &quot;For example you might run an update here or install a build dependency&quot;</span><span class="w"></span> <span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo &quot;Or perhaps you might print out some debugging details&quot;</span><span class="w"></span> <span class="nt">after_script</span><span class="p">:</span><span class="w"></span> <span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo &quot;After script section&quot;</span><span class="w"></span> <span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo &quot;For example you might do some cleanup here&quot;</span><span class="w"></span> <span class="nt">build1</span><span class="p">:</span><span class="w"></span> <span class="w"> </span><span class="nt">stage</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">build</span><span class="w"></span> <span class="w"> </span><span class="nt">script</span><span class="p">:</span><span class="w"></span> <span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">echo &quot;Do your build here&quot;</span><span class="w"></span> </pre></div> <p>Blocks of shell script are either defined as lists like the example above or using one of the multiline string formats in YAML. This works but is not very convenient.</p> <h2>Design constraints</h2> <p>There is a few things that the job format for PTS needs to solve. The main thing being that jobs are submitted for multiple devices that might behave slightly differently. Of course it's possible to use some conditionals in bash to solve this but leaning on shell scripting to fix this is a workaround at best.</p> <p>The things that I needed from the common job execution formats is a way to specify metadata about the job and a way to specify commands to run. One major difference with other CI systems is that a test job running on the target hardware involves rebooting the hardware into various modes and checking if the hardware behaves correctly. </p> <p>Here is an example of the job description language I've come up with:</p> <pre><code>device: hammerhead env { ARCH: aarch64 CODENAME: lg-hammerhead BOOTLOADER: fastboot PMOS_CATEGORY: testing NUMBER: 432 } power { reset } fastboot (BOOTLOADER==&quot;fastboot&quot;) { flash userdata${CODENAME}/userdata.img boot${CODENAME}/boot.img } heimdall (BOOTLOADER==&quot;heimdall&quot;) { flash userdata${CODENAME}/userdata.img flash boot${CODENAME}/boot.img continue } shell { username: root password: 1234 script { uname -a } }</code></pre> <p>This format accepts indentation but it is not required. All nesting is controlled by braces.</p> <p>The top level data structure for this format is the Block. The whole contents of the file is a single block and in the example above the <code>env</code>, <code>power</code> etc blocks are.. Blocks. </p> <p>Blocks can contain three things:</p> <ul><li>Another nested block</li> <li>A Definition, which is a key/value pair</li> <li>A Statement, which is a regular line of text</li> </ul> <p>In the example above definitions are used to specify metadata and environment variables and the script itself is defined as statements.</p> <p>Blocks also have the option to add a condition on them. The conditions are used by the controller daemon to select the right blocks to execute.</p> <p>This is just the syntax though, to make this actually work I wrote a lexer and parser for this format in Python. This produces the following debug output:</p> <pre><code>&lt;ActBlock act &lt;ActDefinition device: hammerhead&gt; &lt;ActBlock env &lt;ActDefinition ARCH: aarch64&gt; &lt;ActDefinition CODENAME: lg-hammerhead&gt; &lt;ActDefinition BOOTLOADER: fastboot&gt; &lt;ActDefinition PMOS_CATEGORY: testing&gt; &lt;ActDefinition NUMBER: 432&gt; &gt; &lt;ActBlock power &lt;ActStatement reset&gt; &gt; &lt;ActBlock fastboot: &lt;ActCondition &lt;ActReference BOOTLOADER&gt; == fastboot&gt; &lt;ActStatement flash userdata${CODENAME}/userdata.img&gt; &lt;ActStatement boot${CODENAME}/boot.img&gt; &gt; &lt;ActBlock heimdall: &lt;ActCondition &lt;ActReference BOOTLOADER&gt; == heimdall&gt; &lt;ActStatement flash userdata${CODENAME}/userdata.img&gt; &lt;ActStatement flash boot${CODENAME}/boot.img&gt; &lt;ActStatement continue&gt; &gt; &lt;ActBlock shell &lt;ActDefinition username: root&gt; &lt;ActDefinition password: 1234&gt; &lt;ActBlock script &lt;ActStatement uname -a&gt; &gt; &gt; &gt;</code></pre> <p>Now the controller needs to actually use the parsed act file to execute the task. After parsing this is reasonably simple. Just iterate over the top level blocks and have a module in the controller that executes that specific task. A <code>power</code> module that takes the contents of the power block and sends the commands to the pi. Some flasher modules to handle the flashing process.</p> <h2>Developer friendliness</h2> <p>The method to execute the blocks as modules is simple to implement, but something that's very important for this part is the developer friendliness. Testing is difficult enough and you don't want to have to deal with overly verbose specification languages.</p> <p>It's great that with conditions and flashing-protocol specific blocks an act can describe how to flash on multiple devices depending on variables. But... that's a level of precision that's not needed for most cases. The <code>fastboot</code> module would give you access to run arbitrary fastboot commands which is great for debugging but for most testjobs you just want to get a kernel/initramfs running on whatever method the specific device supports. So one additional module is needed:</p> <pre><code>boot { # Define a rootfs to flash on the default partition rootfs: something/rootfs.gz # For android devices specify a boot.img bootimg: something/boot.img }</code></pre> <p>This takes the available image for the device and then flashes it to the most default locations for the device. This is something that's defined by the controller configuration. On the non-android devices specifying the rootfs would be enough, it would write the image to the whole target disk. This would be enough for the PinePhone for example.</p> <p>For Android devices things are different ofcourse. There most devices need to have the boot.img and rootfs.img produced by postmarketOS to boot. For those the rootfs can be written to either the system or userdata partition and in many cases boot.img can be loaded to ram instead of flashing. For test cases that can run from initramfs this would mean no wear on the eMMC of the device at all.</p> <p>With this together a minimal test case would be something like this:</p> <pre><code>device: pine64-pinephone boot { rootfs: } shell { username: user password: 147147 script { uname -a } }</code></pre> <p>The <code>boot</code> module will take care of resetting the device, powering on, getting into the correct flasher mode, flashing the images and rebooting the device.</p> <p>After that it's just <code>shell</code> blocks to run actual test scripts.</p> <p>After implementing all of this in the PTS controller I made a small testjob exactly like the one above that loads the PinePhone jumpdrive image. Since that image is small and easy to test with.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>Jumpdrive booted on the PinePhone in the test setup</figcaption></figure> <h2>Detecting success and failure</h2> <p>One thing that's a lot easier in container test scripts than on serial-controlled real hardware is detecting the result of test jobs. There's not multiple streams like stdout/stderr anymore, there's no exit codes anymore. The only thing there is is a string of text.</p> <p>There's two solutions to this and I'm implementing both. The first one is specifying a string to search for to mark a test as sucessful or failed, this is the easiest solution and should work great for output of testsuites.</p> <pre><code>shell { username: user password: 147147 success: linux script { uname -a } }</code></pre> <p>The other one is moving from uart to telnet as soon as possible. If a test image just has telnetd in the initramfs and IPv6 connectivity then the controller can automatically figure out the IPv6 link local address on the phone side and connect to the device with telnet and a preset username/password. This is a bit harder with IPv4 connectivity due to multiple devices being connected and they might have overlapping addresses.</p> <p>One a telnet session is established something closer to a traditional CI suite can function by sending over the script and just executing it instead of automating a shell.</p> <pre><code>telnet { username: user password: 147147 script { uname -a } }</code></pre> <p>Beside this the other blocks can signal a failure. The <code>boot</code> block will mark the test as failed when the device could not recognize the boot image for example.</p> <h2>Next up</h2> <p>With this all the major components have their minimal required functionality working. The next steps is building up the first rack case and deploying an instance of the central controller for postmarketOS. I've already been printing more 3D phone holders for the system. One of the prototypes of this is teased in the top image in the article :)</p> Automated Phone Testing pt.3 setupMartijn BraamTue, 27 Sep 2022 12:32:24 -0000<p>So in the previous post I mentioned the next step was figuring out the job description language... Instead of that I implemented the daemon that sits between the hardware and the central controller.</p> <p>The original design has a daemon that connects to the webinterface and hooks up to all the devices connected to the computer. This works fine for most things but it also means that to restart this daemon in production all the connected devices have to be idle or all the jobs have to be aborted. This can be worked around by having a method to hot-reload configuration for the daemon and deal with the other cases that would require a restart. I opted for the simpeler option of just running one instance of the daemon for every connected device.</p> <p>The daemon is also written in Python, like the other tools. It runs a networking thread, hardware monitoring thread and queue runner thread.</p> <h2>Message queues</h2> <p>In order to not have to poll the webinterface for new tasks a message queue or message bus is required. There are a lot of options available to do this so I limited myself to two options I had already used. Mosquitto and RabbitMQ. These have slightly different feature sets but basically do the same thing. The main difference is that RabbitMQ actually implements a queue system where tasks are loaded into and can be picked from the queue by multiple clients. Clients then ack and nack tasks and tasks get re-queued when something goes wrong. This essentially duplicates quite a few parts of the existing queue functions already in the central controller. Mosquitto is way simpler, it deals with messages instead of tasks. The highest level feature the protocol has is that it can guarantee a message is delivered.</p> <p>I chose Mosquitto for this reason. The throughput for the queue is not nearly high enough that something like RabbitMQ is required to handle the load. The message bus feature of Mosquitto can be used to notify the daemon that a new task is available and then the daemon can fetch the full data over plain old https.</p> <p>The second feature I'm using the message bus for is streaming the logs. Every time a line of data is transmitted from the phone over the serial port the daemon will make that a mqtt message and send it to the RabbitMQ daemon running on the same machine as the webinterface. The webinterface daemon is subscribed to those messages and stores them on disk, ready to render when the job page is requested.</p> <p>With the current implementation the system creates one topic per task and the daemon sends the log messages to that topic. One feature that can be used to make the webinterface more efficient is the websocket protocol support in the mqtt daemon. With this it's no longer required to reload the webinterface for new log messages or fetch chunks through ajax. When the page is open it's possible to subscribe to the topic for the task with javascript and append log messages as they stream in in real time.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>Log messages sent over MQTT</figcaption></figure> <h2>Authentication</h2> <p>With the addition of a message bus, it's now required to authenticate to that as well, increasing the set-up complexity of the system. Since version 2 there's an interesting plugin bundled with the Mosquitto daemon: dynsec.</p> <p>With this plugin accounts, roles and acls can be manipulated at runtime by sending messages to a special topic. With this I can create dynamic accounts for the controllers to connect to the message bus and relay that information on request from the http api to the controller to request on startup.</p> <p>One thing missing from this is that the only way to officially use dynsec seems to be the <code>mosquitto_ctrl</code> commandline utility to modify the dynsec database. I don't like shelling out to executables to get things done since it adds more dependencies outside the package manager for the main language. The protocol used by <code>mosquitto_ctrl</code> is quite simple though, not very well documented but easy to figure out by reading the source.</p> <h2>Flask-MultiMQTT</h2> <p>To connect to Mosquitto from inside a Flask webapplication the most common way is with the <code>Flask-MQTT</code> extension. This has a major downside though that's listed directly at the top of the <a href="">Flask-MQTT documentation</a>; It doesn't work correctly in a threading Flask application, it also fails when hot-reload is enabled in Flask because that spawns threads. This conflicts a lot with the other warning in Flask itself which is that the built-in webserver in flask is not a production server. The production servers are the threading ones.</p> <p>My original plan was to create an extension to do dynsec on top of Flask-MQTT but looking at the amount of code that's actually in Flask-MQTT and the downsides it has I would have to work around I decided to make a new extension for flask that <i>does</i> handle threading. The <a href="">Flask-MultiMQTT</a> is available on pypi now and has most of the features of the Flask-MQTT extension and the extra features I needed. It also includes helpers for doing runtime changes to dynsec.</p> <p>Some notable changes from Flask-MQTT are:</p> <ul><li>Instead of the list of config options like <code>MQTT_HOST</code> etc it can get the most important ones from the <code>MQTT_URI</code> option in the format <code>mqtts://username:password@hostname:port/prefix</code>.</li> <li>Support for a <code>prefix</code> setting that is prefixed to all topics in the application to have all the topics for a project namespaced below a specific prefix.</li> <li>Integrates more with the flask routing. Instead of having the <code>@mqtt.on_message()</code> decorator (and the less documented <code>@mqtt.on_topic(topic)</code> decorators where you still have to manually subscribe there is an <code>@mqtt.topic(topic)</code> decorator that subscribes on connection and handles wildcards exactly like flask does with <code>@mqtt.topic(&quot;number-topic/&lt;int:my_number&gt;/something&quot;)</code>.</li> <li>It adds a <code>mqtt.topic_for()</code> function that acts like <code>flask.url_for()</code> but for topics subscribed with the decorator. This can generate the topic with the placeholders filled in like url_for() does and also supports getting the topic with and without the prefix .</li> <li>Implements <code>mqtt.dynsec_*()</code> functions to manipulate the dynsec database.</li> </ul> <p>This might seem like overkill but it was suprisingly easy to make, even if I didn't make a new extension most time would be spend figuring out how to use dynsec and weird threading issues in Flask.</p> <h2>Log streaming</h2> <p>The serial port data is line-buffered and streamed to an mqtt topic for the task, but this is not as simple as just dumping the line into payload of the mqtt message and sending it off. The controller itself logs about the state of the test system and already parses UART messages to figure out where in the boot process the device is to facilitate automated flashing.</p> <p>The log messages are send as json objects over the message bus. The line is annotated by the source of the message, which is <code>"uart"</code> most of the time. There are also more complex log messages that have the full details about USB plug events.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The USB event generated when the postmarketOS initramfs creates the USB network adapter on the PinePhone</figcaption></figure> <p>Beside uart passthrough messages there's also inline status messages from the controller itself when it's starting the job and flasher messages when the flasher thread is writing a new image to the device. This can be extended to have more annotated sources like having syslog messages passed along once the system is booted and if there is a helper installed.</p> <p>This log streaming can also be extended to have a topic for messages in the other direction, that way it would be possible to get a shell on the running device.</p> <p>With this all together the system can split up the logs over UART in some sections based on hardcoded knowledge of log messages from the kernel and the bootloader and create nice collapsible sections. </p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>A PinePhone booting, flashing an image using Tow-Boot and then bootlooping</figcaption></figure> <h2>Running test jobs</h2> <p>The current system still runs a hardcoded script at the controller when receiving a job instead of parsing a job manifest since I postponed the job description language. The demo above will flash the first attached file to the job, which in my case is <code>pine64-pinephone.img</code>, a small postmarketOS image. Then it will reboot the phone and just do nothing except pass through UART messages.</p> <p>This implementation does not have a way to end jobs yet and have success/failure conditions at the end of the script. There are a few failure conditions that are implemented which I ran into while debugging this system.</p> <p>The first failure condition it can detect is a PinePhone bootlooping. Sometimes the bootloader crashes due to things like insufficient power, or the A64 SoC being in a weird state due to the last test run. When the device keeps switching between the <code>spl</code> and <code>tow-boot</code> state it will mark it as a bootloop and fail the job. Another infinite loop that can easily be triggered by not inserting the battery is it failing directly after starting the kernel. This is what is happening in the screenshot above. This is not something that can be fully automatically detected since a phone rebooting is a supported case.</p> <p>To make this failure condition detectable the job description needs a way to specify if a reboot at a specific point is expected. Or in a more generic way, to specify which state transitions are allowed in specific points in the test run. With this implemented it would remove a whole category of failures that would require manual intervention to reset the system.</p> <p>The third failure condition I encountered was the phone not entering the flashing mode correctly. If the system wants to go to flashing mode but the log starts outputting kernel logs it will mark it as a failure. In my case this failure was triggered because my solder joint on the PinePhone failed so the volume button was not held down correctly.</p> <p>Another thing that needs to be figured out is how to pass test results to the controller. Normally in CI systems failure conditions are easy. You execute the script and if the exit code is not <code>0</code> it will mark the task as a failure. This works when executing over SSH but when running commands over UART that metadata is lost. Some solutions for this would be having a wrapper script that catches the return code and prints it in a predefined format to the UART so the logparser can detect it. Even better is bringing up networking on the system if possible so tests can be executed over something better than a serial port.</p> <p>Having networking would also fix another issue: how to get job artifacts out. If job artifacts are needed and there is only a serial line. The only option is sending some magic bytes over the serial port to tell the controller it's sending a file and then dump the contents of the file with some metadata and encoding. Luckily since dial up modems were a thing people have figured out these protocols in XMODEM, YMODEM and ZMODEM.</p> <p>Since having a network connection cannot be relied on the phone test system would probably need to implement both the "everything runs over a serial line" codepath and the faster and more reliable methods that use networking. For tests where networking can be brought up some helper would be needed inside the test images that are flashed that brings up USB networking like the postmarketOS initramfs does and then communicates with the controller over serial to signal it's IP address.</p> <p>So next part would be figuring out the job description (again) and making the utilities for using inside the test images to help execute them.</p> Automated Phone Testing pt.2 setupMartijn BraamFri, 16 Sep 2022 12:30:37 -0000<p>In <a href="/automated-phone-testing-pt-1/">part 1</a> I hooked up a PinePhone to a Raspberry Pi Pico and wrote the first iteration of the firmware for that setup. For testing I had used one USB port for the Pi Pico, one port for the PINE64 Serial cable and one for the PinePhone itself.</p> <p>By using 3 USB ports per device I would run out of USB ports on the control computer pretty fast. This is why I picked the Pi Pico, it's not only cheap and available but also is able to emulate multiple USB serial ports at the same time. This required a rework of how the USB protocol was implemented.</p> <p>The "default" way to use the USB serial port on the Pico is to call <code>printf()</code> to write data to stdout in the firmware and configure the build scripts to enable USB uart. This will make the Pico create an USB CDC ACM serial port on the usb port and it will be hooked into stdout/stdin. To add a second ACM interface none of this automated setup can be used at all.</p> <p>The second revision of the firmware replaces the <code>printf</code>/<code>getchar</code> usage with directly using the tinyUSB library. Using this I can manually define all USB descriptors I need and hook them up into the parts of the firmware where I need them. With my own descriptors it means the device no longer shows up as Raspberry Pi Pico in <code>lsusb</code>, instead it's now "postmarketOS Test Device". </p> <p>Ignoring all the extra setup code to get tinyUSB running, the changes to the existing parts of the firmware is mostly replacing <code>getchar</code> with <code>tud_cdc_n_read</code> which accepts the port number to read from, and replacing <code>printf</code> with <code>tud_cdc_n_write_str</code>. One additional piece of the firmware now reads bytes from the USB CDC 2 port and writes it to the hardware uart1 on the Pi Pico and vice versa. With this and a tiny extra cable I fully replaced the need of the USB to serial adapter, freeing up one USB port per test device.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>The PinePhone serial port in the headphone jack now connects to the Pi Pico</figcaption></figure> <h2>Power switching</h2> <p>One other feature that was not implemented yet in the previous blog post is power switching. To bring the device in a known state it's easiest to just turn it off and on again. Since having someone sitting beside the test rack to pull out the USB cable is not very practical I needed a way to interrupt the power lines in the USB cable going to the phone. This is slightly more complicated than wiring some test pads directly to the pins of the Pico so to not hold up the software development parts of this I made a test PCB that allows me to control the power of an USB cable:</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Power control board, revision 1</figcaption></figure> <p>This is a piece of protoboard put in the middle of one of the red pine64 USB cables. It has a relay that interrupts the 5V line inside this usb cable and the data lines are passed through directly. It also includes the circuitry to control this relay from the Pi Pico when hooked up to the 2 pins to the left of the board. For those that know electronics; the diode that is missing in this picture is on the bottom of the board.</p> <p>For the final version in the test setup this won't be a seperate board, it will also not use a relay. Relays have a limited number of operations and they make noise. Having mechanical parts in a test setup is not good for reliability in any case. This will be replaced by a solid state method of switching power, most likely a mosfet.</p> <p>This board is then hooked up to the ground and GPIO 6 of the Pi Pico and it will make the <code>p/P</code> command work for switching the phone power.</p> <h2>The central controller</h2> <p>To have a look all the way to the other end of the development for this project, the webapplication that will control it all. This is the interface presented towards the developers that will use the test setup.</p> <p>This piece of software is comparable to something like <a href="">Lava</a> from Linaro. It has a webinterface to submit and view the builds and have an overview of the devices. The things I'm doing differently compared to Lava is not using jinja2 templates for configuration and having a passthrough mode for scheduling a live connection to a device in the test rack.</p> <p>The implementation of the central controller is partially modelled after the way <a href="">Sourcehut Builds</a> works. It will mostly act like a regular CI system but it's backed by physical hardware instead of VMs or containers. It also aims to integrate some of the features that makes builds nice to work with like being able to get a shell into the test environment so you don't have to keep submitting slightly changed CI jobs multiple times to debug things. Also having nice split logs for the CI tasks. </p> <p>The first thing to look at is how devices are defined in the test system. The information is intentionally kept minimal. There's a definition of a device type which is just a name, chipset and architecture. In the case of postmarketOS this name would be the postmarketOS device code like <code>pine64-pinephone</code></p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>Excerpt of the database schema</figcaption></figure> <p>For every device in the test rack a row in the device table would be created. This row can have an addition hardware revision number, like <code>1.2B</code> for a PinePhone, and contains information about which controller it is connected to and which user the maintainer for this device is.</p> <p>This by itself is not enough to do all the required test scheduling things. There needs to be a way to specify extra information that can be queried in the test jobs or can be used for filtering devices. For this there's another part in the schema:</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>There is a <code>variable</code> table that is an arbitrary key-value store for extra data. This data can be used for selecting on which devices to run specific tasks and the values will also be available in things like environment variables. The <code>variable</code> table defines data at three different levels. if <code>device_id</code> and <code>devicetype_id</code> is unset in a row it will be a global variable for the whole test system. Then variables can be defined at the device type level and then again at the device level. The job specification itself also contains a fourth level of variables that will be defined in the job description format.</p> <p>When a job is run all the variables for the current device will be looked up and combined in a cascading system. Device type variables override global variables with the same name, same with device variables and job variables. This data will be then appended to the job that is submitted to the controller and executed.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>Some examples for variables in my current test setup are <code>PMOS_CATEGORY</code> which sets in which postmarketOS device classification category the device is., to make it possible to run a testjob on all the devices in the main or community category. Another is the <code>DTB</code> variable that sets what the device tree name is for the specific devices. This is something that would be a device-type variable for a lot of devices, but would be overriden on the device level for the PinePhone due to hardware revisions that have seperate upstream device tree files</p> <h2>The user facing parts</h2> <p>The webapplication has three levels of authentications. Guests, Developers and Admins. Visitors that are not signed in get the guest role and can see the status dashboard and all the job results, similar how <a href=""></a> lets guests see the state of the postmarketOS build system.</p> <p>Accounts are created for developers, which grants the permission to submit new jobs in the system. For all the other administration tasks there's the admin role which allows linking controllers, defining devices and creating new accounts.</p> <p>The device editing part of the administration interface is already shown in the screenshot above. The admin panel gives access to all the settings in the application so it's not necessary to have a shell on the server running it to do administration tasks.</p> <p>Guests visiting the webapplication will land on the dashboard:</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The central controller dashboard page</figcaption></figure> <p>This shows all the currently running jobs of which there are none since the controller is not implemented yet. It also gives some quick statistics on the side about the overall system.</p> <p>The more detailed information is shown on the Jobs and Devices page. The devices page shows a list of the defined device types and the active registered devices. In my case the device naming tells me which rack case the device is in and the slot number. </p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>The public device list page</figcaption></figure> <p>And finally the jobs page. It shows a list of all queued, running and completed jobs. The internal terminology for the overall system is: jobs are the manifests submitted by the user, tasks are generated from the job for every single matching device. If the job specifies exactly one device with the filters it will generate a single task which might have subtasks for test output legibility. </p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The full job list on the public side</figcaption></figure> <p>Future improvements to this is showing the status of all the tasks in the job in this job listing. For logged in developers there's the option to submit a new job in the sidebar. This links to the job creation page. This is all very similar to the workflow in <a href=""></a> which this is based on.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The job submission page</figcaption></figure> <p>Jobs are submitted using a code editor page. All the information of this job is part of the job manifest so it's easy to copy and paste parts of job configurations. Documentation has to be written once the job description format is stable.</p> <p>Finally there's the job detail page. This shows the job result in real time.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>The job detail page</figcaption></figure> <p>The contents of the job page will mainly be filled by text blocks streamed from the controller. Just like there's an option to download the manifest for a job and jump to the job submission form with the current manifest loaded.</p> <h2>Controllers</h2> <p>The main system that will hook all the parts of the system together is the controller software. This will run on a machine in the test rack and will communicate with the central controller to fetch jobs. The controller will be implemented as a daemon and will be controlled using a command line utility. The API interface between the central controller and the rack controller has not been fully defined yet. Most of the tasks will be a regular REST HTTP api but the test results will need a more complicated data streaming setup.</p> <p>Part of this communication is the possiblity to get a shell on a device that will emulate a regular SSH session. Since in the design the rack controller is not portforwarded this would need to be a reverse shell and it needs to be proxied on the central controller to allow developers to get to that shell from their own machine. This is a bit of a complication but it would make quite a few things a lot simpler like running kernel bisections remotely or even doing remote development in general.</p> <h2>Up to part 3</h2> <p>The next part of this system is figuring out what things are required in the job description format. Submitting a single script to a single phone is easy, dealing with differences between devices is the hard part. Also as a sneak peak into the future parts; the header image shows the 3d printed slots in the rack case the phones will slide into.</p> <p></p> Configuring Blackmagic Design converters in Linux BraamThu, 18 Aug 2022 23:35:25 -0000<p>Blackmagic Design makes a lot of video converters in various sizes and feature sets. These are relatively cheap boxes that convert between HDMI, SDI and optical fiber for video signals. There's also some neat converters that can do audio embedding and de-embedding.</p> <p>Normally these converters are configured using a Windows or OSX utility. Just plug in a converter with an USB cable and change one of the few settings. This is pretty nice except there's no utility for Linux.</p> <h2>Making a Linux utility</h2> <p>I already maintain <a href="">pyatem/openswitcher</a> which is a reverse engineered protocol implementation for the ATEM video switcher devices from Blackmagic Design. I decided to also take a look at the USB protocol for the converters to add this to the pyatem python library.</p> <p>I started by looking at the BiDirectional SDI/HDMI 12G micro converter. To look at the traffic I installed <a href="">USBPCap</a> on a Windows machine. This is not the most convenient tool but it's the quickest way to get USB logs.</p> <p>With USBPCap running I started the official software and changed every setting once and then ended the capture. With this captured traffic I could start to look at how the protocol works using Wireshark.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>Looking at an USB dump using Wireshark</figcaption></figure> <p>The protocol is relatively simple, it only uses USB control transfers and retrieving/writing settings is done by sending a setting name to the device and then reading or writing the value.</p> <p>The more difficult part of this design is that you need to know in advance which settings exist on which converter. There does not seem to be a way to enumerate a list of the settings from the device.</p> <p>Using this information I wrote the <code>pyatem.converter</code> module that implements the USB protocol using libusb and also added a definition that describes all the settings for the BiDirectional converter I used.</p> <p>With the library implemented I greated a simple GTK3 application that presented the settings in a similar way to the official software:</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"><figcaption>Converter Setup on Linux</figcaption></figure> <h2>Supporting more hardware</h2> <p>A setup utility that only works with one model of converters is not super useful. It also creates the risk of overfitting the library to some specific protocol quirk that one model has, and this is exactly what happened in this case.</p> <p>I ordered the HDMI to SDI and SDI to HDMI 3G converters since these are cheap and really common. They are also an older generation of converters. For those not familiar with this hardware: the 12G converters are the 4K capable ones and the 3G converters top out at 1080p30. The number refers to the link speed.</p> <p>I created another packet dump using Windows to find which fields are supported and to my suprise the protocol for these converters is completely different.</p> <p>The protocol for these 3G converters does not work by sending a setting name and then a value. Instead the settings are numbers and the operation is done in a single usb transaction. This is possible since a numeric setting can be set in the wValue field of a read or write operation to signify what you want to read or write.</p> <p>Due to this I had to refactor the whole library to support multiple protocol backends. This would also make it possible to support more kinds of Blackmagic USB protocols for their different products like the ATEM Mini hardware and the monitors. These are normally configured using seperate setup utilities.</p> <h2>Going mobile</h2> <p>Since I also work on postmarketOS I tried running my Linux app on a PinePhone. The only change I had to make it removing the "Blackmagic Design" name from the converter name because the product names are so ridiculously long. After that the Linux desktop setup utility I made Just Works(tm) on Linux phones:</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Converter Setup on postmarketOS</figcaption></figure> <p>After sharing this picture there seemed a bit of interest for having a mobile app for configuring the converters. This makes a lot of sense, the converters are usually spread out all over the place and dragging a laptop around to configure them is annoying.</p> <p>While it's nice that this works on postmarketOS supported phones, the majority of people still have Android and iOS phones. Since I already know how the protocol works and doing a simple Android app shouldn't be too hard... I decided to learn Android development.</p> <h2>The Android app</h2> <p>I had tried to do Android development before and always got stuck very early in the process making me never continue it. It seems like Kotlin is a great improvement over the Java issues I was having though. I managed to get a proof of concept up and running that read the settings in a single converter in a day and continued developing. I also signed up for a Google Play developer account and went through the verification steps. It turns out dealing with the Google Play Console is definetly the hardest and most time consuming part of development.</p> <p>It took about a week to get everything on the google side verified and reviewed. Luckily I did this with by first hello world demo and didn't wait until I had finished the rest of the app otherwise I would've lost a lot more time on this. In the week of waiting I fixed up the multi-protocol setup and refactored the code of the app to be actually nice instead of one large MainActivity.kt file with hardcoded values.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>The Converter Configuration Android app</figcaption></figure> <p>My plan with the Android app is to make it paid and keep the opensource Linux app and libraries free. This makes me able to fund the development of the open source side through the Android app and give a wider audience the ability to actually use the app. The nice thing is that the Play store already takes care of all the payment and licensing for this.I can just keep using the funds from the Android app to get more hardware to reverse engineer and add to the app.</p> <p>The Android app is now in open beta. The beta can be joined through <a href=""></a>. Currently the app only supports the three converters mentioned above but I hope to provide an update soon to at least get the UpDownCross converter working.</p> <p>I'm not planning to make an iOS app for this though, that would require me to buy into the Apple ecosystem with multiple devices to even get started and learn a whole new software stack. I'm also not entirely sure if apps controlling non-class-compliant usb hardware is even possible in the Apple mobile ecosystem.</p> Why I left PINE64 BraamWed, 17 Aug 2022 09:47:04 -0000<p>Linux hardware projects are made or broken by their community support. PINE64 has made some brilliant moves to build up a mobile Linux community, and has also made some major mistakes. This is my view on how PINE64 made the PinePhone a success, and then broke that again through their treatment of the community.</p> <p>I want to start by pointing out that this is <i>me</i> leaving PINE64 and not the projects I'm involved in like postmarketOS. These opinions are my own yadda yadda...</p> <h2>Community Editions and the PinePhone's early life</h2> <p>The original PinePhone was brought up on the existing Linux Mobile projects like Ubuntu Touch, postmarketOS, and Maemo Leste, and also spawned new Linux distributions like Mobian and Danctnix ARM. This grew until there were 25 different projects working on the PinePhone — an apparently thriving community.</p> <p>Following the initial set of Developer Editions, intended for community hackers to build the software with, the first consumer-targeted PinePhone devices were the Community Editions. A batch of PinePhones was produced with a special branded back cover for five community projects: UBPorts, postmarketOS, Mobian, Manjaro, and KDE Plasma Mobile. Every Community Edition phone sold also sent $10 to their respective Linux distribution projects.</p> <p>Working together through these Community Editions, Linux distributions built a software ecosystem that works pretty well on the PinePhone.</p> <h2>The end of community editions</h2> <p>In February 2021, PINE64 <a href="" rel="nofollow noopener">announced the end of the community editions</a>. At this moment, PINE64's focus shifted from supporting a diverse ecosystem of distributions and software projects around the PinePhone to just supporting Manjaro Linux alone.</p> <p>The fact that a useful software ecosystem for the PinePhone exists at all is thanks to the diverse strategy employed by PINE64 in supporting many distributions working together on each of the many facets of the software required. Much of the original hardware bring-up was done by Ubuntu Touch. Mobian developers built the telephony stack via their eg25-manager project. And in my role for the postmarketOS distribution, I developed the camera stack.</p> <p>Manjaro Linux has been largely absent from these efforts. The people working on many of the Linux distributions represented in the community editions tend to work not just on packaging software, but on building software as well. This is not the case for Manjaro, which focuses almost entirely on packaging existing software. Supporting Manjaro has historically done very little to facilitate the development of the software stack which is necessary for these devices to work. In some cases the Manjaro involvement actually causes extra workload for the developers by shipping known broken versions of software and pointing to the developers for support. Which is why <a href=""></a> was started.</p> <p>Regardless, Manjaro is now the sole project endorsed and financially supported by PINE64, at least for the Linux capable devices. As a consequence it has a disproportionate level of influence in how PINE64 develops its products and manages the ecosystem.</p> <h2>The last straw</h2> <p>With community members influence in PINE64 diminished in favor of a Manjaro mono-culture, what was once a vibrant ecosystem has been reduced to a bunch of burnt-out and maligned developers abandoning the project. The development channels are no longer the great collaboration between various distributions developing PinePhone components and there are now only a small number of unpaid developers working on anything important. Many of PINE64's new devices, such as the PinePhone Pro, PineNote, and others, have few to no developers working on the software — a potential death blow for PINE64's model of relying on the community to build the software.</p> <p>Everyone has had a different "last straw". For me, it was the SPI flash situation.</p> <p>There is a substantial change to booting between the PinePhone and PinePhone Pro. Previously, each distribution could develop a self-contained eMMC or microSD card image, including a compatible bootloader and kernel distribution. Installation is as simple as flashing a microSD card with the desired distribution and popping it in.</p> <p>On the PinePhone Pro, the hardware works differently: it prefers to load the bootloader from the eMMC instead of the microSD. This means that when the PinePhone Pro shipped from the factory with Manjaro on the eMMC it will always boot the Manjaro u-Boot, even when booting from a microSD card. We no longer have any control over the bootloader for these devices.</p> <p>There is a solution, however. The hardware can have an SPI flash chip that gives a bit of storage to put U-Boot in and that storage is always preferred over the eMMC and microSD storage. The problem with this is that all the distributions need to agree on a U-Boot build to put in there, and agree to never overwrite it with a distribution-specific version.</p> <p>The solution to this is Tow-Boot: a distribution of U-Boot that can be put in that flash chip. With this the U-Boot firmware can just be treated like system firmware and be updated through fwupd independent of what distributions ship. This would work not only for the PinePhone Pro, but would also enable things like installing your preferred Linux distribution on a PineBook Pro by popping in a flash drive with a UEFI installer, much like you can on any other laptop.</p> <p>Negotiating this solution was hell. Manjaro is incentivized not to agree to this, since it cedes their sole control over the bootloader, and PINE64 listens to Manjaro before anyone else. Furthermore, PINE64 does not actually want to add SPI flash chips to their hardware. Apparently, there has been some issues with people using SPI flash as RW storage on the A64-LTS boards, which would be a support issue.</p> <p>After months of discussions between the community, Manjaro, and PINE64 leadership, we finally were able to convince them to ship the PinePhone Pro with an SPI flash chip with Tow-Boot installed on it.</p> <p>But the Pinebook Pro has a similar boot configuration, and thus a similar problem. Some time after the PinePhone Pro was shipped, it was time for a new Pinebook Pro batch, and this discussion started again. The same arguments were re-iterated by all sides all over again, and the discussion went nowhere. PINE64 representatives went so far as to say, quote, "people who want [an SPI chip] can just solder one on". This batch of Pinebook Pros has ended up shipping without Tow-Boot flashed.</p> <h2>So I left</h2> <p>This is the moment I left. I left all the official channels, stepped down as PINE64 moderator. Left the private developer chat rooms. PINE64 cares only about Manjaro, and Manjaro does not care about working with any other distributions. This is no longer a community that listens to software developers. As a representative of postmarketOS, there is no further reason for me to be directly involved with PINE64 if the only opinions that matter are those of Manjaro.</p> <p>Like many others, I have become burnt out on this ecosystem. So I quit. I am no longer getting random requests to make Manjaro's factory software work. No longer am I enduring the stress and frustration of these meaningless discussions behind the scenes, and after not being in the PINE64 for some weeks I can definitely say I'm way less stressed out.</p> <p>Now I can just focus on making postmarketOS work better. On the PINE64 hardware, and all the many other devices supported by postmarketOS.</p> <p>I hope that future vendors will make better choices, and listen to the actual whole community. Maybe even help with the software development side.</p> Automated Phone Testing pt.1 BraamThu, 04 Aug 2022 11:14:24 -0000<p>Testing things is one of the most important aspects of a Linux distribution, especially one for phones where people rely on it being in a state where calls are possible.</p> <p>For postmarketOS the testing is done manually for a large part. There's CI jobs running that verify that packages build correctly but that's as far as the automated part goes. For releases like service packs there is quiet a manual test process that involves upgrading the installation and checking if the device still boots and that the upgraded applications actually work.</p> <p>With the growth of postmarketOS the things that need to be tested have grown massively. If there is a change that affects multiple devices then that process will take the majority of the time to get an image running with the changes on all the target devices, and in many cases it involves getting other device maintainers to try on their devices for big releases.</p> <h2>Automation</h2> <p>Automated testing for the devices postmarketOS supports is not very easy. Phones don't have the interfaces to fully automate a test process. To get a new image flashed on a phone it usually takes a few steps of holding the right key combinations and plugging in the usb cable at the right moment. This process is also significantly different for a lot of devices to complicate things further.</p> <p>The goals for the automated testing are quite ambitious. The plan is to get as many devices as possible in a server rack hooked up to a computer with wires soldered to the phone to trigger the key combinations to control the boot process.</p> <p>For the software side this would require some interface to let multiple developers schedule test jobs on the devices they need and an interface to keep track of the state of all the connected hardware. This is quite a lot like the scheduling and interface for a regular CI system and large parts of this system will be modelled after how Gitlab CI works. </p> <p>This whole system will consist of many components:</p> <ul><li>A central webapplication that keeps track of all the available hardware and schedules jobs. The webapplication does not contain any implementation details about hardware except for the names.</li> <li>A server application that connects to the central webapplication and registers connected devices. This application has the responsibilty of tracking the state of devices and asks for new jobs from the central server when a device is free. There can be many instances of this server application so there can be multiple test racks maintained by different developers.</li> <li>An application that is spawned for every job that actually executes the testcase and parses the output from the serial port of the device.</li> <li>A piece of hardware that can press the buttons on the phone and deal with plugging in the power at the right moments. This hardware should be mostly a generic PCB that can deal with the most common interfaces for devices. For devices with special requirements a new board can be made that controls that.</li> <li>A case design to hold many phones in a relatively dense configuration. It is estimated that there can fit around 8 phones in a 2U rack space with a generic enclosure and some 3D printed holders.</li> </ul> <p>Splitting the software up this way should make this test setup scalable. The most complicated parts seem to be the central webapplication that should present a nice webinterface and deals with making it easy to run a test job on multiple devices, and the runner application that actually needs to deal with the hardware specific implementation details.</p> <p>Since this is quite a setup to build I've decided to start as small as possible. First get a test running by making some prototype hardware and a prototype runner application that only supports the PinePhone.</p> <h2>The initial test hardware</h2> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>For the initial test hardware I'm using an off-the-shelf Raspberry Pi pico on a breadboard. Initial design revisions were based around an Atmel atmega32u4 to implement an usb-to-serial adapter and a second serial port for hardware control. Due to the chip shortage the popular Atmel parts are practically impossible to get.</p> <p>The Pi Pico has a microcontroller that is able to emulate multiple USB serial adapters just like the Atmega32u4 and after dealing with getting the SDK running is actually quite easy to write firmware for.</p> <p>For this initial revision I'm only running a single USB serial device on the Pi Pico since the PinePhone has an easily accessible serial port with a PINE64 debug adapter. For controlling the buttons I have soldered a few wires to various test points on the PinePhone PCB.</p> <p>The buttons on the PinePhone normally work by shorting a signal to ground. This is easily emulated with a microcontroller by having the gpio the test point is connected to configured as an input so the pin on the Pico becomes a high resistence that doesn't influence the PinePhone. When pressing the button the gpio can be set to output 0 so the signal is connected to ground.</p> <p>After some testing with the Pico this works great, it took a while to figure out that the Raspberry Pi Pico enables internal pull-down resistors by default on the gpios, even when it's set to input. This caused the phone to think all the buttons were held down all the time.</p> <h2>Control protocol</h2> <p>To actually control the buttons from a computer a protocol is needed for sending those commands to the Pi Pico. After coming up with a custom protocol first I got pointed to <a href="">cdba</a>. This is a tool for booting images on boards which is exactly what I need.</p> <p>This tool is designed to work with some specific control boards which I don't have, but the protocol used for those boards is incredibly simple.</p> <p>Every command is a single character written to the serial port. For enabling power to the board a <code>P</code> is sent. For disabling the power a <code>p</code> is sent instead. This uppercase/lowercase system is also followed for holding down the power button <code>B/b</code> and the button required to get into a flasher mode <code>R/r</code>. </p> <p>This is the protocol I implemented in my first iteration of this firmware. The nice thing is that the hardware should also work with cdba, if it is a fastboot device at least.</p> <p>The code at this point is <a href="">in this paste</a>.</p> <h2>Test application</h2> <p>To verify this design I wrote a small test application in python. It connects to two serial ports and takes a PinePhone disk image to boot into.</p> <p>The code used for the first sucessful flash is <a href="">in this paste</a>.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>This application does multiple things. It first connects to the serial port of the Raspberry Pi Pico and resets all the state. Then it will hold the power button of the PinePhone for 15 seconds to force the phone in a known state.</p> <p>It also connects to the PINE64 uart debug adapter and reads the serial debug logs of Tow-Boot and the kernel. By looking for specific lines in the serial output it knows where in the boot process the phone is and it uses that to hold the volume button to get Tow-Boot into the USB Mass Storage mode. </p> <p>It then simply dd-s the disk image on the eMMC of the phone and restarts the device. Now the phone will boot into this new installation because this time the volume button is not held down while booting.</p> <p>The things that needs to be implemented after this detecting when the device is booted and logging in on the serial console so it can run the test script.</p> <p>This iteration of the script also hardcodes a lot of sleep times and device paths. Hardcoding the path to the block device works somewhat reliably on my laptop but it will fail in production when multiple devices are connected and booting in a random order. This can be fixed by specifing which USB ports the phone and test board are plugged into instead and using udev to figure out what block device and serial device belongs to which phone.</p> <h2>Next steps</h2> <p>The most important next thing to figure out in the application is designing a job description system so developers can write testcases to run. Also using this setup finding quirks in the boot process can be ironed out, like the application only flashing the phone correctly half the time because it sometimes boots the OS instead of getting into the bootloader.</p> <p>I have also already written some major parts of the central webapplication that actually deals with registering devices and data about those that can be used as variables in the test jobs. </p> <p>Once those parts integrate it would be important to get a second device up and running in the test rig like the Oneplus 6 to avoid overfitting the design to the PinePhone.</p> Comparing Hare to Rust and Zig BraamTue, 03 May 2022 13:46:11 -0000<p>Since the public release of <a href="">Hare</a> there have been a lot of comments asking how it compares to Rust and Zig. The original release post does not compare Hare to other popular languages and homepage also has no comparisons with other languages.</p> <p>This is in line with every other programming language out there. I checked <a href=""></a> and did not see a comparison to other programming languages. A similar thing with</p> <p>But in this post I will compare them anyway</p> <h2>The baseline</h2> <p>To do a proper comparison I will start with this code snippet from the Hare homepage:</p> <pre><code>use fmt; export fn main() void = { const greetings = [ &quot;Hello, world!&quot;, &quot;¡Hola Mundo!&quot;, &quot;Γειά σου Κόσμε!&quot;, &quot;Привет, мир!&quot;, &quot;こんにちは世界!&quot;, ]; for (let i = 0z; i &lt; len(greetings); i += 1) { fmt::println(greetings[i])!; }; };</code></pre> <p>This is a nice small exampe that can be used for reference and it prints Hello, world! in a few languages. What better to use as a language comparison than code that output different languages?</p> <h2>Comparing to Rust</h2> <p>To compare the Hare snippet to rust I used the <code>rustc</code> rust compiler:</p> <div class="highlight"><pre><span></span><span class="gp">$ </span>rustc helloworld.ha <span class="go">error: expected one of `!` or `::`, found keyword `fn`</span> <span class="go"> --&gt; helloworld.ha:3:8</span> <span class="go"> |</span> <span class="go">3 | export fn main() void = {</span> <span class="go"> | ^^ expected one of `!` or `::`</span> <span class="go">error: aborting due to previous error</span> </pre></div> <p>And just as I assumed. Hare is not Rust. It won't compile with the same compiler and it has a different syntax. Honestly this was kind of expected when checking out <a href=""></a> versus <a href=""></a> and it already looked like comparing apples to oranges</p> <h2>Comparing to Zig</h2> <p>Lets try Zig then. From quickly glancing at the Zig homepage it seems to have similar goals to Hare but with a completely different implementation.</p> <div class="highlight"><pre><span></span><span class="err">$</span><span class="w"> </span><span class="n">zig</span><span class="w"> </span><span class="n">cc</span><span class="w"> </span><span class="n">helloworld</span><span class="p">.</span><span class="n">ha</span><span class="w"></span> <span class="n">ld</span><span class="p">.</span><span class="n">lld</span><span class="o">:</span><span class="w"> </span><span class="k">error</span><span class="o">:</span><span class="w"> </span><span class="n">helloworld</span><span class="p">.</span><span class="n">ha</span><span class="o">:</span><span class="mi">1</span><span class="o">:</span><span class="w"> </span><span class="n">unknown</span><span class="w"> </span><span class="n">directive</span><span class="o">:</span><span class="w"> </span><span class="n">use</span><span class="w"></span> <span class="o">&gt;&gt;&gt;</span><span class="w"> </span><span class="n">use</span><span class="w"> </span><span class="n">fmt</span><span class="p">;</span><span class="w"></span> <span class="o">&gt;&gt;&gt;</span><span class="w"> </span><span class="o">^</span><span class="w"></span> </pre></div> <p>For some reason Hare is also not Zig. While <code>rustc</code> got to the third line, Zig already failed on the first. It looks like it's better to use the Hare compiler for Hare code instead.</p> Taking pictures the hard way BraamWed, 30 Mar 2022 12:41:07 -0000<p>I've never really been that interested in taking pictures on film. My first camera was digital and digital seems superior in every way. I can take as much pictures as I want, they're instantly ready and can be reviewed on the camera itself. Also one of the major upsides is that pictures are basically free.</p> <p>There's one thing that's not really cheap though on digital, larger sensor sizes. And with larger I mean full-frame 35mm and up. All my digital cameras are APS-C (~60% smaller than full frame) or below. This works absolutely great but there's some benefits to having a larger sensor sizes. Here's a comparison between the sensor size of a phone and a digital camera:</p> <figure class="kg-card kg-gallery-card"><div class="kg-gallery-container"><div class="kg-gallery-row"><div class="kg-gallery-image"><img src="" class="kg-image" width="600" height="800"></div><div class="kg-gallery-image"><img src="" class="kg-image" width="600" height="800"></div><div class="kg-gallery-image"><img src="" class="kg-image" width="600" height="797"></div></div></div></figure> <p>This is a comparison between a Pixel 2 and a DSLR. The larger sensor of the DSLR camera makes it possible to have natural blurry backgrounds. The larger the sensor the blurrier the background can be, and this also changes the distance at what you can make a picture of something and still have a blurry background. This is why portrait photography usually easily has a blurry background but landscape photography doesn't. It's in the combination of the sensor size, distance to the subject and the aperture size of the lens.</p> <p>Smartphone cameras are a great example of sensor size since the size difference between a professional camera and a phone camera is in the order of 10x smaller. In the pictures above you can see that the phone picture on the left still has a lot of detail in the background, this is only the natural background blur from the lens/sensor in the phone. The middle picture is taken with the same phone in "portrait photography" mode. This uses AI magic to try to emulate the look of a professional camera with a large sensor with a lot of artifacts like inconsistent blurring on the background, and no blurring in reflections. The picture on the right is taken with the largest sensor camera I have, which is the APS-C size, resulting in a nice smooth background blur.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Sensor size comparison, from wikipedia</figcaption></figure> <p>It's possible to get a great APS-C sensor size digital camera for ~300 euro. But as soon as you go up in sensor size you're up into 1000+ euro. The price increases even more for going above 35mm full frame, a medium format camera will be starting at 3000 euro. This is all excluding lenses.</p> <p>For me the reason for getting into analog photography is twofold. Instead of paying thousands of euros for a larger sensor digital camera I can just get a old second hand film camera since the 35mm full frame size I want is based on the size of a normal analog film roll. This brings the price down to below 50 euro for the camera.</p> <p>The second reason is that I got sent a blank 35mm film roll, which was the nudge I needed to get the camera and get into the ecosystem.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Kodak Kodacolor 200, my first roll of film</figcaption></figure> <p>The next part of this journey is figuring out which camera to get. Thanks to practically everyone having moved to digital photography it's very easy to find old analog cameras for as low as 15 euro. An example for this would be a Nikon Nikomat, since I already have Nikon F-mount lenses it would be an obvious choice.</p> <p>Instead of going for the cheapest option I decided to go for a Nikon F90 instead. This is the newest most featureful film camera by Nikon I could find for a reasonable price. I found the F90 for 49 euro ex. shipping. There's newer models but those were more scarce and the price jumps up to 1000 euro again, I guess they have some collectors value I don't really care about.</p> <p>The main reason for picking this specific camera over a fully mechanical one is that getting into analog photography is scary when picking a fully mechanical/manual one. I get only 36 shots, no feedback until I took all the shots and the quickest way to get annoyed by doing analog photography would be getting the first developed film roll back full with out of focus and under/over exposed pictures. The F90 gives me all the manual options and has compatability with autofocus on most of my existing lenses, but it also has a full-auto point and shoot mode.</p> <p>This camera is also able to detect quite a list of faults with the film feeding and having the film roll correctly inserted which helps calm my nerves about everything working smoothly when taking pictures.</p> <p>I'm quite happy with this how this decision turned out. I'll probably also get one of the older mechanical cameras next now that I have slightly more experience.</p> <h2>The first developed pictures</h2> <p>Since this is color film and I did not want to buy a lot of tools and chemicals I sent of my film to be developed at a lab and scanned. There's multiple options to get that done. The "old" way in the Netherlands would be to go to a supermarket and have someone put it in one of the large developing machines and have some nice prints an hour later. Before that it was done by getting a special envelope at the supermarket, several supermarket chains had these, and putting the film roll in it and noting on the envelope which size you would like the photos printed at.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Return envelope with the developed film in it</figcaption></figure> <p>After sending off the envelope to a lab you'd retrieve another envelope with developed film strips, 4 or 5 pictures per strip, and instructions for ordering re-prints from specific frame numbers at different sizes or multiple copies. To actually get those prints you'd have to give back the film strips again to get them printed ofcourse. A lot of back and forth to get some pictures developed.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"><figcaption>Table in ordering more copies of a picture in one of 3 sizes</figcaption></figure> <p>This process later got modernized a bit, giving the possibility to order a photo CD instead of prints. This service is still offered to this day by some supermarket chains but it seems like this never got modernized again in the last 20 years. </p> <p>I can't get over how stupid this is. Someone is still running a full lab to develop film and scan it, and they still make picture CDs but for some reason when you order a CD which can hold 700 MB they only give compressed downscaled jpeg pictures. And with compressed I mean they could've delivered the full film roll on a 1.44MB floppy.</p> <p>This made me opt to get the film roll scanned by an actual physical photography store (they still exist!), asking to just email me the highest quality original scans. </p> <p>I got sent edited jpegs instead :(</p> <h2>The pictures</h2> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>This is the first picture on my first film roll. Well... actually the first one I intended to take. The first one is a blurry picture of my desk because I accidentally pressed the shutter release button directly after loading the film roll. I took this with a Nikkor 50mm 1:1.8D lens, the only actual full frame compatible lens I own.</p> <p>I'm pretty happy how this one turned out, I took it on the automatic mode and it's suprisingly non-blurry for how much the chickens moved around. The amount of film grain is more than I expected, but it looks pretty nice. </p> <p>Another picture I like quite a lot is this one:</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>It's not a particularly interesting scene, except that it looks like this picture was taken 30 years ago. The film gives it an immediate vintage feel, combined with the street being unusually empty due to corona restrictions. </p> <p>One interesting thing is that the nice green trees were not green at all when taking this picture, this is taken in fall and the trees were very brightly yellow with a bunch of brown trees in between.</p> <p>There's also the glaring scanning artifact in the center of the frame, this seemed to have happened on quite a lot of the pictures.</p> <p>Since I couldn't find a picture hosting system I liked and also could save metadata about analog film photography, I made my own. Here's a gallery of some of the other pictures from this film roll: <a href=""></a></p> <h2>The next roll of film</h2> <p>While having a lot of fun shooting on this first roll of film I ordered a box of filmrolls so I can continue shooting. Since my parents always used Fujifilm rolls back when everyone took pictures this way I decided to also try a Fujifilm roll next. I found the Fujicolor Pro 400H which seemed great, it had everything. The green box, the "pro" in the name. I made a slight mistake while reading the product page though.</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p>I've accidentally ordered some 120 medium format film instead of the 135 film I wanted. So it seems the way forward now is getting one of the 15 euro medium format film cameras. </p> <p>Medium format film is something I know even less about. The idea seems simple, make the film frame way larger for more detail. It seems like the vast majority of medium format cameras have fixed focus lenses though. The film scans I've seen online taken through such cameras make it look like there's way less detail than my 35mm film pictures though. The alternative is getting one of the medium format SLRs with an actual focusable lens but those are again ridiculously overpriced for a box with a hole for a lens in it. The reason for that is pretty simple: They are Hasselblad cameras so I'm paying the price for a collectors item. There's also a few alternative cameras that are pretty similar to the Hasselblad but they are also very overpriced.</p> <p>Anyway, I got a roll of Fujifilm Superia X-tra 400 after that from a nice physical photography store. It's still in my camera and almost full. Can't wait to see how it turns out!</p> <figure class="kg-card kg-image-card"><img src="" class="kg-image"></figure> <p></p> The end of the nice GTK button BraamThu, 24 Mar 2022 18:43:07 -0000<p>The above picture is a GTK3 button with the Adwaita theme. I like this button. I would even say this is by far my favorite button.</p> <p>I personally feel like Adwaita has been the pinnacle of Linux design and that the Gnome design guidelines are absolutely great. Apps are laid out in a reasonably consistent way. The purpose of every widget is very clear. The design even works very well on Linux phone formfactors.</p> <p>The design for Gnome apps has stayed fairly consistent for at least a decade, there's been some minor design changes over the years which you can easily see by opening Gnome Control Center since it seems to be in a perpetual cycle of moving some panels to a new styling but leaving other ones untouched.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>Here is an example of an older panel. Nice boxes with square corners.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>This is a newer panel. It uses the ListBox with nice rounded corners. I quite like this design.</p> <p>I think Gnome has really nailed this design. But in practice I'm using Adwaita dark on most systems. The dark theme, while not officially supported as a normal application theme, works absolutely brilliantly and is a great example of how to design a dark theme.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>Everything is still very clear in the dark variant and the general theme of Adwaita has been nicely maintained. It doesn't look like it's just an inverted color scheme for Adwaita or Adwaita with the brightness lowered.</p> <p>Thanks to Libhandy it's even possible to make nice responsive layouts that scale between desktop use and mobile use without looking out of place. The new widgets introduced with Libhandy neatly fit in with the normal GTK3 theming of widgets. I've been making use of these new widgets a lot when designing applications and I think some of these make desktop design a lot easier and should've been part of GTK3 at the start.</p> <h2>GTK 4</h2> <p>Now GTK3 is the "Old school" way to do UI. GTK4 has been in development for a bit and has improved a lot of the internals. One of the great upsides is that it can take more advantage of the GPU when rendering the UI.</p> <p>The Adwaita theme has also been nicely carried over and looks very similar to the GTK3 counterpart. </p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>This has all the features I like from GTK 3. The only issue with it is that font rendering looks horrific, but that might just be my machine. Also the dark theme is quite nice.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <h2>Libadwaita</h2> <p>So now we're in the other part of my blog post. The things I don't like. When I want to make a gtk4 application for mobile I would need libhandy, but libhandy for gtk4 is not a thing. The "solution" is libadwaita. This provides the widgets I need but it comes with the downside of having some of the worst decisions in application theming.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>This is a libadwaita application. With the new libadwaita theming it falls in the pitfalls of modern design. Everything is flat now. Borders on widgets have been eradicated! Blood for the flat design gods!</p> <p>I personally don't like the flat look. I think Microsoft was stupid for doing it. I think Apple is stupid for doing it and I've been praising Adwaita for being the sane option in an insane world. This is sadly over now.</p> <p>I don't think removing all borders and gradients counts as design. That's just lazy and completely devoid of having any personality. </p> <p>My design opinion aside, I think the theme has objective issues as well. This is just a huge decrease in accessibility. The buttons in the headerbar look the same as titles now. The only hint that New List is a button is because it happens to be a split button and has a dropdown icon. It looks identical to the window title otherwise. </p> <p>For some reason the only button controls in the headerbar with a slight hint of buttoniness are the window controls because they have a background. Ironically these are the only controls in the old theme that didn't look like buttons.</p> <p>I feel like the designers of this new theme have never sit down with anyone who's not a "techie" to explain to them how to use a computer. While a lot of people now instinctively hunt for labels that have hover effects, for a lot of people who are just not represented in online computer communities because they're just using the computer as a tool this is completely weird. I have had to explain to people tons of times that the random word in the UI somewhere in an application is actually a button they can press to invoke an action.</p> <h2>The dark theme</h2> <p>I've not been able to make the libadwaita apps on my system change theming or even switch to the dark variant, so I'll only have screenshots from around the internet.</p> <figure class="kg-card kg-image-card kg-width-wide"><img src="" class="kg-image"></figure> <p>In this dark theme the edges and divider in the theme suddenly become light. I find this really jarring and it looks like it's just an inverted theme. This is a minor complaint and otherwise the dark theme is pretty well done except for the flatness issues also present in the light theme.</p> <h2>The community</h2> <p>Now one of the worst parts is that everywhere I only even hint at not completely loving the new libadwaita theme I instantly get shut down and disagreed with before I can even get the chance to give some feedback. Apparently not liking flat themes makes me a madman in this world. Why am I not allowed to even have opinions about the look of the operating system I'm using?</p> <p>The feedback I get is that I should move to QT/KDE, but I think that theming has had the same issues for way longer already and I do really like the Gnome HIG.</p> <p>I'm also an application developer, the only choice I have is either staying on developing gtk3/libhandy apps for as long that's supported or start with making libadwaita applications which means I'll be making apps that I don't like the look of, which is incredibly depressing.</p> <p>Gnome activists have already been doing the whole "Don't theme my app" thing, why are they theming my app now? this goes both ways!</p>