Linux hardware projects are made or broken by their community support. PINE64 has made some brilliant moves to build up a mobile Linux community, and has also made some major mistakes. This is my view on how PINE64 made the PinePhone a success, and then broke that again through their treatment of the community.

I want to start by pointing out that this is me leaving PINE64 and not the projects I'm involved in like postmarketOS. These opinions are my own yadda yadda...

Community Editions and the PinePhone's early life

The original PinePhone was brought up on the existing Linux Mobile projects like Ubuntu Touch, postmarketOS, and Maemo Leste, and also spawned new Linux distributions like Mobian and Danctnix ARM. This grew until there were 25 different projects working on the PinePhone — an apparently thriving community.

Following the initial set of Developer Editions, intended for community hackers to build the software with, the first consumer-targeted PinePhone devices were the Community Editions. A batch of PinePhones was produced with a special branded back cover for five community projects: UBPorts, postmarketOS, Mobian, Manjaro, and KDE Plasma Mobile. Every Community Edition phone sold also sent $10 to their respective Linux distribution projects.

Working together through these Community Editions, Linux distributions built a software ecosystem that works pretty well on the PinePhone.

The end of community editions

In February 2021, PINE64 announced the end of the community editions. At this moment, PINE64's focus shifted from supporting a diverse ecosystem of distributions and software projects around the PinePhone to just supporting Manjaro Linux alone.

The fact that a useful software ecosystem for the PinePhone exists at all is thanks to the diverse strategy employed by PINE64 in supporting many distributions working together on each of the many facets of the software required. Much of the original hardware bring-up was done by Ubuntu Touch. Mobian developers built the telephony stack via their eg25-manager project. And in my role for the postmarketOS distribution, I developed the camera stack.

Manjaro Linux has been largely absent from these efforts. The people working on many of the Linux distributions represented in the community editions tend to work not just on packaging software, but on building software as well. This is not the case for Manjaro, which focuses almost entirely on packaging existing software. Supporting Manjaro has historically done very little to facilitate the development of the software stack which is necessary for these devices to work. In some cases the Manjaro involvement actually causes extra workload for the developers by shipping known broken versions of software and pointing to the developers for support. Which is why https://dont-ship.it/ was started.

Regardless, Manjaro is now the sole project endorsed and financially supported by PINE64, at least for the Linux capable devices. As a consequence it has a disproportionate level of influence in how PINE64 develops its products and manages the ecosystem.

The last straw

With community members influence in PINE64 diminished in favor of a Manjaro mono-culture, what was once a vibrant ecosystem has been reduced to a bunch of burnt-out and maligned developers abandoning the project. The development channels are no longer the great collaboration between various distributions developing PinePhone components and there are now only a small number of unpaid developers working on anything important. Many of PINE64's new devices, such as the PinePhone Pro, PineNote, and others, have few to no developers working on the software — a potential death blow for PINE64's model of relying on the community to build the software.

Everyone has had a different "last straw". For me, it was the SPI flash situation.

There is a substantial change to booting between the PinePhone and PinePhone Pro. Previously, each distribution could develop a self-contained eMMC or microSD card image, including a compatible bootloader and kernel distribution. Installation is as simple as flashing a microSD card with the desired distribution and popping it in.

On the PinePhone Pro, the hardware works differently: it prefers to load the bootloader from the eMMC instead of the microSD. This means that when the PinePhone Pro shipped from the factory with Manjaro on the eMMC it will always boot the Manjaro u-Boot, even when booting from a microSD card. We no longer have any control over the bootloader for these devices.

There is a solution, however. The hardware can have an SPI flash chip that gives a bit of storage to put U-Boot in and that storage is always preferred over the eMMC and microSD storage. The problem with this is that all the distributions need to agree on a U-Boot build to put in there, and agree to never overwrite it with a distribution-specific version.

The solution to this is Tow-Boot: a distribution of U-Boot that can be put in that flash chip. With this the U-Boot firmware can just be treated like system firmware and be updated through fwupd independent of what distributions ship. This would work not only for the PinePhone Pro, but would also enable things like installing your preferred Linux distribution on a PineBook Pro by popping in a flash drive with a UEFI installer, much like you can on any other laptop.

Negotiating this solution was hell. Manjaro is incentivized not to agree to this, since it cedes their sole control over the bootloader, and PINE64 listens to Manjaro before anyone else. Furthermore, PINE64 does not actually want to add SPI flash chips to their hardware. Apparently, there has been some issues with people using SPI flash as RW storage on the A64-LTS boards, which would be a support issue.

After months of discussions between the community, Manjaro, and PINE64 leadership, we finally were able to convince them to ship the PinePhone Pro with an SPI flash chip with Tow-Boot installed on it.

But the Pinebook Pro has a similar boot configuration, and thus a similar problem. Some time after the PinePhone Pro was shipped, it was time for a new Pinebook Pro batch, and this discussion started again. The same arguments were re-iterated by all sides all over again, and the discussion went nowhere. PINE64 representatives went so far as to say, quote, "people who want [an SPI chip] can just solder one on". This batch of Pinebook Pros has ended up shipping without Tow-Boot flashed.

So I left

This is the moment I left. I left all the official channels, stepped down as PINE64 moderator. Left the private developer chat rooms. PINE64 cares only about Manjaro, and Manjaro does not care about working with any other distributions. This is no longer a community that listens to software developers. As a representative of postmarketOS, there is no further reason for me to be directly involved with PINE64 if the only opinions that matter are those of Manjaro.

Like many others, I have become burnt out on this ecosystem. So I quit. I am no longer getting random requests to make Manjaro's factory software work. No longer am I enduring the stress and frustration of these meaningless discussions behind the scenes, and after not being in the PINE64 for some weeks I can definitely say I'm way less stressed out.

Now I can just focus on making postmarketOS work better. On the PINE64 hardware, and all the many other devices supported by postmarketOS.

I hope that future vendors will make better choices, and listen to the actual whole community. Maybe even help with the software development side.