Thread by @zygoon

So I wanted to fix a one-liner bug in a package I contributed to @ubuntu. It’s also a comparative journey, as I will be comparing that to @fedora. This was a journey, or rather, it still is, because the process takes days or weeks to complete.

So, if you don’t know how this works in general. Linux Distributions take, curate and sometimes maintain various free and open source software from around the web.

Most of the time the upstream developer is not the package maintainer. Most of the time packagers specialize.

Packaging is _hard_, like open-math-problem hard. We have plenty of history and ideas people explored but systems tend to gravitate to imperfect-but-popular and novel-and-unused.

There is surprisingly little tech difference from a packager POV between the major distributions.

But there is a lot of difference in the policy each distribution enforces. This is how Debian and Ubuntu can be different, while being technically very alike.

You can think of the policy as a combination of the technical and non-technical packaging rules.

There are all kinds of rules but here we care about grouping them into two sets: those that define how a proper package looks like and those that define the life-cycle of a package.

Apart from rules there are the various systems involved. Services and tools you need to use to get the package off your machine and into the process which ultimately allows others to install it.

Those are not related to the .deb or .rpm file formats but instead to the tooling around those built by a particular distribution. The places you upload your package, the scripts and cron jobs and web services and all the other magic that takes it from there.

So back to a package. My package.

You see I’m a weird kind of upstream that also tries to be the downstream maintainer. It’s mainly because this is a pet project and not something popular where professional packages would look after a package in a particular distribution.

I do that for several reasons. Since this is a hobby I mainly do it for fun and to learn more about how distributions work in practice.

So, the small software project I’m referring to is zmk. ZMK is just a library of makefiles that make writing plain makefiles nice and productive, while providing a very rich feature set and speed (disclaimer: I’m biased)

About a week ago I realized zmk has a bug. It’s a subtle bug that involves creating a symbolic link in the wrong sub-directory. It’s also only present in specific cases.

I’ve identified and fixed the problem upstream. I’ve added tests, documented the (now past) problem in the manual page documenting symbolic links and made a new upstream release.

I followed up with packaging changes to make the new release, 0.5.1 available in Fedora and Debian (unstable).

I managed to work my way up the maintainer permission stack from sponsorship to per-package maintainer in both Debian and Fedora.

Being a per-package-maintainer means that I’m trusted enough to modify the package in a given distribution and to send updates to the rest of the online systems that ultimately may accept or refuse it based on some feedback.

In Fedora that was remarkably easy to get, because you get it automatically at the end of the new package process. In Debian it took more luck than effort, since I’m a friend of a DD who could grant me that permission.

DD is a Debian term for „Debian developer”. It’s a privilege allowing a person to maintain any package (subject to complex rules) in the system. A DD can also give a DM (debian maintainer) upload rights to a specific package.

If you don’t have those permissions, each time you want to make a change you need to seek a sponsor.

A sponsor is just someone who has the upload permissions and is willing to do the work to review your changes and upload it in your place.

Sponsorship is a lot of work, since you put your responsibility on the line and (typically) need to catch all the mistakes made be the prospective packager.

So back to my software project.

The technical fix was just a one-liner. I took some time to re-factor and simplify the particular area that caused the bug, add tests and documentation, so the final patch was much larger than the one line, but for all the good reasons.

I’ve uploaded the new upstream package to Debian. There it ends up in the terribly outdated system which apparently runs on cron jobs and perl scripts so you need to arm yourself with patience.

Using the „tracker” system I could check the status of my package as it made the journey from Debian unstable to Debian testing.

If you want to check the tracker entry of my package it is at tracker.debian.org/pkg/zmk but the contents change daily so you may not see what I refer to

Tracker aggregates information from other systems but it is clearly not event based. Everything is slow in Debian. You have to wait for one system to catch up with another. You can see that autopkg tests ran but tracker doesn’t know yet. This applies to nearly everything.

It’s like we’re in the Shire and we’re not in a hurry to do anything. Just wait a few more hours or dozens of hours for things to eventually show the current status.

In contrast Fedora is much more responsive. You make a change and it’s live and instant. This is really cool from a weekend-packager point of view because rapid feedback is just so much more rewarding and engaging.

And I ran out of tweets in a thread yaiks! To be continued :)

Fortunately I can just carry on adding more.

So back to Debian. I did my dput - the program that FTPs your source package to the Debian infrastructure and says „kthxbye” regardless of failure or success.

Debian is mail based, so any updates happen by mail (so quaint)

Fortunately this time I was lucky. My gpg key was not expired so my package was not silently dropped. I got the email response saying that the upload is accepted.

After a few hours tracker showed that information as well. It also told me my package will not be going to testing.

You see, Debian is in freeze mode. It is like an oil tanker trying to turn around. Everything is slow and full of inertia. Debian is trying to release so it’s moving slowly and carefully.

My package did not have autopkgtest - aka Debian integration tests, so it was going nowhere

Since I really care about testing and quality. I wired the existing test suite to exercise the collection of examples using the installed version of zmk and uploaded another Debian version to the system.

Several hours later I was greeted with the promising message that my package will be considered for promotion from unstable to testing in…. just 20 days.

Twenty. Days.

Yes, wizard is on time and all that but man that’s slow.

On the upside, the fixed package is available in Ubuntu Impish, because Ubuntu is not yet in freeze mode and the development release (impish) is pulling packages from Debian Unstable all the time.

In the meantime I did the similar process in Fedora.

I used a git wrapper to check out the packaging tree. Committed the new version and used the same tool to do a build in the Fedora infrastructure.

The „rawhide” or never-released always unstable version of Fedora was instantly updated to have the fix. The remaining releases too a „git merge” and „fedpkg build” to handle. Well, *almost*.

I had to send the package to bodhi first.

What? Yeah all the FOSS things are terrible on naming.

Bodhi should have been called the Fedora Update Thing and that would have been much more descriptive.

Bodhi is the system that takes a build I did remotely in the infrastructure and allows people to participate in testing

You can grab the updated package of any bodhi „transaction” (not the official term but it fits). If the package is popular many people do that (I hope).

My package is totally unused so nobody complained or cheered for several days. The package was updated automatically.

So if you are running any currently supported version of Fedora, you can dnf install zmk and have the latest and greatest software available.

Fedora differs on Policy. Debian doesn’t allow such updates so easily. I honestly don’t even remember what I’m supposed to do.

So let’s switch back to Ubuntu for a moment.

I’m way more familiar with Ubuntu so I know there is the Stable Release Update (or SRU) process that is used to update the stable distribution.

The process is complex and full of exceptions for packages where it is impractical to use

One such example is snapd which has a blanket permission to just ship new major versions at any time. Most packages cannot do that.

My package cannot do that either. I have to make a small patch that just fixes the issue and apply for a review.

This is all documented on the Ubuntu wiki (convoluted and long but accurate) but it only worked this time because an ex-colleague (disclaimer: I used to work for Canonical) just pointed me to the right place. Ubuntu developers _are_ nice so I’m sure others would get similar help

The SRU process requires one to file a Launchpad (Ubuntu bug tracker) _package_ bug. Describe everything carefully and clearly. Prepare a „debdiff” (diff between two version of a Debian source package) and subscribe the review team.

And wait

Because everything takes time.

So you wait and wait and wait.

Waiting is the crucial component of interacting with FOSS distributions. You are either trusted so you get fewer reviews of your work or you are not trusted and you are forced to wait for someone to have time to double check your changes.

Don’t get me wrong. This is good.

This is why Debian is not a cesspool of typo-squatting, bitcoin mining malware.

This process is the _only_ line of defense between „apt-get install malware” working and all the trust in a distribution collapsing.

It is not a perfect process by any means but given the two extremes: free reign in a sandbox and careful co-existence and review, there is just so much you can do.

But this system has a cost.

For a one line change I had to file this bug. bugs.launchpad.net/ubuntu/+source…

There are literally dozens of lines of text explaining all the things about the problem, all the things about the solution and the possible impact.

There’s the debdiff with all the overhead and joy of working with quilt (the thing that manages patches in a Debian source package).

The 100s of lines of text describing how this process works that everyone interested has to read.

So for a one line change, I spent closer to a day of work to get the fix across the upstream/downstream chasm, where the Users dwell.

Can the process of interacting with @ubuntu and @debian be any closer to that in @fedora without sacrificing something? I guess no, it cannot.

I wish it was though. I think it should.

If you managed to reach this point: congratulations and thank you for your attention.