The Expense of Unprotected Free Software

It's high time FOSS maintainers got a bit of appreciation

Poul-Henning Kamp

"Lasse is not available at this time. Please leave a message"

Ten years ago, my friend Philip gave me a week to come up with a closing keynote for the 2014 FOSDEM (Free and Open-source Software Developers' European Meeting) in Brussels, Belgium. I obliged and presented "Operation Orchestra."

Edward Snowden, who was charged with leaking top-secret documents in 2013, was still fresh in my mind, so my departure point was simply: "What would I do about open source if I were the director of the NSA (National Security Agency) with a billion-dollar budget?"

To make a very serious topic palatable, I asked my audience to imagine I was a junior NSA staffer sent to Brussels to brief some NATO gathering but had, by accident, wandered into FOSDEM instead.

I could have made more of that joke, and maybe I should have. Still, I think the keynote was OK. Certainly, I was satisfied when it was over. But you should judge for yourself. The video is on the Internet, and the topic is still relevant.

In fact, it was recently discovered, entirely by accident, that the almost universally used XZ compression software had been sabotaged by some nebulous but sophisticated actor, under the guise of a thin identity.

My first reaction was a severe outbreak of "IHTSITYSBITYS" — a tendency to launch into tirades of "I hate to say I told you so, but I told you so" that licensed professionals have cautioned me might be exacerbated by my advancing age and choice of profession. I try to control the urge as best I can, but there was simply no escaping it this time.

You see, what had happened to XZ was exactly what I had warned that FOSDEM audience about 10 years earlier—an audience that included hundreds of movers and shakers in the larger FOSS (free and open-source software) ecosystem.

After I calmed down, my first move was to find out if the original maintainer of XZ was someone I knew. If so, I wanted to send him a note of support. But I found no trace in my email archives, which wasn't entirely unexpected since I'm not into data compression. I then searched the fellow's name but got nothing I could use. Next, I searched for photos since my brain is equally bad at names and faces. What I found was a screen full of images of a grandfatherly type enjoying himself with his clarinet and a group of jazz buddies.

I stared, mesmerized for a time by those images, and then—out of the blue—anger hit me like a hammer striking an anvil: "I hope that is the XZ maintainer. I hope he's having a really swell time with his band right now. I hope all those people who are desperately trying to call him have only the landline number he dropped seven years ago. I hope the aliens who land their flying saucer in front of him as he makes his way home, to warn him of the impending collapse of our civilization, will be politely brushed off and told to send him an email instead. And I hope he checks his XZ mailbox only on alternate Wednesdays, whenever his wife of 35 years happens to be away at her Book-of-the-Month-Club meeting."

And why do I feel this way? Because Lasse is not our lackey. This is not his problem; it's our problem, and it has been for at least a decade now. I am glad to see that only a few people have thought otherwise, and even happier to learn that they've been told in unmistakable terms that they're wrong. I am also happy to see that people who could have reacted to the problem 10 years ago are finally getting around to it now since those aliens do have a point: This is no way to run a civilization.

Over the past couple of decades, our world has certainly transformed—to the point where washing machines no longer can start without a connection to the Internet. We've reached that point by drawing on an almost unlimited credit line extended by FOSS volunteers like Lasse. In many cases, these folks have been more than happy to forgive that debt (along with the compounded interest), but that simply masks the problem until these generous souls finally lose interest, retire, or simply die off.

Once that happens, our technical debt will rapidly soar, and that will continue unless and until some similarly trustworthy and community-minded volunteers pick up the job—under the same inherently unfair terms that their predecessors accepted.

I wish we could run a civilization in this manner, but we now live in a world where mad men stage armed incursions into other countries and wantonly kill unarmed civilians just because they can, and where organized criminals are rewarded handsomely for anything they can do to wreck computers, manipulate gullible users, and destroy infrastructure.

Which is to say we can just forget about finding any simple solutions here. We also can't invest hope in any of the technical solutions that might be proposed since this is the very definition of a political problem—bearing in mind that trust is just about the most complex concept there is.

Fixing this is going to be messy, expensive, and slow. Yet we have no choice but to take on the challenge since we now know that—with stakes this high—a saboteur must be willing to spend at least three years taking over as a maintainer of XZ in place of Lasse.

FOSS maintainers, on the other hand, can't afford to be so patient. Having seen the XZ sabotage, some of them are now taking a deep breath and thinking, "OK, I guess I won't be retiring anytime soon then..." Others are wistfully thinking, "I'd happily do this for the next 30 years, but I have a mortgage to pay." And others would be fully justified in thinking, "Maybe I should just delete my repository since nobody else seems to care."

Still, until the big guns manage to sort things out, we just have to take care of things however we can. The best we can hope for, of course, is to convince companies, institutions, and governments that it would be a really good idea to cut monthly checks for those people who maintain the software that these organizations absolutely depend upon. So, if you live in any country other than Germany, you should tell your politicians about the German "Sovereign Tech Fund" and "Prototype Fund," and ask why your country isn't doing much the same thing.

We should also nominate FOSS maintainers for official recognition. Almost all countries offer some kind of medal for "Unselfish Service to Society." By devoting some of these to FOSS maintainers, we could make them feel appreciated while raising general awareness that this area of endeavor is even a thing.

If nothing else, we can all at least convey our appreciation directly:

"Hope life is good with you, Lasse. Catch you next time!"

Poul-Henning Kamp has haunted the Unix world for 40 years and written a lot of widely used open source software, including bits of FreeBSD and the Varnish HTTP Cache. Living in Denmark with his wife, two cats, and three lawn-mower robots, he remains unconvinced that an older/wiser correlation exists.

Originally published in Queue vol. 22, no. 3—
Comment on this article in the ACM Digital Library

More related articles:

Amanda Casari, Julia Ferraioli, Juniper Lovato - Beyond the Repository
Much of the existing research about open source elects to study software repositories instead of ecosystems. An open source repository most often refers to the artifacts recorded in a version control system and occasionally includes interactions around the repository itself. An open source ecosystem refers to a collection of repositories, the community, their interactions, incentives, behavioral norms, and culture. The decentralized nature of open source makes holistic analysis of the ecosystem an arduous task, with communities and identities intersecting in organic and evolving ways. Despite these complexities, the increased scrutiny on software security and supply chains makes it of the utmost importance to take an ecosystem-based approach when performing research about open source.

Guenever Aldrich, Danny Tsang, Jason McKenney - Three-part Harmony for Program Managers Who Just Don't Get It, Yet
This article examines three tools in the system acquisitions toolbox that can work to expedite development and procurement while mitigating programmatic risk: OSS, open standards, and the Agile/Scrum software development processes are all powerful additions to the DoD acquisition program management toolbox.

Jessie Frazelle - Open-source Firmware
Open-source firmware can help bring computing to a more secure place by making the actions of firmware more visible and less likely to do harm. This article’s goal is to make readers feel empowered to demand more from vendors who can help drive this change.

Marshall Kirk McKusick, George V. Neville-Neil - Thread Scheduling in FreeBSD 5.2
A busy system makes thousands of scheduling decisions per second, so the speed with which scheduling decisions are made is critical to the performance of the system as a whole. This article - excerpted from the forthcoming book, “The Design and Implementation of the FreeBSD Operating System“ - uses the example of the open source FreeBSD system to help us understand thread scheduling. The original FreeBSD scheduler was designed in the 1980s for large uniprocessor systems. Although it continues to work well in that environment today, the new ULE scheduler was designed specifically to optimize multiprocessor and multithread environments. This article first studies the original FreeBSD scheduler, then describes the new ULE scheduler.