Free and Open Source Software—and Other Market Failures

Open source is not a goal as much as a means to an end.

Poul-Henning Kamp

Economists and others focused on money—like my bank—have had a lot of trouble figuring out the FOSS (Free and Open Source Software) phenomenon, and eventually they seem to have reached the conclusion that it just makes no sense. So, they go with the flow.

Recently, Very Serious People in the FOSS movement have started to write long and thoughtful opinion pieces about how it has all come apart and will end in tears and regret—unless!

Allow me to disagree.

1991 and All That

Computing has scientific roots, and if it is not open source, it is not science. The only way to win a Nobel Prize for a box welded shut is if you teleport the cat out before Schrödinger's infernal device kills it.

Computing started out as open source, and as everybody knows from computer-science paper citations, almost all fundamental algorithms and a lot of other good stuff was published in CACM for all to read and use.

Commercial exploitation of scientific results is almost never open source. With few and rare exceptions, the difference between a scientist and an inventor is that the latter is not going to reveal anything unless society grants them a monopoly on the invention.

So, it was no surprise and perfectly normal that commercial exploitation of computing would also go closed source—patents and all—and for politicians to expect "the market" to sort it all out.

If and when the abstract concept economists call "the market" works, it seems to be the least awful of possibilities, but there are several kinds of market failures.

One mode of failure is that one player corners the entire market and—having eliminated all the competition—gets to decide what is good for everybody and what it should cost. The schoolbook example is Standard Oil. Only governments can fix that kind of market failure, and they are invariably too slow and too reluctant to do so.

Another subtler kind of market failure is lack of imagination and attention to customers' needs, where even though there may be competition—and even cutthroat competition—the products are just not what the customers actually want to buy.

These market failures are invariably corrected by outsiders who can see what the insiders do not. The U.S. car industry is a great example and has needed three solid whacks over the head in my lifetime:

• The first was Volvo with its "safety belts" and "crush-zones," which not only totally lacked style, but also might have given consumers the idea that cars are dangerous rather than fun.

• Next it was Toyota selling cars that just worked and kept on just working; no market survey had ever reported this as being important for potential car buyers.

• Most recently, it was Tesla with its ridiculous electric car, a technology everybody in the U.S. car industry knew to be a total dead end when what people really wanted was an affordable and street-legal M1 Abrams tank.

And that brings me to where FOSS fits in.

Some of you will remember commercial computing in the 1980s, and if so, you will remember how it utterly sucked.

IBM, the unimaginably huge monopoly, was so big that "nobody got fired for buying IBM," and it was IBM's way or no way. Then there was everybody else, sometimes referred to as "the seven dwarves," and they all wanted to be IBM instead of IBM. None of them realized that when customers asked for "anything but IBM," it was not about the letters on the nameplate but about the abuse of power.

When the Unix revolution arrived in the mid- to late-1980s, everybody would try to sell you their "open Unix computer," but everything from their product catalog to the sales force's behavior screamed "vendor lock-in." And that was on top of their products being pretty crappy and overpriced in the first place, and everybody actively and deliberately trying to be incompatible with everybody else.

Ideally, some smart company should have cashed in on that market failure, and Commodore nearly did: The CBM900 was supposed to be a good, cheap Unix computer to be sold over the counter, but for reasons of production capacity, CBM picked the Amiga instead.

The Unix market failure festered to the point where the company I worked for in 1988 plunked down money for the largest Unix computer one could buy that was not an Amdahl mainframe, only to be told by the supplier, "If you compile C programs on this computer, we cannot support you!"

Out of that utter market failure came Minix, (Net/Free/Open)BSD, and Linux, at a median year of approximately 1991. I can absolutely guarantee that if we had been able to buy a reasonably priced and solid Unix for our 32-bit PCs—no strings attached—nobody would be running FreeBSD or Linux today, except possibly as an obscure hobby.

Bill Gates would also have had a lot less of our money.

But instead of seeing the light and selling what customers wanted, the Unix vendors doubled down on their delusion of becoming IBM instead of IBM, even interpreting IBM's waning market share as proof that the winds were finally in their sails. In reality, IBM mainframe customers discovered to their dismay that their expensive 3745 front end was not so much a masterpiece of communication as a carefully engineered bottleneck, which any cheap PC could run circles around. Not long after, they also discovered that the even more expensive CPU was not good for "modern workloads" such as web pages.

Then that dot-com thing happened, fueled by the availability of FOSS operating systems, which did a much better job than any operating system you could buy, not just for the price, but in absolute terms of performance on any given piece of hardware.

Thus, out of utter market failure, was the FOSS movement born.

Sic Transit Gloria LUG

A lot of us FOSS people pay our cost of living with money we get from companies that—one way or another—make a business out of FOSS or derivatives thereof.

This is what makes some people sad, particularly those who thought they saw a glorious revolution happening and hurried up to the cathedral to nail some manifesto or other to the door. They are not at fault for thinking there was a revolution going on, because there was—just not the one they thought.

The largest Linux user group in the world was the Skåne Sjælland Linux User Group (aka SSLUG), which boasted tens of thousands of members. They held major conferences, and they influenced public policy because no politician can afford to piss off potential voters when they come in well-organized batches of thousands.

The one single thing all members of SSLUG had in common was being pissed at IT vendors who made their jobs and lives miserable. Imagine being employed as an IT person in some company that rolls out Windows Vista. Not because they needed to. Not because they wanted to. Because Microsoft forced them to.

I won't remind you how badly Windows Vista sucked; I will let author Neil Gaiman do that, by quoting what he wrote in his blog on March 31, 2008:

"Sitting in an airport, getting really sick of Windows Vista on a laptop. It doesn't work—I'm tired of waiting seconds for things to appear on the screen, of taking half a minute or more for something that ought to happen instantly to occur. The poor computer obviously can't run Vista, and shouldn't have been Vista certified, and the author isn't impressed with the way that a number of things that were easy in XP have got harder, nor with the fact it took most of a day for Vista to run a search on the hard disk to find a mislaid introduction I was working on, nor with its refusal to read or copy a bunch of files on a DVD Dave McKean burned for me (I wound up dragging them from the DVD to one of Dave's MacBooks, from there to my iPod, and from my iPod to the computer)."

Imagine being the IT person to inflict that user experience on colleagues because Microsoft forced you to. One could forge horseshoes from cold iron with that anger.

But that anger was all there was, really, as there was not much else the FOSS movement could or would agree on, and there was plenty to disagree about, which is why there are three BSDs and who knows how many Linux distros.

Now that the smoke has finally cleared, none of the old Unix vendors exist anymore, and IBM, as a sad shadow of former glory, has sought commercial asylum for its venerable trademark by buying Red Hat and handing over the steering wheel.

There is also no more SSLUG, not even a homepage: The last person to leave must have turned off the web server.

I Come Not to Praise FOSS

The F in FOSS was never silent.

In retrospect, it seems clear that open source was not so much the goal itself as a means to an end, which is freedom: freedom to fix broken things, freedom from people who thought they could clutch the source code tightly and wield our ignorance of it as a weapon to force us all to pay for and run Windows Vista.

But the FOSS movement has won what it wanted, and no matter how much oldsters dream about their glorious days as young revolutionaries, it is not coming back, because the frustrations and anger of IT in 2024 are entirely different from those of 1991.

One very big difference is that more and more people have realized that source code is a liability rather than an asset. For some, that realization came creeping along the path from young teenage FOSS activists in the late 1990s to CIOs of BigCorp today. For most of us, I expect, it was the increasingly crushing workload of maintaining legacy code bases. But the thing that will convince anyone is that one single server still runs OS version N-4, because we have not yet found out why it stops working when we attempt to upgrade it.

But we can figure it out.

And we will figure it out.

Because we have the source code.

We have all 562,227 lines of perl5 source code for it.

Poul-Henning Kamp has haunted the Unix world for 40 years and written a lot of widely used open source software, including bits of FreeBSD and the Varnish HTTP Cache. Living in Denmark with his wife, two cats, and three lawn-mower robots, he remains unconvinced that an older-wiser correlation exists.

Originally published in Queue vol. 22, no. 1—
Comment on this article in the ACM Digital Library