The March/April issue of acmqueue is out now

acmqueue is free for ACM professional members. Non-members can purchase an annual subscription for $19.99 or a single issue for $6.99.

Download the app from iTunes or Google Play,
or view within your browser.

More information here

The Bike Shed


  Download PDF version of this article PDF

A Generation Lost in the Bazaar

Quality happens only when someone is responsible for it.

Poul-Henning Kamp

Thirteen years ago, Eric Raymond's book The Cathedral and the Bazaar (O'Reilly Media, 2001) redefined our vocabulary and all but promised an end to the waterfall model and big software companies, thanks to the new grass-roots open source software development movement. I found the book thought provoking, but it did not convince me. On the other hand, being deeply involved in open source, I couldn't help but think that it would be nice if he was right.

The book I brought to the beach house this summer is also thought provoking, much more so than Raymond's (which it even mentions rather positively): Frederick P. Brooks's The Design of Design (Addison-Wesley Professional, 2010). As much as I find myself nodding in agreement and as much as I enjoy Brooks's command of language and subject matter, the book also makes me sad and disappointed.

Thirteen years ago also marks the apogee of the dot-com euphoria, where every teenager was a Web programmer and every college dropout had a Web startup. I had genuine fun trying to teach some of those greenhorns about the good old-fashioned tricks of the trade—test-restoring backups, scripting operating-system installs, version control, etc. Hindsight, of course, is 20/20 (i.e., events may have been less fun than you remember), and there is no escaping that the entire dot-com era was a disaster for IT/CS in general and for software quality and Unix in particular.

I have not seen any competent analysis of how much bigger the IT industry became during the dot-com years. My own estimate is that—counted in the kinds of jobs that would until then have been behind the locked steel doors of the IT department—our trade grew by two orders of magnitude, or if you prefer, by more than 10,000 percent.

Getting hooked on computers is easy—almost anybody can make a program work, just as almost anybody can nail two pieces of wood together in a few tries. The trouble is that the market for two pieces of wood nailed together—inexpertly—is fairly small outside of the "proud grandfather" segment, and getting from there to a decent set of chairs or fitted cupboards takes talent, practice, and education. The extra 9,900 percent had neither practice nor education when they arrived in our trade, and before they ever had the chance to acquire it, the party was over and most of them were out of a job. I will charitably assume that those who managed to hang on were the most talented and most skilled, but even then there is no escaping that as IT professionals they mostly sucked because of their lack of ballast.

The bazaar meme advocated by Raymond, "Just hack it," as opposed to the carefully designed cathedrals of the pre-dot-com years, unfortunately did, not die with the dot-com madness, and today Unix is rapidly sinking under its weight.

I updated my laptop. I have been running the development version of FreeBSD for 18 years straight now, and compiling even my Spartan work environment from source code takes a full day, because it involves trying to make sense and architecture out of Raymond's anarchistic software bazaar.

At the top level, the FreeBSD ports collection is an attempt to create a map of the bazaar that makes it easy for FreeBSD users to find what they need. In practice this map consists, right now, of 22,198 files that give a summary description of each stall in the bazaar—a couple of lines telling you roughly what that stall offers and where you can read more about it. Also included are 23,214 Makefiles that tell you what to do with the software you find in each stall. These Makefiles also try to inform you of the choices you should consider, which options to choose, and what would be sensible defaults for them. The map also conveniently comes with 24,400 patch files to smooth over the lack of craftsmanship of many of the wares offered, but, generally, it is lack of portability that creates a need for these patch files.

Finally, the map helpfully tells you that if you want to have www/firefox, you will first need to get devel/nspr, security/nss, databases/sqlite3, and so on. Once you look up those in the map and find their dependencies, and recursively look up their dependencies, you will have a shopping list of the 122 packages you will need before you can get to www/firefox.

Modularity and code reuse are, of course, A Good Thing. Even in the most trivially simple case, however, the CS/IT dogma of code reuse is totally foreign in the bazaar: the software in the FreeBSD ports collection contains at least 1,342 copied and pasted cryptographic algorithms.

If that resistance/ignorance of code reuse had resulted in self-contained and independent packages of software, the price of the code duplication might actually have been a good tradeoff for ease of package management. But that was not the case: the packages form a tangled web of haphazard dependencies that results in much code duplication and waste.

Here is one example of an ironic piece of waste: Sam Leffler's graphics/libtiff is one of the 122 packages on the road to www/firefox, yet the resulting Firefox browser does not render TIFF images. For reasons I have not tried to uncover, 10 of the 122 packages need Perl and seven need Python; one of them, devel/glib20, needs both languages for reasons I cannot even imagine.

Further down the shopping list are repeated applications of the Peter Principle, the idea that in an organization where promotion is based on achievement, success, and merit, that organization's members will eventually be promoted beyond their level of ability. The principle is commonly phrased, "Employees tend to rise to their level of incompetence." Applying the principle to software, you will find that you need three different versions of the make program, a macroprocessor, an assembler, and many other interesting packages. At the bottom of the food chain, so to speak, is libtool, which tries to hide the fact that there is no standardized way to build a shared library in Unix. Instead of standardizing how to do that across all Unixen—something that would take just a single flag to the ld(1) command—the Peter Principle was applied and made it libtool's job instead. The Peter Principle is indeed strong in this case—the source code for devel/libtool weighs in at 414,740 lines. Half that line count is test cases, which in principle is commendable, but in practice it is just the Peter Principle at work: the tests elaborately explore the functionality of the complex solution for a problem that should not exist in the first place. Even more maddening is that 31,085 of those lines are in a single unreadably ugly shell script called configure. The idea is that the configure script performs approximately 200 automated tests, so that the user is not burdened with configuring libtool manually. This is a horribly bad idea, already much criticized back in the 1980s when it appeared, as it allows source code to pretend to be portable behind the veneer of the configure script, rather than actually having the quality of portability to begin with. It is a travesty that the configure idea survived.

The 1980s saw very different Unix implementations: Cray-1s with their 24-bit pointers, Amdahl UTS mainframe Unix, a multitude of more or less competently executed SysV+BSD mashups from the minicomputer makers, the almost—but not quite—Unix shims from vendors such as Data General, and even the genuine Unix clone Coherent from the paint company Mark Williams.

The configure scripts back then were written by hand and did things like figure out if this was most like a BSD- or a SysV-style Unix, and then copied one or the other Makefile and maybe also a .h file into place. Later the configure scripts became more ambitious, and as an almost predictable application of the Peter Principle, rather than standardize Unix to eliminate the need for them, somebody wrote a program, autoconf, to write the configure scripts.

Today's Unix/Posix-like operating systems, even including IBM's z/OS mainframe version, as seen with 1980 eyes are identical; yet the 31,085 lines of configure for libtool still check if <sys/stat.h> and <stdlib.h> exist, even though the Unixen, which lacked them, had neither sufficient memory to execute libtool nor disks big enough for its 16-MB source code.

How did that happen?

Well, autoconf, for reasons that have never made sense, was written in the obscure M4 macro language, which means that the actual tests look like this:

## Whether `make' supports order-only prerequisites.
AC_CACHE_CHECK([whether ${MAKE-make} supports order-only prerequisites],
  [mkdir conftest.dir
   cd conftest.dir
   touch b
   touch a
cat >confmk << 'END'
a: b | c
a b c:
       touch $[]@
  touch c
  if ${MAKE-make} -s -q -f confmk >/dev/null 2>&1; then
  cd ..
  rm -rf conftest.dir
if test $lt_cv_make_order_only = yes; then

Needless to say, this is more than most programmers would ever want to put up with, even if they had the skill, so the input files for autoconf happen by copy and paste, often hiding behind increasingly bloated standard macros covering "standard tests" such as those mentioned earlier, which look for compatibility problems not seen in the past 20 years.

This is probably also why libtool's configure probes no fewer than 26 different names for the Fortran compiler my system does not have, and then spends another 26 tests to find out if each of these nonexistent Fortran compilers supports the -g option.

That is the sorry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a clueless generation of IT "professionals" who wouldn't recognize sound IT architecture if you hit them over the head with it. It is hard to believe today, but under this embarrassing mess lies the ruins of the beautiful cathedral of Unix, deservedly famous for its simplicity of design, its economy of features, and its elegance of execution. (Sic transit gloria mundi, etc.)

One of Brooks's many excellent points is that quality happens only if somebody has the responsibility for it, and that "somebody" can be no more than one single person—with an exception for a dynamic duo. I am surprised that Brooks does not cite Unix as an example of this claim, since we can pinpoint with almost surgical precision the moment that Unix started to fragment: in the early 1990s when AT&T spun off Unix to commercialize it, thereby robbing it of its architects.

More than once in recent years, others have reached the same conclusion as Brooks. Some have tried to impose a kind of sanity, or even to lay down the law formally in the form of technical standards, hoping to bring order and structure to the bazaar. So far they have all failed spectacularly, because the generation of lost dot-com wunderkinder in the bazaar has never seen a cathedral and therefore cannot even imagine why you would want one in the first place, much less what it should look like. It is a sad irony, indeed, that those who most need to read it may find The Design of Design entirely incomprehensible. But to anyone who has ever wondered whether using m4 macros to configure autoconf to write a shell script to look for 26 Fortran compilers in order to build a Web browser was a bit of a detour, Brooks offers well-reasoned hope that there can be a better way.


Poul-Henning Kamp ( has programmed computers for 26 years and is the inspiration behind His software has been widely adopted as under-the-hood building blocks in both open source and commercial products. His most recent project is the Varnish HTTP accelerator, which is used to speed up large Web sites such as Facebook.

© 2012 ACM 1542-7730/12/0800 $10.00


Originally published in Queue vol. 10, no. 8
see this item in the ACM Digital Library

For more articles and columns like this, check out the latest issue of acmqueue magazine



Brendan Gregg - The Flame Graph
This visualization of software execution is a new necessity for performance profiling and debugging.

Ivar Jacobson, Ian Spence, Brian Kerr - Use-Case 2.0
The Hub of Software Development

Tyler McMullen - It Probably Works
Probabilistic algorithms are all around us--not only are they acceptable, but some programmers actually seek out chances to use them.

Kate Matsudaira - The Science of Managing Data Science
Lessons learned managing a data science research team


Displaying 10 most recent comments. Read the full list here

tony | Tue, 30 Dec 2014 14:41:15 UTC

The solution seems obvious. The author should simply hire a few dozen top-end architects, coders, QA, and process experts and task them to create a new Unix. This one will be so small and wonderful that everyone will use it.

Now, this can easily cost $7M/year for salaries alone.

The chaotic mess that is Unix was created by hordes of people largely working for free. Without them, there would be no mess because there would be no code. We'd be running pure ASCII terminals and would browse using Lynx.

Unixy bliss can be had for $7M/year. So, throw down.

Mike | Tue, 30 Dec 2014 16:17:42 UTC

As I matured as a developer (managed code on the .NET Framework) over the last few years, I found myself very often thinking about designing and implementing a new programming language, and then building a new operating system with it. Both language and OS would have many restrictions, the idea being that there will be one way to do something.

I came to this idea time and time again, whenever I got frustrated with the sort of things described in this article. I would use a tool, only to realize I would have created it just slighty different myself, and whenever I followed through on such a thought experiment, I would then see the need for my version of the tool to be built upon a solid foundation, so as to avoid re-inventing the wheel. These thought experiments would always come to the same conclusion in the end: if I want to have tools my way, I'd have to implement an OS and a programming language!

It's unfortunate I am a mediocre programmer, and a lazy one (in the sense that my day job takes all my energy for programming). I realize my ideas would take many years to implement, and that I could never do it alone, but also that if others would help, my ideas would be watered down, because as a team grows, so it becomes more difficult to work towards that singular vision.

Oh well, at least being a programmer I can sort of understand the bizar stuff you find at the bazaar. Doesn't mean I always enjoy spending time on configuration, but at least I know 'how to repair my car' so to speak.

Thanks for writing the article, and for giving me the space to share my feelings on the subject!

Wayne Manners | Tue, 30 Dec 2014 17:14:51 UTC

Yeah, allowing plebs to develop software is a disgrace. Software engineers should be rigorously licensed and made legally liable for their work the way structural engineers are.

Jeremy Allison | Tue, 30 Dec 2014 20:03:21 UTC

Faulty memory ? :-)

"Instead of standardizing how to do that across all Unixensomething that would take just a single flag to the ld(1) command"

I chuckled when I read this. Exactly *how* did the author expect this to get done across the UNIXes of the day ? By persuading Sun ? Or AT&T, or maybe the OSF ? Or possibly HP ? Or Fujitsu or or IBM or any of the other of the hundreds of warring UNIX companies, all of whom *HATED* each other and saw every single difference as an advantage for their "side", every difference as something to celebrate in making software difficult to port ?

Does the author not remember these days ? I certainly do :-).

autoconf fixed a real problem, and still does today. After all, linker flags in FreeBSD are *still* different from GNU ld, which is as close to a standard as exists these days (and I know because I have to work around them).

Adopt a GNU-derived solution on a *BSD platform ! Sacrilege ! Burn the heretic :-).

Such attitudes still shine out loud and clear from the author.

There is a reason the Microsoft Windows won the UNIX wars. We would all do well to remember that.

Orion Blastar | Tue, 30 Dec 2014 20:34:03 UTC

The Free and Open Source movement is misunderstood. It is free as in speech not as in free beer.

The Bizarre Market is more like it. The big software companies found a way to profit off free and open source projects.

Microsoft open sourced .Net as Mono and others took the bait. Then when Xamarian and Mono made progress, Microsoft buys out their company to improve Visual Studio and then makes Roslyn and open sources .Net again to see if any other developers will take the bait.

Jayson Vantuyl | Wed, 31 Dec 2014 08:25:58 UTC

I don't exactly disagree with the distaste for the reality of modern FOSS. That said, I'm not sure what alternative we really have.

What this comes down to is "control". Business has never been particularly good at exercising its control responsibly. Even your article says as much in calling out Unix falling apart due to AT&T exercising its control in its interests.

I agree that it would be nice to have more quality in Open Source. I agree this would happen if there were better people in control. Sadly, this article doesn't address how to realize that need. Indeed, it's mostly a rose-colored look back at a perfect time that didn't exactly exist in the way it's implied.

Conveniently, blame is applied with a brush from a BSD-aficionado at what happens to be the alternative GNU environments. Yeah, you don't like GNU, you want it to be "better", and you're sore that it's so popular. Well, you can't ignore that there's just *more* there.

At the end of the day, FreeBSD and friends haven't dominated because RMS and ESR were right. They relied on the numbers of the horde. Operating at a fraction of their true capacity, the Bazaar out-produced the commercial software industry where it counted.

As it turns out, this was a very, very good thing. In the absence of somebody able to responsibly exercise control outside of the corporate world,, the best we could do was to outcompete the business interests that would wield their control irresponsibly. We weren't fighting the war between quality and quantity. We were fighting the war between populist control and corporatist control.

Thank $DIETY that we won (or, at least, held some of the most important ground). Corporate domination of software doesn't necessarily increase quality any more than corporate welfare creates jobs. Quality is something to be cut by overzealous MBAs even more than it's something to be ignored by overzealous newb developers.

Frankly, that's better than the alternative. Look at Apple. I love their products and the BSD core they've built upon is respectable. That said, I wouldn't trust them without the weight of a FOSS-dominated ecosystem to keep them in check. Money does as it wants, and an Apple-only world isn't much prettier than the Microsoft one we narrowly seem have averted.

So, yeah... I get it. Quality would be nice. It requires somebody to direct it. You're right. So what? You've pointed out that the Emperor has no clothes, but you haven't told us where to find new ones.

As for a solution, I look to Python. Guido's done a great job making a solid platform by any metric that matters. That's FOSS. Python is more Bazaar than cathedral in a lot of ways--it's just Cathedral where it counts.

Similarly, the Linux kernel is not a terrible project. It's not your cathedral but it works and responds to its users in a way that FreeBSD just hasn't (or, rather, in a way that just prevented FreeBSD from acquiring a majority of users).

Once you can figure out how to replicate that kind of success (or, it seems, maybe even understand what success was to be had there), you might have something to actually contribute to the conversation. Until then, this sort of whining and pining for a fictional past is just more heat than light.

gbonehead | Wed, 31 Dec 2014 18:49:47 UTC

Ah, the UNIX-HATERS Handbook is being reincarned! :)

Igor | Sat, 10 Jan 2015 00:50:35 UTC

The author might be completely right (I don't have as many years of experience, haven't seen the UNIX in the 80's or early 90's), but I think that the solution is to fix code instead of complaining. Hacks are not going anywhere. autoconf is bad? libtool is bad? Start a project to rewrite it, dedicate some FreeBSD resources to make it better, or hell - rewrite it yourself. I would love to help, but I am a hack at writing code (IT professional with scripting).

What is definitely right is that software in general is more and more bloated and eats more and more resources. The aforementioned elegance is gone. I attribute a lot of this however to using higher and higher languages, not caring about resources.

whatever | Mon, 12 Jan 2015 01:40:08 UTC

Igor: He's not going to write a line of code to address it, because he's just a bitch--not a problem solver.

Yet arrogant tight ass pretends that he's superior to others while lamenting that the world isn't perfect. There are few things worse than snobby coders, which is what you are. You're nothing special. You haven't contributed anything that matters. Oh, you wrote some crappy code to speed up Facebook? Wow, you're really important. You're as replaceable as the people you're criticizing, and you're just as much of a code monkey as they are.

Coding is just like anything else humans do. It's a big mess. You can't clean it up without tearing it down, and nobody wants to lose 20 years of functionality that's been built because some crybaby can't handle reality. People like you are a dime a dozen and tend not to be the people who actually get things done.

Deal with it."

sciolist | Wed, 10 Jun 2015 17:40:02 UTC

Democracy is based on the assumption that a million men are wiser than one man. Hows that again? I missed something.

Autocracy is based on the assumption that one man is wiser than a million men. Lets play that over again, too. Who decides?


So I guess it comes down to which side of this argument you take. Where your bias lies, in other words. I think the key through the horns of the dilemma is being able to communicate about the things you care about; being able to admit you might, possibly, have been wrong, or even worse, right about something; being able to make the singular contribution that only you can make in the context of what might be best for the group you are working for or with. The assumption here, of course, is that everyone is always striving to be the best person that they can be. Constantly striving and discerning how they can make the best out of the situation for everyone.

But we're lazy. And greedy. And willful. And deceitful. And noble. And brilliant. And humble. And industrious. Welcome to the Human Condition. We keep trying to define normative behaviors ("should" or "ought" things) in terms of an either/or model. Maybe we should re-think that. Maybe it should be an "and" model. In this case it doesn't matter much since this is basically a blog post. But at the very least, let's be civil about it, whatever the personal bias you have on the matter. Ad hominem attacks just don't help anyone and tend to move focus away from the point at hand and into the cults of personality and rhetoric. So STFU 'whatever'.

Displaying 10 most recent comments. Read the full list here
Leave this field empty

Post a Comment:

© 2016 ACM, Inc. All Rights Reserved.