Comments

• William Payne | Mon, 20 Aug 2012 14:31:12 UTC

  So should we then flee to the relative oasis of sanity that is Plan 9?

• phd_student_doom | Mon, 20 Aug 2012 15:10:25 UTC

  Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the 'chaos' is what makes programming fun. 

• Matt Welsh | Mon, 20 Aug 2012 15:13:22 UTC

  Nice article. I think we all predicted this would happen when the open source "movement" started to gain momentum a decade ago. To be fair, a lot of good engineering goes into open source projects, although it's often hard to see because of the smoke screen of amateur contributions and longwinded threads on the email lists. It's not all hackers working in their spare time: Google, for example, contributes a lot of code back to open source projects (e.g., Guido van Rossum works at Google). There also seems to be a great deal of entropy at work: Witness all of the failed efforts to "clean up" BSD and Linux that never went anywhere. Chaos is king.

• Poul-Henning Kamp | Mon, 20 Aug 2012 15:25:27 UTC

  Matt,  My point is not that there are no cathedrals today, but that people don't recognize them as such or see any value in them.

• Ken Stox | Mon, 20 Aug 2012 15:48:13 UTC

  Like many other "good" ideas, the Bazaar has been over applied, and used in situations where it is a bad solution. Give a man a hammer...  That being said, there are situations where the Bazaar is a great path to take. 
  
  I would argue that UNIX began to fragment in the late 1980's when AT&T's licensing terms became more onerous than the customers were willing to accept, leading to the creation of OSF. AIX and HP/UX forked off the System V.2 codebase. The only reason SUN veered from BSD to System V.4 was because AT&T owned a substantial portion of them at the time. 
  
  I suspect that we will oscillate around the strange attractor that is the Cathedral and the other that is the Bazaar. After all, isn't that what Chaos is all about?

• iain | Mon, 20 Aug 2012 15:48:55 UTC

  Contained within this rant are some good points:
  Code reuse is good
  Removing unneeded dependancies is good
  Fixing up libtool to not check for fortran would be brilliant
  
  However, surrounding these good points are a lot of non-sequiturs and nostalgic romanticised thinkings harking back to the good ol' days when all you needed was a terminal and a |. I'm sure it was lovely back then, but these days some of us like being able to plug in our USB devices and have them automatically detected, being able to just click on the wifi network we want to connect to.
  
  So libtiff is an implicit dependency for Firefox even though Firefox cannot display tiff images, and this is an example of how badly designed the whole system is. Is it not more likely that someone somewhere (possibly the author, in his haste to show us how hardcore he was because he compiles everything from source) had forgotten to pass a --disable-tiff to one of the other 121 packages?
  
  Surely even the complaint that Firefox uses 122 other packages shows that code reuse, rather than being "totally foreign in the bazaar" is actually quite a common occurance?
  
  The bazaar (I'm assuming this is to mean the free software world) is anarchistic? From all the software projects I've worked on over the past 15 years in Free Software, the one thing they have in common is a very strong sense of hierarchy so that clearly can't be an anarchy, can they? Large distributions have smart people who are thinking about the best way to put them together, people can't just fling their code into them, so these aren't anarchies either.
  
  There is also a fair greater sense of reverence to Eric Raymond and his ramblings (the man has written articles about how he is the embodiment of the god Pan, for goodness sake) in this article than I have seen in the Free Software world. The man is a hack, anyone who has ever had to look at his code would be able to tell you he isn't very good at it. He stopped being relevant to the Free Software community circa 1999.
  
  This rant seems to be the ranting of an out of touch old hacker, who's a bit annoyed technology has got too complicated for him to understand, but rather than seeing that as the inevitable progress of technology (a modern car is considerably more complex than Henry Ford's, the modern aeroplane is more complicated than the one the Wright Bros used, and the modern computer OS is more complex than the operating systems of the early 80s), he concludes that it was better in the old days, and everyone today is a moron.

• Kevin Vilbig | Mon, 20 Aug 2012 15:54:38 UTC

  What are the cathedrals?

• philip andrew | Mon, 20 Aug 2012 15:54:51 UTC

  There are problems when building a Cathedral, a small problem in one part of the code has a tendency to be difficult to discover and fix when the system gets very large. Well, from my experience on a large C++ project, 400,000 lines of code, we found a few problems. 
  One, its very hard to debug some problems even with the core dump from the customer with 100 threads dumped, looking through the threads call stacks. 
  Two, its very hard to understand the whole thing. 
  Three, the dependancies all had to be compatible with each other.
  Four, when something broke, the whole system crashed as it was one single process.
  
  We ended up breaking the large program up into many small programs which communicated via Corba, you could insert JNI here or any other communications method. The small programs were then understandable, maintainable, testable. What they depended upon was relative small, one program did Authentication, one did Authorization, one did Document Storage.
  
  Basically the architecture changed from a single large C++ program to a broken up system.
  
  A Cathedral could work, communism could work if everyone followed the rules, if everything worked well and worked according to good rules. However, in our real world, nothing works the way we expected. Software has bugs, bugs you have never even through of. Things fail. People fail.
  
  At least in the Bazzar model I can understand the many small programs. In the Cathedral model, I cannot, at least with C++, it reaches a complexity limit for the human brain, thats also why we don't program in data-flow. With some other language and tools and rules perhaps it can be done. Perhaps ADA, that was the idea. (Well I prefer Haskell for the perfect world).

• Mark dugger | Mon, 20 Aug 2012 15:57:16 UTC

  I respectfully disagree.  The Cathedral model only works if one is capable of defining the scope of the problem completely from the outset of the program.  In systems programming, hardware doesn't change nearly as much as an organic customer.  Therefore development processes need to change given what the code is interfacing. 
  
  Poul is an excellent developer and I've enjoyed his contributions to FreeBSD and varnish, but his criticisms of the bazaar model do not hold water.  Poul argues that code quality can only be assured if quality can be accounted for easily, I would argue that code quality can only be assured if the users of the code had leverage.  In free software, the currency that the users yield is popularity and reputation, which developers can flip into fame and respect.  Code quality is hidden from the user most of the time, but the incentive that the programmer can provide the user is useful features that are released often, which may not heed the best architectural design.  In this case, the incentive for developers to write code first fulfilled the needs of the user first before engineering concerned were addressed.  
  
  What he says is true when he described the computing life as simpler once upon a time.  People once used computers for computers to do work for the most part, and now modern open source unix has to be built in a manner where computers are used both for computers and people to do work directly.  While code was simpler, many users of UNIX found the platform(s) wanting.  By the early 90s commercial UNIX was certainly written in a Cathedral model.  All of them, most certainly by virtue of a market that lacked choice given that hardware and OS were tethered, were found wanting.  
  
  His thesis of the Cathedral model being superior holds true if and only if the Cathedral in question reacts quickly and effectively to the demands of the user.  Unfortunately for the Unix world, before the rise of the bazaar model, said Cathedrals in the Unix world have only cooperated enough to turn into an oligopoly.  Before the rise of Linux and *BSD, the world had to suffer through Unix as its detractors described in the Unix Haters Handbook.  To wit, the question shouldn't be whether or not which development process gives one better code, but which one ships better products.  The Bazaar model is the development process gives one access to better user feedback as it welcomes development contributions directly, and henceforth ships successful products more often.

• Phil Regnauld | Mon, 20 Aug 2012 16:04:28 UTC

  Poul-Henning, spot on. But it's an onset of different factors, not just the "open source" movement. This perfect storm: a lack of vision, the internet, combined with a penchant for instant gratification ("TL;DR" mindset) has brought this about. You only have to read the second comment to convince yourself ("the 'chaos' is what makes programming fun"). No chance I'll be flying a plane where someone like this has contributed code :) But I think you may have another article to write on quality and responsibility. Cf. Dan's excellent talk on software testing (Flying Linux - www.usenix.org/event/lisa04/tech/talks/klein.pdf)

• Jeremy Huffman | Mon, 20 Aug 2012 16:04:49 UTC

  Maybe instead of writing a rant, you could start building the replacment for the autotools - encompassing a more declaritive shared library management system? If it worked, if there was a clear migration from the current tools and if it presented clear advantages while being simpler to use then it would eventually be adopted. Its a massive challenge; anyone who has tried to bite off a small piece of it has just made the problem worse so far, but that doesn't mean it cannot be done.

• KP Freds | Mon, 20 Aug 2012 16:10:45 UTC

  phd_student_doom, I honestly can't tell if your post is satire or not.
  
  Assuming it's not, go hang up your keyboard and stick to the amateur leagues. In the same way that an amateur woodworker should not be building houses, an amateur programmer should not be writing software.
  
  Poul Henning-Kamp doesn't address this, but a side-effect of the IT boom -- including the current VC bubble -- was that anyone that could remotely do the job was hired. This lead to inexperienced engineers having a voice and role grossly disproportionate to their abilities, and ultimately, lead to 15+ years of wheel re-invention and history repetition amongst junior engineers who lacked any adult supervision.
  
  In fact, companies such as ycombinator are fueled by and intentionally encourage youthful naivety, as they require a steady stream of heady grist for the VC mill. All it takes for them to turn tremendous profits is one successful exit out of a hundred failures.
  
  Your distasteful lack of respect for expertise and experience is the result of this lack of adult supervision.

• metageek | Mon, 20 Aug 2012 16:11:54 UTC

  This is a typical engineering point of view. Engineers like to think that the world could be designed, however Nature shows the opposite. Any sufficiently complex system cannot be built out of design, it has to be *evolved*. Evolution is messy (anarchic?): pieces that work get reused, pieces that do not are lost. There must be something good about autoconf that has enabled it to survive and eliminate those awful "designed" makefiles that needed manual tweeking even though they did not check for 15 different Fortran compilers (the status quo of the 1980s). Does it matter that autoconf looks for those now extinct compilers? No, but it gives it robustness which is why it has survived. Someday a new, less bloated, alternative to autoconf will appear and it will slowly overtake autoconf. In the meantime we are doing very well with it, thank you very much. 
  
  Software is the most complex construction that mankind has so far created. It is so complex that it can only be evolved much like Nature's products are. Engineers can continue living in their nice little artificial world of linear systems and planned devices. Those that venture in the real-world of complexity and are successful will take up Nature's ideas.
  
  Goodbye Unix of the 80s. Linux is here to stay

• rdm | Mon, 20 Aug 2012 16:15:22 UTC

  1. I agree that Eric Raymond's characterization of good software development processes leaves something to be desired.
  
  2. But I also agree with the author, here, that the "destruction of the cathedral" predates Eric Raymond's prose
  
  3. I agree that modern systems have many inefficiencies and inadequacies, and that if someone were responsible for fixing them that they would get fixed.
  
  I think, though, that if we want to understand why these inefficiencies last as long as they do, we need to look at their costs:  the incremental costs are trivial, for the people currently responsible for propagating them.
  
  (The systemic risks are much higher -- these kinds of inefficiencies are what would be called a "business opportunity" in some circles.  But there are also other "business issues" lurking in the wings.  For example: the incremental value of a lot of software is approximately the same as a [perhaps badly printed] marketing flyer, and the incremental production costs are much lower.)
  
  

• drsoong | Mon, 20 Aug 2012 16:32:25 UTC

  > Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the 'chaos' is what makes programming fun.  -- phd_student_doom
  
  From this attitude, I can see that you're never going to be a good programmer. Other people are going to end up having to fix your crap. Typical attitude for "rockstar" programmers who lack the skill, intelligence, or foundation to be decent software engineers. You'd better find a different line of work.

• Fazal Majid | Mon, 20 Aug 2012 16:38:59 UTC

  I have my own set of makefiles and patches, grown organically over the last 12 years to do something similar to FreeBSD's ports, except on Solaris and OS X. One of the reasons why it takes a full day for PHK to rebuild is the shocking lack of parallelism. Only 215 out of 578 packages build properly when GNU make's -j flag for parallel compiles is used, and even then, more often than not, more than half the total build time is spent in the non-parallelizable "configure" step, limiting your opportunities to parallelize by Amdahl's law.
  
  The open source ecosystem cannot be a soviet-style planned economy, some duplication of features by competing projects is inevitable, and probably even desirable as competition fosters innovation. The question is when too much is too much, as with the proliferation of crypto libraries or crackpot build automation tools.

• KP Freds | Mon, 20 Aug 2012 16:47:20 UTC

  metageek, sir, that is a load of bollocks: a simple-minded excuse for compounding ill-considered decisions.
  
  You don't have to look far for better examples of engineering decision making.
  
  As a side note, this may be the first time I've seen someone defending autoconf in the past 15 years.

• Poul-Henning Kamp | Mon, 20 Aug 2012 16:52:15 UTC

  @Iain: "The bazaar (I'm assuming this is to mean the free software world)  [...]"  This clearly proves that you are one of the lost generation:  You have no idea what a cathedral is in the first place.
  
  @philip andrew, @Mark and others: I'm not arguing that cathedrals is a better solution, they are certainly not without their own kind of problems.  What I'm pointing out that is people like iain don't even know what they are in the first place.
  
  @Jeremy Huffman: I have only a small calendar and must live with it.
  
  @metageek: You're _totally_ missing the point.

• Jacob Hallén | Mon, 20 Aug 2012 16:54:10 UTC

  The *BSD distributions are legacy code. To maintain an operating system distribution you need a massive effort. While nobody does it perfectly, there are some organizations in the Linux sphere that do quite a good job. Debian/Ubuntu for instance.
  The fact that your Firefox build-depends on 122 different packages is not strange. To build some graphics libraries that are prerequisites for building Firefox, you will need TIFF support, even though Firefox doesn't support TIFF.
  The way around this problem for a user is to not build the entire software suite, but instead use binary packages from a trusted source.
  
  To think that the problems of distribution management at this scale can be solved by up-front design is very naïve.

• David Kastrup | Mon, 20 Aug 2012 16:56:12 UTC

  At Mark Dugger: "I respectfully disagree.  The Cathedral model only works if one is capable of defining the scope of the problem completely from the outset of the program.  In systems programming, hardware doesn't change nearly as much as an organic customer.  Therefore development processes need to change given what the code is interfacing."
  
  You confuse the Cathedral/Bazaar dichotomy, which is about continuity of the executive, with working to plan, which is about continuity of targets.
  
  In fact, combining changing targets with changing responsible programmers is a recipe for cruft: they don't have the knowledge which code pieces have been tailored to different targets and should be replaced, rewritten, or removed.  So some idioms may stay around when they are no longer a good match to reality.
  
  For example, take a look at the original fork system call: why was this a sensible design choice?  Because it had a very elegant implementation sitting well with the memory subsystem: fork just swapped out the current process but proceeded without marking the in-memory copy invalid.  In the time of demand-paging, this idiom makes rather little sense as a primitive.

• Tom Alexander | Mon, 20 Aug 2012 17:13:21 UTC

  I'd also recommend reading this much shorter rant from him about Autotools. https://www.varnish-cache.org/docs/2.1/phk/autocrap.html

• metageek | Mon, 20 Aug 2012 17:14:44 UTC

  @Poul-Henning Kamp: I thought the point was whether you can plan the entirety of the construct versus building it in an evolutionary way. My argument is: 
   1) it is impossible to plan the entirety of a complex system at the outset, 
   2) most software is indeed sufficiently complex to fit point 1
   3) evolutionary construction is the solution: smaller parts of the system are assembled through objective criteria
   4) code bloat that is not harmful will last for a long time
  
  if someone cared to remove the fortran checks, autoconf would not work much more efficiently than it does now, so leaving it in is fitness neutral (which is why it is still there, since it costs energy to remove it)
  
  Apologies if I am still missing the point ;)
  
  @KP Freds: I am not defending autoconf, just explaining why it has so far dominated. I am not making excuses for not believing that classical engineering approaches do not scale up to large and nonlinear constructions. Nature keeps showing how it is done, and while it is messy and chaotic, it is incredibly efficient and robust. 
  
  I have cursed against autoconf many times too, but the energy to go around its problems is orders of magnitude less than to make an "efficient" alternative.
  
  

• maht | Mon, 20 Aug 2012 17:26:33 UTC

  Talking of plan9, we have a saying, coined by Boyd Roberts, "this is not Linux, there are rules".
  
  Dave Presotto also graced us with "Linux, by amateurs, for amateurs".
  
  Which, I think, summarises the spirit of your post.
  
  Plan9, where even #ifdef is eschewed.

• Poul-Henning Kamp | Mon, 20 Aug 2012 17:28:39 UTC

  metageek: cathedrals can evolve, many of them did, some of them have changed major style three times or more.  Some of them evolve to this day.  And those are built out of stone :-)
  
  My point really is that Brooks has written a wonderful book, but a lot of people are simply not equipped with the necessary mental models and background knowledge to understand what The Old Man is talking about.

• Poul-Henning Kamp | Mon, 20 Aug 2012 17:30:00 UTC

  If plan9 had been available in anywhere near runnable form in 1992, I wouldn't have spent two decades on FreeBSD.  I actually asked and couldn't get a license.

• Steve | Mon, 20 Aug 2012 17:36:12 UTC

  @Matt Welsh: Funny you should mention Google. They are far from innocent, having built their own Makefile generator known as "gyp", which have in turn their own portability problems, and often bundling dependencies along with their source, not updating software to work with newer versions of those dependencies. These things happen out of necessity though. By definition, "hacks" are things that accomplish the short term goal in the short term. PHK rightly laments the long term cost. Whether you are talking about software or business, the short term goal is usually the winner. Twas ever thus...

• Shawn Garbett | Mon, 20 Aug 2012 17:48:31 UTC

  I view the problem is that the chance of project failure increases with scope. This causes most "cathedral" designs to fail. Address this key issue and the bazaar will move into the background. It'll be like the discovery of steel impacting on the architecture of buildings if you succeed. Till then the market consists mostly of yurts and tents.
  
  This said, I wouldn't run national security, a mars probe or space shuttle on bazaar code. I.e., how important is the project? Important enough there will be an investment of time/money to solve the design issues?

• Eric S. Raymond | Mon, 20 Aug 2012 17:52:05 UTC

  There are a number of ways in which this rant is defective.  One of them is mistaking a local problem for a global one; yes, autoconf is a pile of festering crocks, but the step where this demonstrates that the bazaar model is a failure is just a handwave and an assertion.
  
  But the largest failure is in the unstated assumption that there is an alternative - that at the scale of Linux or *BSD you can have anything *but* what  Kamp dismisses as "chaos".  It is worth remembering that these artifacts are orders of magnitudes larger in LOC and other measures of complexity than the successful cathedrals of programming legend.  They are not just operating systems in the traditional sense but application suites of unprecedented size.
  
  Nothing this vast can ever have a single person responsible for overall quality, because that task is beyond the cognitive capability of any individual human being.  The implicit alternative Kamp is proposing is impossible - it doesn't scale.
   

• iain | Mon, 20 Aug 2012 17:53:09 UTC

  @Poul-Henning Kamp: Great come back. I was going to leave it, but no, I won't. I know perfectly well what the cathedral is, however none of your complaints stem from any difference between the two models, and rather stem from your belief that all systems should be simple enough for you to understand and build in under a day. For example, I'm pretty certain your libtiff complaint was your own fault and if you had passed the correct options to the configure file you would have not needed to build it.
  
  None of the good points in your rant are anything new, people have been asking why libtool needs to check for fortran for years, people have been reusing code and trying to reduce dependencies for years.

• Poul-Henning Kamp | Mon, 20 Aug 2012 18:01:05 UTC

  @ESR:  Your reading of my piece is highly defective :-)
  
  I'm not saying the bazaar method is defective, I'm saying it is so "successful" that most people don't even know what a cathedral is anymore.
  
  Try checking the comments in hackernews or reddit, people are forcefully making that point for me :-)

• Poul-Henning Kamp | Mon, 20 Aug 2012 18:02:41 UTC

  @iain:  I am pretty sure my piece includes this text: "This is a horribly bad idea, already much criticized back in the 1980s when it appeared,"  :-)

• RORVI | Mon, 20 Aug 2012 18:07:41 UTC

  Let me ask you something ....Why do you think these "cathedral breaking programmers" appeared ? Is it because people like you oftenly refuse to answer a "dumb question" ? Or is it because books like "Teach yourself C++ in 24 hours" appeared ? When was the last time you answered a question of one of these "dumb programmers" ? The lack of consistency in coding nowadays can also be the result of a very fast growing industry,that needs more,better code that is written very fast.Also,nobody mentors nowadays.I have yet to find somebody to spend some of their time to answer a question or two and give some advice.(i myself am 18,and i started programming 2 years ago !).So one of the most painful questions is : where are the mentors,the experienced programmers and ,for example,a website where they can select beginners in order to mentor them(this should be done for free,or for symbolic prices,not for 200$ a course,since in some countries the average salary is low-in my country is 300 $).So,in the conclusion,i would like to point out the fact that you ask for better,but you don't do anything to help shape the experts that will create great software.

• Andre | Mon, 20 Aug 2012 18:11:00 UTC

  "Patches welcome."
  
  The sound of crickets that come after those words is the problem.  Not the Baazar, Cathedral, Developer, Engineer or Hack.
  

• David F. Skoll | Mon, 20 Aug 2012 18:14:35 UTC

  Your article was interesting and thought provoking.  You are right: Open-source systems like Linux are a mess for developers.  They are insanely inconsistent.
  
  But like democracy, the system is the worst system except for all the others.  Proprietary systems like Microsoft Windows have plenty of API inconsistencies, weird API warts, magical and mysterious registry settings, etc.
  
  I think these system simply reflect human nature, which is to be wildly creative but (alas) inconsistent, lazy and messy.
  

• Poul-Henning Kamp | Mon, 20 Aug 2012 18:20:48 UTC

  @RORVI:
  
  I think you pinpoint a lot of the auxillary causes correctly.
  
  Please notice that I don't blame all the dot-com wunderkinds for grabbing opportunity, I merely note that it was a catastrophy for CS/IT as a discipline that they got the opportunity in the first place.
  
  With respect to mentoring, I have mentored a fair number of people myself, most of them FreeBSD committers, and I have always done it for free.
  
  But I have to make a living, so I can't mentor everybody who wants me to.
  
  I also have to be pretty ruthless to my mailbox, and I hate that, but I cannot spend 2-3 hours a day on email that doesn't put bread on my table.  You'd be surprised at the number of emails I get from students whenever a professor gives homework related to code I have written...
  
  So all in all, I don't feel that your claim that "I don't do anything" is justified.
  
  But I can certainly understand your frustration:  I started out in a back-water country, with no net-access, and had to teach myself everything from books I had to order home from abroad, at great expense.  Mentors were nowhere to be had.

• rayohauno | Mon, 20 Aug 2012 18:28:56 UTC

  @metageek beautiful words !!! I fully agree.

• anon | Mon, 20 Aug 2012 18:42:23 UTC

  "as opposed to the carefully designed cathedrals of the pre-dot-com years"
  Right there in paragraph 6 is the flaw in your thesis.
  Neither Cathedral nor Bazaar entails Quality. 

• CuriousWayne | Mon, 20 Aug 2012 18:44:50 UTC

  It's been awhile, but some cobwebs shook loose while reading this post.  The Cathedral and the Bazaar was presented as a metaphor for centralized control of code in the Data Center (AKA the priesthood of Data Processing, hence, a Cathedral) versus distributed programming (AKA Linux).  
  
  From the first reading, I sensed a weakness in the metaphor; while the Data Center as Cathedral works OK, Linux was not a Bazaar.  Linus Torvalds was King of Linux and dictated what got in and what did not for the longest time.  Linus was a single programmer scratching his own itch for a usable, affordable OS (the most effecient and effective programming team possible).
  
  Furthermore, the DotCom bubble was not a Bazaar. Bazaars are competitive but predictable; it is highly unlikely for someone with too much money to come into a Bazaar and buy out all of the vendors' stuff.  
  
  Yet that was exactly how a Silicon Valley VC, after the bubble burst, described to me why the bubble happened in the first place; too much money was chasing too few opportunities, and the VC community lost their quality controls on their investment process.
  
  I have respect for Mr. Kamp and his perspective on software engineering.  I have respect for the modern paradigm of Web programming that breaks a simple idea like a website into a billion moving, dependent parts with specialists running all over the place.  The two perspectives are somewhat mutually exclusive (or similar) in that they are solving different technical problems using coding techniques.
  
  Having worked in other engineering disciplines, none have the challenges or the leverage that software has.  Software has no universal and unambiguous design language, nor is it likely to have one anytime soon (stuff changes too fast; the design language is more dependent on the makeup of the team than the problem). 
  
  By contrast, any junior engineer in any other discipline (anyone know of an exception to this?) can spec a design and get it made on the other side of the planet by people that share nothin else than a universal design language.
  
  Here's what I think is a more apt metaphor for today vs legacy than the cathedral and the bazaar:  if you had a performance problem in legacy days, you might get fired if you 'threw hardware at it'.  If you have a performance problem today, you might get fired if you don't, because hardware is now often cheaper than paying to tune code.

• Poul-Henning Kamp | Mon, 20 Aug 2012 18:45:02 UTC

  @anon:
  
  Where do you see "quality" in the text you quote ?
  
  Just because something is carefully designed doesn't mean that it is good.
  
  See "galloping gertie" for just one of many examples of the opposite.

• Poul-Henning Kamp | Mon, 20 Aug 2012 18:47:53 UTC

  Damn, I hate it when I remember an even better example seconds later:
  
  @anon:  Best example:  The Hubble telescopes main mirror:  The mirror that was most precisely ground wrong, ever.

• Tommy McGuire | Mon, 20 Aug 2012 18:53:45 UTC

  PHK is said to have been programming computers for 26 years. I would have thought he would have remembered the '80's and '90's at least as well as I do, but I could be wrong. I also thought that the whole point of the BSD effort was to build software that anyone could do anything they wanted do with; certainly IBM did that, when they created an ld(1) that required a good deal more than one flag to build a shared library, among other things.
  
  Should standards have done something about that? Sure, but standards are not handed down by the Almighty, or even by Fred Brooks. They're built by people with many agendas, including ensuring that the standard does not inconvenience their market share. (This might make PHK smile; it does for me: the memory of the environment variable POSIX_ME_HARDER.)
  
  Now, I suppose you could blame the hideous environment that resulted in the terrible solution of the autotools on the bazaar, in that AT&T did commercially license Unix in the early '90's and thereby allow more people to soil it's pristine beauty. But to say that was "robbing it of its architects" seems to me to be a bit too much. I knew some of the people working on AIX at IBM at that time, and they were all very good, and all as dedicated to the beauty of the cathedral as anyone you could put inside any fortified wall you want to build. But, heck, if I really wanted to pinpoint the moment Unix began to fragment, I might have to go back to the seventh edition, which did not have the odd proliferation of system calls that includes socket(2).
  
  So, if any wunderkinder really deserve blame, they are probably not kinder any more. And since the bazaar is so bad, remember that there are cathedrals out there for you to worship at; they must be better. Right?

• Paul W. Homer | Mon, 20 Aug 2012 18:57:00 UTC

  You are right. The stuff we have has been degenerating for years, and each new wave of kids blows us off because they lack the knowledge to really understand the technologies that they are wielding or creating. I also think that it's gotten significantly worse in the last decade, that we've right in the middle of a 'dark ages' that is dragging us backwards.
  
  At first I just ranted but nobody wants to listen, they're too busy re-writing the same wheels over and over again. These days when I blog I try to be more positive, but not always:
  
  http://theprogrammersparadox.blogspot.ca/2011/05/software-virgin.html
  
  It's all very disappointing, and the trend has been for it to get worse. I think it's going to be a long time (generations) before the winds change, people tend to wear blinders when exposed to too much complexity, which forms a vicious cycle. 
  
  Paul.
  PS. I miss Unix too, my favorite version was a pure, clean, Sys V from AT&T in the early nineties. 

• Poul-Henning Kamp | Mon, 20 Aug 2012 19:03:59 UTC

  @Tommy McGuire:
  
  I have it from first hand, that people at Bell Labs found the socket API hideous, and the SVID IPC and STREAMS was even less to their liking.
  
  With respect to standards, yes: The UNIX-vendors shot themselves (and us) in the foot bigtime, by photocopying the SVR manuals into "the UNIX standard" (how ever many there were of those), rather than look at what the future would require of them.  That's why we still call N system calls just to open a TCP socket connection to somewhere, rather than a convenient single function in a well designed API.
  
  But the lost dot-com generation is an entirely different issue:  They have essentially wiped out all CS history before 1990, by never bothering to read any of it, and that means we have to slowly relearn what we once knew about the craft of IT.
  
  And the particular fact that the do not even recognize the architected cathedral, is a particular expensive bit of amnesia.

• Justin Alan Ryan | Mon, 20 Aug 2012 19:10:58 UTC

  Cathedrals are built by some of the greediest, yet least financially stable companies in the world.
  
  I also find it really interesting that the author insists on using FreeBSD on his laptop, but clearly knows less about free software and dependencies than I know about my Ubuntu hosts.
  
  There are some sticky points when it comes to dependencies.  A cathedral would provide us with less choices.  Why does glib depend on both Perl and Python? Because glib is a library that many people use from Perl and Python, and it's set to build those bindings.
  
  Buy a fucking mac, install XCode, and tell me about the beauty of the cathedral.

• Poul-Henning Kamp | Mon, 20 Aug 2012 19:21:21 UTC

  Dear Justin,
  
  Thanks for illustrating my point about the ignorance of the lost generation so forcefully.
  
  You clearly have no idea about what a cathedral is in this context, you clearly did not bother to find out and you spew random insults at the author, without even bothering to check if you were about to make a total ass of youself thereby.
  
  The reason I run FreeBSD is that I wrote a very significant part of it and therefore I probably know more about dependencies than you ever will.  Building the perl/python bindings as part of glibc is ass backwards in my book.  As for your "fucking mac", last I checked, my name was 35 times in the kernel of the open source version ("Darwin") of the OSX kernel.

• Peter Lindhard | Mon, 20 Aug 2012 19:43:38 UTC

  Perhaps it could be a general point in OSS projects, that contributors are reminded/persuaded to reread their parts at a later date. I honestly found it disturbing, to read what seems like disappointment that newcomers don't/can't do 20 years of code-clean up.
  
  Author seems blind to new challenges/interests, that new developers face/embrace. To be very honest antisocial ranting apparent in several publications by this author, makes for yet another terrible read. 
  

• mcur | Mon, 20 Aug 2012 19:50:47 UTC

  Mark Williams wasn't a paint company.
  
  http://en.wikipedia.org/wiki/Mark_Williams_Company

• Sven Türpe | Mon, 20 Aug 2012 19:54:50 UTC

  You are free to go to a restaurant rather than to the bazaar. You seem, however, to insist on preparing your meal yourself. Yes, this involves a variety of imperfect tools, and it makes a mess that you'll have to clean up befor you're done. But such was your choice, you wanted to prepare a meal and not go to a restaurant where the chef would have done it for you. In the restaurant, you install Firefox by dragging an orange icon over to the Applications folder, that's it. Somebody else takes care of the 122 ingredients for you.

• Rick Russell | Mon, 20 Aug 2012 20:26:14 UTC

  Mr. Kamp, your credentials are without fault, and I certainly am not qualified to question any of the specifics you discuss. 
  
  But despite all these perfectly legitimate observations, a good Linux distribution like Ubuntu is not only easy to install, but it is easy to maintain. The issues of package dependency, whatever they may be, are effectively hidden behind a rock-solid package management system that seems to have very few failure modes, at least to this 80s UNIX user and occasional programmer.
  
  I remember installing software on AIX, SunOS, Solaris and IRIX systems back in the late 80s & early 90s, before the Linux revolution and I would say that, subjectively, it was much more difficult to get working software back then. I don't just mean open source, although open source was part of it. Perhaps it was because the dependency mapping was not as complete -- you'd install a piece of commercial software on your Solaris environment and find out that one part of the software required a version of Motif you didn't have, and then you find out that version of Motif isn't compatible with the X terminals your labs use, and then... ? 
  
  There may be errors in the dependency tree for some applications in FreeBSD, but at least the dependency tree *exists* in some form. If individual packages are claiming unnecessary dependencies, then maybe the people who write that software need to clean up their act. In the bazaar, you have to tolerate such sloppiness. In the cathedral, that software might not be available at all because it never sees the light of day. Neither solution is optimal, but the bazaar seems preferable, to me, since at least I have a shot at getting it to work.

• Elizabeth Marston | Mon, 20 Aug 2012 20:30:09 UTC

  Great article. It's a shame how much ignorant ageism is in the comments  --- winning an argument on reason rather than emotion is, ironically, an application of rationality at which engineers are often poor. 
  
  What cathedrals have going for them is that they are beautiful. What bazaars have going for them is life. 
  
  FOSS didn't win the mindshare wars through propaganda and hazy hippie ideas about sharing; it won by being better; by building more slowly; by being inclusive. And inclusivity always means mess, because not everyone has the same intentions or tastes. 
  
  And how is UNIX dying? Please support this claim. From where I sit, it looks like it's winning by default. Code I write for Linux tends to resist bitrot. Unlike, say, the code I've written for Windows-land, where MS effectively says "and now for something completely different" every four years (R.I.P., C#) and thereby generates enormous waste. 
  
  It's that careful, incremental, inclusivity-loving bazaar mentality that allows us to bring the full force of our UNIX and C heritage to bear on any problem, and is the essence of code reuse. 
  
  The results speak for themselves. My apartment has a single, old, moldering Windows box and a slew of iOS, OSX, Android and Linux devices. That UNIX:Windows ratio has been constant for about ten years. What's changed is that my retired parents' place is now approaching that ratio as well. 
  
  Put more clearly: software that has no unnecessary parts is software that can never be put to any purpose except those foreseen (and permitted) by the designer. It is as fragile and as beautiful as a snowflake, and tends to last about as long before someone has a better, entirely incompatible idea. It's what killed Esperanto, and it's what will kill Windows (and many other commercial projects.)
  
  Top-down design works great for as long as everyone shares (or at least respects) the intentions of the designer. But that necessarily means quashing other voices. Perhaps some of those voices aren't as clear, or those speaking have impoverished mental models of how the OS functions; but others will simply be different opinions about an underdetermined solution space. 
  
  

• Elizabeth Marston | Mon, 20 Aug 2012 20:51:55 UTC

  (Quick clarification on my last post: -- the phrase 'what will kill Windows' isn't clear in context, as Windows has many unnecessary parts :3 What will kill Windows is the _anxiety_ that its developers have about the unnecessary parts, and their repeated attempts to deprecate whatever isn't cool this week.)
  

• Hector Correa | Mon, 20 Aug 2012 20:53:47 UTC

  I find it interesting that even systems owned by a single company (like MS Windows) are also messy. 
  
  Microsoft has done a great job of simplifying the installation of Windows (particularly compared to what you describe for your system!) but as David F. Skoll mentioned in a previous comment the API is equally messy if you look behind the scenes. Heck, there were even some well publicized stories about some of the hacks buried in the Windows source code to account for wacky scenarios (e.g. http://www.kuro5hin.org/story/2004/2/15/71552/7795)
  
  Even more interesting, is that none of these systems (Windows or Unix) are typically maintained by "casual software developers" that just took on development overnight during the dot-com era. 
  
  I think the mess inside these systems is just the nature of large systems, maintained by a large number of people over a long time period.
  

• Mike Smith | Mon, 20 Aug 2012 21:16:28 UTC

  Poul,
  
  Thanks for putting this together so cogently.  Your rants are always thought-provoking and a good read.
  
  Your point about design had me wondering; why is "good design" currently trendy when it comes to UX, but uncool when it comes to software (and many other engineering disciplines).  Is it really just a socialisation problem?
  
   = Mike
  

• Poul-Henning Kamp | Mon, 20 Aug 2012 21:19:50 UTC

  Isn't that simply because "civilians" see UX, but not source code, and there are more people with a sense of style, æsthetics and architecture amongst the "civilians" ?

• Elizabeth Marston | Mon, 20 Aug 2012 21:43:42 UTC

  Poul: With a background that straddles both UX and low-level dev, I've been wondering this myself. At first I thought it was an aesthetic thing. But now I think that it's just straight-up conflict of interest, like we're used to seeing elsewhere in the public sphere. Developers can't advocate for users because they're _developers_ -- their best interests are in long-term architectural stability and elegance, not in ease-of-use or rapid production. Developers are intrinsically lazy, and do not wish to make work for themselves; and in every other context, this is a good thing. Do not expect a developer to agitate passionately against the best interests of developers, as a UX specialist must often do. 
  
  

• Jonathan Abbey | Mon, 20 Aug 2012 21:45:25 UTC

  I hope you will follow this column with one that more fully illustrates the nature of these cathedrals, ideally with examples for us to enjoy.

• David F. Skoll | Mon, 20 Aug 2012 22:01:29 UTC

  Oh, some more random thoughts... I've been programming professionally for about 18 years and have participated in my share of both open-source and closed-source projects.
  
  In my experience, no matter how careful you are and how good your intentions are, almost all large software projects tend to have a few ugly dark nuggets at their core.  Perhaps it was done that way to work around limitations of the development environment, or perhaps the original design was found lacking and it was easier to patch things up than do a major redesign.  But we're only human: We can't predict everything and sometimes a workaround (aka a hack) is more efficient and more productive than a redesign.  Sometimes the perfect really is the enemy of the good.
  
  Again writing from personal experience, I'm much more likely to try to exorcise the ugly dark nuggets from an open-source project than from a closed-source one.  In the open-source project, my peers can read my code and I might be subject to their derision.  In a closed-source project, everything can be swept nicely under the rug.
  
  (I realize that "open source" does not mean "bazaar", but "closed source" usually does mean "cathedral.")

• Kartik Agaram | Mon, 20 Aug 2012 22:09:57 UTC

  @Mike Smith, @phk: Reading your conversation it occurred to me that for all the benefits of commandlines that we geeks like to rant about, a GUI has one key benefit. Anybody who looks at it can instantly spot ugliness and cruft to some extent. By contrast, a commandline-oriented stack is a lot less discoverable, and encourages the formation of various 'commons' like dotfiles, extra commandline flags, open file handles. Since nobody is monitoring these areas for cruft, we developers dump indiscriminately on them.
  
  This point is independent of how savvy the reader is, I think. My visual cortex instantly shows me what's crufty about a UI. Dotfiles on the other hand I have to deliberately go dig around in to criticize.

• Anton Striklov | Mon, 20 Aug 2012 22:11:16 UTC

  As much as I like FreeBSD one cannot equate the success / failure of the whole bazaar ecosystem to it. FreeBSD is awesome in some aspects and terrible at others, and while many projects lag behind it, plenty far surpass it. It's package management system for example if far inferior to the one present in Gentoo. But why stick with Unix OS's for analysis ? The sheer volume of non-OS related successful projects developed under the bazaar model is staggering !

• ford p | Mon, 20 Aug 2012 23:54:18 UTC

  This article is horseshit.  Total flamebait and founded on the obviously false assumption that you ever need to compile a whole distro from source.  Was this article written in the 1980s and you just got around to UUCPing it off from paper tape?

• Bernd Jendrissek | Tue, 21 Aug 2012 02:54:11 UTC

  Much of your rant seems predicated on the badness of the autotools. While some of your criticisms are certainly valid (even though I might not necessarily agree what the response should be), you seem to be burning the sources of inelegant software in effigy, by attacking the autotools. The creators of the autotools are not the ones responsible for the problems which the autotools were designed to address!
  
  Secondly, some of your criticisms (here and in your autocrap rant) against the autotools are again misplaced: you seem to want to hold the autotools' creators responsible for the misuses to which others put them, sometimes due to innocent ignorance, other times due to an arrogant sense of superiority (poor translation: I want to convey the german word "besserwisserisch").
  
  As others have pointed out, I'm not surprised either that there are features in a large dependency graph that go into a black hole, like the TIFF support you mention. My curiosity is somewhat piqued to know how this irony resolves. And specifically, is this a run-time dependency or a build-time dependency?
  
  But here's another irony: you don't recognize the deliberate architecture to which the autotools were built. I wasn't around at the time to know if they were originally built cathedral-style (their development now seems very bazaar-like), and you might not appreciate the architectural style used, but boy, you'd better believe it, much of your interaction with them was *designed*. In my experience most of the "friction" in interacting with the autotools is due to the very sort of meta-ignorance you write of: not even knowing what the tools' raison d'être are, or "disagreeing" with them (as if that is possible). I like to say: Those who do not understand the autotools are doomed to reinvent them, poorly. Please understand that I mean a philosophical understanding; I'm sure you know very well what AC_CHECK_LIBS does, for example. And yes, I think I can agree with your criticism of keeping around 20-years-superseded feature checks.
  
  P.S. I know I'm conflating cathedral-as-designed-artifact with cathedral-as-home-of-a-priesthood.

• mike mccafferty | Tue, 21 Aug 2012 03:19:19 UTC

  The dilution of a profession's core skills due to population explosion strikes me as an interesting twist on Naomi Klein's "Shock Doctrine."  Dotcom-era opportunities create a virtual Gold Rush, people with wildly varying degrees of of technical skill rush into a space, and the IT population growth rate is such that experienced hands simply cannot transmit best practices fast enough to keep pace with the population increase.  Soon the notion of "state of the art" has changed, because a large percentage of the population doesn't know where to look, and doesn't know who to ask for pointers.  It's the  "telephone" game, played out on the clock and at a velocity that prevents error correction.
  
  Thanks for recommending out the book.  I know that I will benefit from the lessons contained within.

• Terry Lambert | Tue, 21 Aug 2012 03:51:25 UTC

  When Sun switched over to SVR4, it became impossible to to get BattleMap to run on any decent hardware.  Now it's sold by McCabe software as McCabe IQ.  It's down from $50,000 a seat to $45,000 for three floating licenses.
  
  I think one of the issues with the Bazaar model is that younger programmers, particularly the ones Poul-Henning identified as coming out of the .bomb era as being hired out of their second year in CS coursework to be cubicle warmers at startups which went nowhere, wouldn't recognize a decent CASE tool if it bit them on the ass.
  
  Complexity is manageable through abstraction, and through good tools, and anyone who doesn't understand that is doomed to recreate their own version of the autotools mess in their own projects.  Eventually I will probably run into them in McDonalds while they are asking me "Do you want fries with that".
  
  Also, I take umbrage at Phillip Andrew's "break everything into tiny programs" model of development; an inability to manage complexity is no reason to drink the Daniel J. Bernstein Kool-Aid that has djbdns _still_ unable to do zone transfers, and synchronizing primary and secondary DNS servers "is an exercise left for the student".  The attack surface on 50 small programs is way larger than the attack surface on one large program.  It's why we still build monolithic kernels.
  
  Poul-Henning and I haven't always seen eye to eye on everything, but his post here is spot on.  Good on you, PHK.

• Max Kanat-Alexander | Tue, 21 Aug 2012 05:33:19 UTC

  First, let me say that this is an excellent article. For many years, I was the Chief Architect of the Bugzilla Project and heavily involved in the open source community as a whole, and I can assure all readers here that there will never be a model of software development in which developer responsibility can be neglected. In every model that will ever exist, developer understanding and responsibility will be the key component to quality. There are methodologies that make quality hard and methodologies that make quality easy, but there are no models that create quality code without guidance from some unified viewpoint and individual responsibility for software design down to the lowest levels of development.
  
  The bazaar has all the problems that anarchy has--if a few good people step up to accomplish and lead things, then they go well. If nobody steps up to be responsible for quality, then even though things get built, they can't be nice things.
  
  The cathedral has all the problems of a dictatorship. If the dictator is wise and benevolent, then the project prospers and is beautiful. If the dictator is ignorant or possessed of motives beyond the good of the project, then the project can fail and be disastrous.
  
  Our modern economy throws in one additional crowbar to the cathedral as well--that most people making centrally-controlled projects are creating them under the banner of organizations that do not want external contributions and which do not which to collaborate with others. The cathedral has been creating just as much duplication of effort in the past many decades as has the bazaar.
  
  And finally, an even more salient point to touch on is that perhaps "the bazaar" is itself a myth. The best open-source projects are, in reality, cathedrals in which anybody is allowed to build pieces, but only after they have passed the approval of the central control. We could certainly image a "bazaar" to exist--some set of projects that have somebody birthed themselves out of anarchy. But there's no situation in which that's true. Software is conceived, created, and guided by people, and when those people have great understanding and all take strong responsibility, excellent work is done. When they neglect their code or assume that "somebody else will take care of it," then ugly disorder ensues.
  
  -Max

• Justen | Tue, 21 Aug 2012 05:45:25 UTC

  Sounds like your problem is a crap package management system more than anything. I can't believe you have to actually manually research and add dependencies. What decade are you living in?
  
  I'd also like to note that development methodology has little do to with the problem of quality. The internet is swimming with horrific windows shareware that is not only *not* package managed, and full of bugs and compatibility issues even though they depend on an essentially monolithic platform and set of libraries, but still inexplicably expects us to pay for "full" features. One of the greatest joys I discovered in moving to Linux was all the free, relatively bug-free, fully functional basic utility software of the kind that every computer needs to be useful - all in a central packaging system that not only installs software in one click (along with all its dependencies) but keeps track of when it needs to be updated too! And don't even get me started on Mac, a platform so terrified of free (as in beer or as in freedom) and open source that I was just asked to pay $8 for a script that finds and consolidates duplicate music files in iTunes. Are you *serious*? How is that not a built-in feature of every media library manager in existence, let alone one of the keystones of the Apple experience.
  
  Nah man, I think you're looking at this through too narrow a lens. And maybe you need to get off FreeBSD, the red headed stepchild of of ghetto *nixes.

• Justen | Tue, 21 Aug 2012 06:43:22 UTC

  After reading the further discussion in the comments, I have a greater understanding of what you're actually trying to complain about here. But you've turned what ought to be a simple lamentation about cruft into a jermiad. There are reasons why this cruft sticks around: because somewhere out there, someone wants to install a bit of *nix software on a truly ancient machine where it will actually matter; because a few thousand clocks is infinitesimal in cost to any given user compared to the cost to a developer to carefully excise it; because the vast majority of users don't know it's there and wouldn't even know why it's bad if they did; because most of us use binary package systems, only supplementing with compiled-from-source binaries when they are too obscure to merit inclusion in the repositories. 
  
  The *nix ecosystem is incredibly massive. If it's true that you can only get a cathedral from continuity of executive, and that a single virtuoso, or rarely a dynamic duo, must manage an entire project with an iron fist in order to achieve cathedraldom; then where is this godlike ego you propose to manage all of *nix? Or shall we commit the even bigger sin of tossing it all out and starting over so that, this time, we can do it "right"? Even when standards are proposed, or when they emerge from common practices, there will always be the outliers (*coughbsdcough*) who insist on doing it differently and they'll still want to run common *nix software. Inevitably some monster like autoconf and friends will arrive on the scene to facilitate that.
  
  Now as to M4, I have no apologies for that. Ugh. I thought Perl was opaque until the first time I tried to configure sendmail.

• Pete | Tue, 21 Aug 2012 07:10:25 UTC

  There are some good points here, but there's a couple glaring problems.
  
  The first half of the article laments the mass of "college dropouts" who entered the technology field around the turn of the millennium.  The second half of the article laments the mess of legacy technical decisions that we are stuck with.  But for the most part, the former did not cause the latter!
  
  For example, you spend a while talking about the disaster that is autoconf.  (I think this is one of those tar-babies that most people don't bother talking about any more, because everybody knows how bad it is.)  But it was about the furthest thing from a dot-com-era college project: it predates that party by about a decade.  It was created to solve a real problem (hand-editing configure scripts), back in the day.
  
  This leads straight into the second problem.  It's perhaps not fair to complain about autoconf itself, since it solved a real problem back in the early 1990's, but it is fair to complain about modern projects like Firefox continuing to adopt it, rather than being more portable from the start.  But Firefox, for the most part, *works*.  It exists.  It delivers what people want.  Imagine a web browser built with more solid software engineering principles.  Does it give users anything they don't have today?  (Firefox is pretty stable, and it does everything I think I need.)  Does it make life better for its developers?  (Apparently there are developers willing to work on Firefox as it is today for free, but I don't see anyone doing the same for a serious alternative.)  What tangible benefit would it actually have, except to make some of us feel better about it?  We have working, free, open-source web browsers, which are faster and more capable than ever.  I wish they had a better architecture, but it's hard to complain too much when things are so much better than 10 years ago.
  
  Finally, you also dish out against Unix and autoconf quite a bit, but I'm not sure if this is because other operating systems are any better, or because it's simply the most public one, so we see all the problems.
  

• holger | Tue, 21 Aug 2012 10:22:05 UTC

  The true cause indeed seems to be the explosion of IT "professionals" we saw in the years leading up to and during the dot com boom. Things just had to get done (and quickly) because otherwise someone else could have been first to market. Creating an incredible demand for IT "somethings" and attracting lots of people with little or no formal education in the field. But the problem is not only lack of education: today the core problem applies to cathedral and bazaar software in exactly the same way. Because what coincided with the dot com boom (of not being the cause of it) was that hardware advanced faster than ever before in those years. Meaning you could pretty much get away with less then optimal code (and coding practices) most of the time. 
  
  Almost nobody cares about efficient design (much less code efficiency) anymore, it's all about quick design and development efficiency. Once something (mostly) works, it's good enough. Needless to say, it usually isn't,
  but that isn't easily noticable on the developers machine. But deploy that code to a hundred thousand servers or a couple million users, and the overall balance shifts - inefficient code translates to wasted computing resources and wasted energy on a very large scale. And to make matters worse, object oriented languages, while being in invaluable design tool to the programmer who is experienced with them, make it unbelievably easy to write horribly inefficient and bloated code without even noticing if you lack said experience. (Not to mention to include more dependencies than you can count if code reuse means "someone else probably already wrote something similar, so i'll just google" to you).
  
  Someone actually suggested using the libgmp bindings for bit manipulations in some language that apparently didn't readily offer them in a comment on thedailywtf.com because he googled and found out that it had something similar (for bignums if I remember correctly, but hey, you can always convert and back, right?). That was one of my largest wtfs i ever saw there.
  
  And even the better programmers out there often end up adding layers over layers of abstraction to make an overly complex problem somewhat tractable without ever questioning the underlying design once there is one that seems to work "good enough". In a bazaar style project, there's at least the off chance someone is going to revisit things. In a cathedral style project, design decisions tend to be set in stone once the first release is out "that's just the way things are done around here".

• Alkonaut | Tue, 21 Aug 2012 10:41:13 UTC

  Don't confuse the "bazaar" with Linux. Linux is a perfect example of a cathedral. Complete with its high priest. The linux *distros* (e.g. GNU/Linux) shows the real "chaos" of the bazaar. 

• foljs | Tue, 21 Aug 2012 10:42:18 UTC

  @phd_student_doom
  """Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the 'chaos' is what makes programming fun."""
  
  Well, speaking as someone of your age, sure, it will be a very "fun" IT landscape indeed when ignorants like you take over (even more so considering this BS non-response was written by a "phd student").
  
  This "chaos" is what makes programming fun?
  
  Spoken like someone who doesn't understand accidental and essential complexity at all, and cherishes trivial bullshit tinkering over actually making something, much less making something beautiful. 
  
  

• Glenn French II | Tue, 21 Aug 2012 12:32:15 UTC

  I figured that I would chime in once I read all of the other unprepared and ill-advised comments.
  
  On another note, thanks for all of the contributions.

• David MacKenzie | Tue, 21 Aug 2012 13:34:45 UTC

  As the creator of Autoconf, I'd like to apologize to everyone for using m4. I chose it to minimize dependencies, as it was on every Unix-like system already. Many of us were on 28kbps dialup and people would complain about the time needed to download a big package like Perl. Then I hit so many Unix m4 limitations and bugs that I ended up requiring the GNU version anyway. I probably should have started over at that point with another approach. It's true that I did most of that work before I'd earned my CS degree, but no one more experienced deigned to tackle the practical problems I needed to solve, so I had to. (Larry Wall had half-tackled them.)
  
  When I did the initial design of Automake a few years later, Perl had become widespread enough that I could get away with specifying it as the implementation language. (I wasn't involved with libtool.)
  
  The core GNU project (not to be confused with the ports or other packages like X11) does have a quality assurance dictator of sorts in Richard Stallman and the GNU Coding Standards. A lot of the work I and other maintainers did on GNU was bringing other peoples' code up to those quality standards, so that our versions of POSIX commands and libraries had fewer arbitrary limitations and buggy corner cases than the Unix versions that originated in Bell Labs. Our source code was also supposed to be readable and well documented. We had the advantage of another decade or two in processing power and RAM at our disposal compared to the original Unix developers, who by then were working on Plan 9 which was elegant but never really became available to use until it was itself a dead project.
  
  When I started Autoconf, there were no full Unix-like operating systems with all the source code available to everyone, so we had to bootstrap ours on the variety of existing systems and accomodate their variations.
  
  It doesn't solve any practical problems to say that applications like Firefox should be written in a more portable way so there isn't a need for something like Autoconf. Unless you want to reduce the whole global software industry to a single company or government regulation, communist-style, you can't control what other people are going to do. And so you need to work around the diversity.
  
  The dependencies-of-dependencies tree can indeed get large when you're leveraging general-purpose tools. And keep in mind that although you need to build some seemingly unrelated things in order to end up with Firefox, those things might end up getting used for something else you're also building. Most people just download binary distributions, anyway.
  
  What I haven't seen so far in this discussion is a workable real-world proposal for what to do differently other than a few incidental details about particular software. The complaint seems to be about the big-picture approach, but I suspect Eric Raymond is right that no cathedral can be large enough to encompass a whole industry. And that's effectively what the FreeBSD ports system contains.
  

• Frank Greco | Tue, 21 Aug 2012 14:48:55 UTC

  Brilliant article.  I've been thinking of the same stuff since the Cathedral book.  It seems in other endevours there exists quick spike, multi-creator mediocrities and high-priest design masterpieces.   From building architecture to music to paintings to TV shows to finance, et al.   Apparently software is no different.

• Poul-Henning Kamp | Tue, 21 Aug 2012 17:58:42 UTC

  @David MacKenzie:
  
  I don't dispute that autoconf was a pragmatic solution to the madness of UNIX vendors back in the 1980'ies and 1990'ies.  It certainly saved time for me back then.
  
  But today it no longer serves a useful purpose, other than add incomprehensible onamentation and woodoo incantations to open source software, for no credible gain in overall quality or portability.
  
  I am certainly not looking for "The One Unix To Rule Them All", but I would really love to see some intelligent standardization fill the worst pot-holes, so we don't have to waste so much time while autoconf checks for  and libtool looks for 26 fortran compilers, twice.
  
  The one thing cathedrals is better at than bazaars, is throwing junk overboard, that's probably the main reason why I wish we had more cathedral and less bazaar.
  
  As for M4: Apology accepted, but can we please loose it now ?

• Marty Fouts | Tue, 21 Aug 2012 19:09:26 UTC

  Mr Kamp is right, but it is clear from the responses that he is not well understood.
  
  I have programmed professionally since 1972.   At that time the computer industry was already twenty years old, but those of us who entered it studied its history and learned from it.  Sometime between 1980 and 1990 that willingness to learn lessons from past projects disappeared, while a sense of hubris entered the industry, epitomized by the conversion of our job titles from "programmer" to "engineer" that took root in the 1980s.
  
  Now, the computer industry is always twenty years old, and those who come to the profession no longer learn the lessons of the past.  Recently, an algorithms "expert" I interviewed, admitted to having no idea who Don Knuth was, or why they should be aware of his work.  Such interviews are becoming more, not less, common.  I can point at hundreds of such examples of dumbing-down in the industry: OS kernel developers who can not describe the difference between a write-through and a write-back cache; people claiming to be C++ experts who can not enumerate the member function in the class definition 'class f {};' C# programmers who can associate the concepts involved in boxing with the underlying mechanisms of stacks and heaps, nor describe the machine level implementation of either. The list is endless.
  
  Nor is it limited to the people who entered the industry in the early 2000s and remain.  It is a trend that continues to this day.  University computer science programs are laughable, and the produce both the above, and at the other extreme, the sort of highly intelligent but extremely inexperienced people that companies like Google hire and that go on to produce abominations like the build system for ChromiumOS.  Or the people at Microsoft, who because of the nature of its career path are always moving to new projects for which they have no experience.  Microsoft's software looks like it was written by people with three years of experience and no mentoring because it was.  Even programmers who have been there for two decades have rarely worked in one area for more than a couple of years before moving on to something else.
  
  We are sinking in a morass of mediocrity.  In a world where every programmer, regardless their background, experience, or training, is a 'senior software engineer', we should expect no less.

• Roboknight | Tue, 21 Aug 2012 20:02:19 UTC

  I think most of the commenters here pretty much missed the point.
  1) Yes, this is a rant.
  2) The point of the article isn't the fact that it requires 122 packages to build Firefox, or even the additional bloat that goes with it.  Poul is mainly concerned with how we got here.  It does wander around the mark a bit, but generally, I'd have to agree with the argument.
  
  I myself have built MANY projects and typically don't lament the fact that it takes 122 packages to build something (although I have been there too), but you can see the inexperience in many of the projects, especially the ones that have a claim along the lines:  "This project was great, but was poorly designed, so we are going to rewrite it."  This doesn't just lead to something requiring 122 packages to build, but to having things like Gnome and KDE installed on the same system because some package you wanted to use wasn't built with the libraries you chose to install on your system.  This especially occurs when someone has written a nice tool that you would like to use, but uses a different set of libraries than you do.  Compviz, while a really cool effort, suffered from this as well as other issues.
  
  I also agree with Poul that while I'm excited that open software has taken off, I'm saddened to see that the Cathedral model is more difficult to find and has pretty much been overrun by the Bazaar model.  I had hoped that the two models of software development would feed off of one another and the deficiencies in both models would be overcome and platforms like Linux/BSD would have been able to avoid "development bloat" and been a little more cross-compatible than they are.  Maybe avoiding clashes like Gnome v KDE or having several thousand copies of encryption code.
  
  Finally, to those that completely disagree with Poul, look at the opensource landscape.  Take a look at how many repeat projects, dead ends, and restarts there are out there.  In this kind of environment, I'd agree that what is out there is certainly an "evolution", but even evolution has a plan that does typically strip away inefficiency.  I can't say that truly occurs with opensource.  The projects that seem to succeed beyond others are the ones that have a single driving force moving them toward a common end goal.  Apple has picked up clang, for example, and that seems to keep moving forward.  Google has picked up a few projects.  The ones that have not been "picked up" and are successful, still have dedicated developers working everyday to make them better.  I'd liken those developers less to a cathedral and more to a glass tower, where everyone waits and watches for the next development.  Every once in a while, something comes in and out of the tower (i.e. outside patches and so forth), but in general, things are added and integrated on the schedule laid out by those that are dedicated.  Typically, those dedicated individuals have put in the time and effort it takes to become the "good carpenter" that Poul talked about.  But there are many more "wood pushers", to use an old Chessmaster term, out there.  And their influence really turns the Bazaar into a difficult place to develop, or even build.  Yes, you can be successful, but it seems that it was once easier to do so.

• Alexander Vlasov | Tue, 21 Aug 2012 22:30:41 UTC

  I, for one, can't believe that a person with PHK's skills and experience really doesn't know why compiling glib needs both perl & python or why libtiff may be required to build firefox & its dependencies from scratch. 
  So this is merely a provocation. Nice try.

• sl | Wed, 22 Aug 2012 01:06:33 UTC

  @Poul-Henning:
  Come to Plan 9. We don't have autoconf at all.

• Ivan | Wed, 22 Aug 2012 02:34:30 UTC

  Seems silly to rant about code reuse but then suggest rewriting autotools when sane build systems like cmake exist.

• Klapauzius | Wed, 22 Aug 2012 09:47:13 UTC

  I think the poster bases his argument - or rant, if you will - on an intellectually questionable trick: Yes, there are the dotcom-JavaScript-PhP-Wunderkinds (what Jamie zawinsky describes as "Attention Deficit Teenagers"), but what they do is build silly, bloated Webpages. They don't really program on an application or even system level because they can't. If they try, they fail spectactularly, case in point being the disaster that is GNOME 3 (which in the end shows that the OSS model works very well in axing failed projects, much better than in the corporate world). But the author makes the leap from these Web-spamming "New media" whizkids to system programming which is a totally different animal.
  
  I also believe that the author is somewhat behind the curve as far as the consolidation efforts in the UNIX world are concerned. Whereas there used to be dozens of more or less incompatible UNIX'en out there (Tru64 anyone ?), today there are basically Linux and BSD. And yes, there are the remnants of having to support all these platforms (most of which have joined the choir invisible), but so what. If there's a line of shell code in configure that never gets executed any more, i can imagine worse problems. And i *LOVE* the fact that i'm currently running more or less identical Linux installs more or less hassle-free on Hardware as different as a PowerPC box, an x86 PC and an ARM-powered Netbook. I could surely run it just as hassle-free on an IBM mainframe, but i haven't stacked one in my cellar just yet :-)
  
  If that isn't progress, i don't know what is.

• Poul-Henning Kamp | Wed, 22 Aug 2012 11:05:51 UTC

  @sl: If I had been able to get hold of p9 when I asked for it back around 1990, I would not have spent two decades on FreeBSD.
  
  @Klapauzius:  I think there is plenty of time before progress goes too far :-)

• Christophe de Dinechin | Wed, 22 Aug 2012 14:31:41 UTC

  > One of Brooks's many excellent points is that quality happens only
  > if somebody has the responsibility for it, and that "somebody" can
  > be no more than one single personwith an exception for a dynamic duo.
  
  I think that this argument is bogus. Quality is defined by usage, not by manufacturing. Today's cars don't rust anymore, and it's not because a single person took charge, it's on the contrary because solid processes were put into place. The same holds true for programming.
  
  In the late 1980s, I wrote a 3D game that entered the Guinness Book of Records. Back then, me, myself and I, as a single person, could deliver a world-class video game on my spare time as a student. It was considered "high quality" back then. And it would seem to support the point that a single person can deliver quality.
  
  Yet that game wouldn't pass any standard today. It was all of 17000 lines of code (puny by today's standard), and all in assembly, very ugly by today's standards. It didn't use any well-defined 3D API, since there weren't any. Instead, it blasted pixels on a fixed screen geometry using self-modifying code. Nobody would do that today. It computed approximations of 3D transforms that used mostly additions because multiplications were expensive. Who would avoid multiplications today when doing 3D transforms? The name of "quality" has changed.
  
  There's no way one of my kids, who are brighter than I am, can deliver a world class game all by themselves today. A single texture in a modern video game contains more data than my entire game. More importantly, that humble texture refers to more accumulated knowledge about color spaces, compression, lighting and dozens of other complex mathematical fields than was necessary back in 1990. Who is responsible for the quality of the texture? The single guy who, all by himself, invented the JPEG, TIFF and PNG standards? Or the dedicated artist who tries to implement in his texture the vision shared with an entire team about the game?
  
  We always need to anthropomorphize, to put a single face behind every great achievement. For many people, Linux is the brainchild of Linus Torvalds, the iPhone is Steve Job's vision. That is just not true. Cathedrals were never built by a single person, never dreamt of by a single person.
  
  I'd actually go further. Quality needs at least two people: someone to build, and someone to use. And quality emerges when the user is happy with what was built. Currently, more and more, cathedrals emerge only from the bazaar. The crafting process is messy, but who cares. The result is solid, beautiful, interesting, usable, whatever.

• Dan Cross | Wed, 22 Aug 2012 14:38:18 UTC

  I think this article is decent, but the thesis is insufficiently stated and supported, and many readers are missing the point.
  
  The point, as I understand it, is that programmers work in environments that lack good example s of design.  This is secondary to the development model used (beautiful things evolve in bazaars just as they do in cathedrals, despite Brooks's apparent suggestion that an individual who mandates quality is essential at some level), and more indicative of a lack of relevant experience, aesthetics, and a rejection of the responsibility to learn these things because it is not "the way it's always been done."  Except what we do now is not the way it's always been done, and too many people don't know enough about the history of the profession to realize that.
  
  The autotools vignette is an aside, but is drawing most of the attention in the response.  Perhaps Kamp's point is that autotools solved the wrong problem; treating a symptom instead of the underlying issue, like a bandaid that became a permanent fixture.  How could it have been done better?  The answer to that specific question doesn't matter.
  
  The real problem is that no one now is asking the question, "how can we do better?" (present tense as opposed to past).  And no one is asking that because they don't even realize that they should.  The intellectual curiosity to ask isn't there.  That's the real problem, not whether libtool searches for FORTRAN compilers from systems that no one has used in 15 years, or whether some command has options that haven't mattered in 20.  If someone bothered to ask why and decided to clean it up, it would get fixed.  E.g., if someone decided to ask why invocations of 'ld' are so different across systems and get them cleaned up, it might happen (though that's a much larger and harder problem).
  
  This is the inevitable result when people stop questioning the basics and start taking them for granted.  Other disciplines stress proficiency in the fundamentals: professional basketball players practice basic shooting and dribbling.  The computer industry does not, in favor of 'graduating' to flashy whizz-bang fluff.  The result is that people do not stop to say, "hey, maybe there's a better way to do this?" and cruft accumulates for decades: why did it take nearly 25 years after the Morris Internet worm to remove gets() from the C standard library?
  
  On the other hand, there are many who have (and would continue) to argue that Unix is not the model of good design and elegance that Kamp holds to be.  For example, why does the command line interface on my Macintosh laptop (which has a high-resolution hardware accelerated bitmapped display, multicore processor with a >2GHz clock, 4 gigabytes of RAM and integrated keyboard and mouse) emulate a TTY communicating at 9600 BAUD?  Why the applications continue to use a cursor addressing command set from the 1970s?  Plan 9 did better, and even though it's been generally available as an example for most of the past decade, the Unix/Linux folks don't want to evolve.  Perhaps Kamp couldn't get a license in 1990; that doesn't excuse him for not learning and applying its lessons and from 2003 on.
  
  So that's the real problem: people are unwilling to question the most basic premises on which they build systems.  It has little to do with cathedrals or bazaars or myriad PHP-jockeys who can't tell the difference between a regular expression and an arithmetic expression, it's about the unwillingness to question entrenched assumptions, and cut through the complexity to get to the underlying simple, elegant solution at the most basic levels.  Or maybe people do understand these things and don't care because it's not worth the effort along dimensions they care about (but Kamp does not).
  
  This article was a rant against autotools so much as a plea to start questioning whether things are as good as they can be.  Unfortunately, the short answer to that is almost always going to be, "No."

• Irene Knapp | Wed, 22 Aug 2012 15:07:29 UTC

  I agree with everything you say, and based on the number of negative comments I thought I should add my vote for cathedrals.  But there's no point in saying it; the bazaar-dwellers are too convinced they're right.

• denisroyle | Wed, 22 Aug 2012 16:55:01 UTC

  The computer paradigm affords the absolute precedence of the opcode over the operand. This leads to the unquestioned precedence of source over data of delivery mechanism over content of industry over user. The binary abstraction at its core is a profoundly irrelevant and divisive concept capable only of encouraging naval gazing and generating counter-productive noise. Stop IT.
  
  Different enough?

• Poul-Henning Kamp | Wed, 22 Aug 2012 18:15:03 UTC

  @Dan Cross:
  
  Nice rant :-)
  
  I disagree with you about the lack of curiosity:  There is plenty of curiosity, which is why the lost generation throws themselves at any fad that comes around, no matter how hare-brained it might be.
  
  Many of these fads are reinventions of stuff that was invented long ago, and explored and reported on, but nobody in the lost generation would ever dream about reading CS and IT research from before year 2000:  They cannot even imagine what they might learn from it.
  
  When I recommended my readers read The Mythical Man-Month, I subsequently received several emails from people saying "You said this was a good book, it's about *mainframes* for goodness sake, why should I read about mainframes ??!"
  
  That attitude, is also represented in may comments to the current column, and to me indicates that a large number of young-ish practitioners simply cannot fathom that there might be a higher level of abstraction at all.
  
  In other words:  They wouldn't recognize a cathedral if they saw one.
  
  Poul-Henning
  
  PS: With respect to your terminal window, that's an interesting case, and it's a very good question.  My guess is that command line interfaces simply have qualities, for instance economy of bandwidth, that allow their wrapping in antique simulations of printing terminals to survive.  I have yet to see a credible substitute or improvement, but I have seen several graphically designed programs getting much improved with the addition of a command line, even if just in the form of ALT-F, ALT-S menu shortcuts.
  
  PPS: As for what I have and have not done inspired by p9, that's a long story, told in details in FreeBSD's version control system.

• Dan Cross | Wed, 22 Aug 2012 20:25:35 UTC

  Poul-Henning:
  
  The intent with my intellectual curiosity remark had to do with questioning basic assumptions (e.g., "we use autoconf because we've always used autoconf").  This should not be taken to imply that people aren't curious about other things; my concern is that the curiosity about the basics is no longer there ("do we need to use autoconf?").
  
  Had I written your polemic, I would have avoided the cathedral metaphor: it's too overloaded, and I feel that your point has become obscured in the ensuing rhetoric.  If I understand you correctly, you mention cathedrals because, by and large, they are beautiful.  You are then lamenting the seeming inability of the 'lost generation' to recognize beauty in design and implementation of software, as evidenced by the bloated layers upon layers of cruft that have become the norm (autoconf is just a specific example).  I would have used another metaphor to more clearly highlight the argument; say, contrasting between the works of a dutch master or a child's finger painting.  Or maybe between a Jackson Pollock and a child's finger painting.  There are superficial similarities, but one has sublime aspects of beauty that the other does not, and these take a certain about of refinement to appreciate.  It's that refinement that's missing in too many cases. Many would argue that similar refinement has been missing from the Unix community almost since its inception: are they wrong?  To some extent these things ARE subjective, and calling one group a 'lost generation' without acknowledging that one may be lost oneself is a tad bit dishonest.  Like the Sphere from "flatland" that cannot fathom a 4th or higher dimension, some of the older generation struggle similarly; I know I sure do.
  
  Sure, things are reinvented and the reinventers aren't even aware of it because they haven't studied the literature well enough to become acquainted with the full body of work in the relevant field, but the same is true of experienced practitioners as well: e.g., your rediscovery of d-heaps.  The seemingly recent rediscovery of DSLs by the design-patterns set.  Lisp being rediscovered again and again and again.  Etc.
  
  PS: I have nothing against command line interfaces.  In fact, by and large I prefer them.  What I object to is the Unix-centric notion that command line interfaces must be tied to the implementation details of 1970s era hardware devices that have become mostly irrelevant (I realize people still occasionally use serial terminals, but as a primary environment that has become as rare as caring about the name of the FORTRAN-77 compiler on some obscure Unix variant).  As far as credible substitutes for the Unix pseudo-tty model: Plan 9 has already been mentioned and the command windows of rio and acme are perfectly usable (many would argue superior) and don't emulate VT100s or have any notion of a BAUD rate.

• Jussi Pakkanen | Wed, 22 Aug 2012 21:11:52 UTC

  Autotools truly is terrible. For those who want more gory details, here's a blog post I wrote some time ago on the subject:
  
  http://voices.canonical.com/jussi.pakkanen/2011/09/13/autotools/

• Poul-Henning Kamp | Wed, 22 Aug 2012 23:07:41 UTC

  @Dan Cross:
  
  The main reason I chose the Cathedral metaphor, as bad and overloaded as it is, was because I think ESR's ideological manifesto is a large part of the problem.  "Bazaar won, f**k cathedrals" as one commenter summed up his view to me.
  
  And I think "lost generation" is a both fair and apt description, because most of that generation have grown used to things "being shit all the way down" as my friend Arthur is fond of saying, and cannot even imagine that it can be any better.
  
  We are talking about people here who expect printer drivers to crash their kernel, who troubleshoots by pointing and clicking on any icon which seems remotely related and who thinks that virus scanners and firewalls are where security comes from.
  
  And yes, it's the IT-people I'm talking about here, not the average user/person, but the people who are employed in IT jobs.
  
  A lot of us tend to forget this part of the IT world, because we have long since made choices in our lives that puts as much distances between us and that IT world as possible.
  
  "I don't do windows" as the T-shirt says.
  
  But that world is still there, and the lost generation are still there, and they are in charge of IT at your dentist, you plumber, your town-hall and your sewage treatment plant.

• sl | Thu, 23 Aug 2012 00:59:00 UTC

  @Poul-Henning:
  
  It's never too late: http://code.google.com/p/plan9front/

• denisroyle@gmail.com | Thu, 23 Aug 2012 02:38:32 UTC

  Well there's a few things that we can say about the kids. They didn't create the environment in which they can tinker. They didn't set the criteria for entry. They didn't set up the education system that is meant to train them. & They didn't allow the standard to fragment. 
  
  The fact that a carbuncle can even be imagined let alone implemented demonstrates that the computer software engineer is no engineer at all. If you viewed the heat emissions off every process rather than how terse and **obtuse  you can get your C code then you might have gathered the empirical data you require to protect your standard. 
  
  Even the French knew that long long ago.  
  

• Dan Cross | Thu, 23 Aug 2012 12:40:02 UTC

  Poul-Henning:
  
  I understand what you are getting at, and appreciate your point.  But by presenting your argument in the language of the Cathedral and the Bazaar, you create an implicit false dichotomy: that if cathedrals are beautiful, then bazaars must be ugly.  While that may often hold true (e.g. autoconf), it does not follow that it is always true.  And I'm afraid you have constructed a situation where the majority of discussion now focuses on this false dichotomy rather than the real issue: finding and communicating examples of beauty in software, no matter what their source, in order to impart a sense of both quality and aesthetics in newer programmers.
  
  Did Eric Raymond's essay create a 'lost' generation of "IT professionals" who came into the industry to meet a huge demand for bodies during the dot-com boom, and then never left?  That's debatable, and I think it's more likely that Raymond gave many people the sense that they had the right to contribute, even if they didn't have the skill, knowledge or sophistication to do so.  And I'm positive it motivated many of those to contribute, even if they'd been better off learning a bit more beforehand.  But I also remember the dot-com era pretty well, and my sense was that most of the 'lost' types you are referring to had never heard of Eric S. Raymond, let alone the Cathedral and the Bazaar, that most wanted to work with resume-padding commercial systems, not "open source" and that most were just there for the money.  The IT types you refer to are much more closely aligned with the Windows administrators who are struggling to retain their positions as Microsoft and organizations like it (and the products they produce) slide further and further from relevance.
  
  Regardless, if we assume for a moment that Raymond's missive was one of the catalysts in creating this lost generation, is directly attacking his metaphor the most effective way to repair that damage?  Certainly not, as evidenced by many of the purely reactionary responses you see here.  An axiom of military training is that frontal assaults rarely work; enfilade fire through the flank is far more effective.  Attacking the Cathedral and the Bazaar is a frontal assault.
  
  But all of this is an aside.  It has little to do with what is beautiful and how to educate the next generation about that.
  
  My further point in my responses is that beauty is subjective.  I don't mean to retreat into the intellectual dishonesty of relativism (where one might start to question the difference between Jackson Pollock and a child finger-painting.  That's absurd.  Unless the child is some kind of virtuoso, of course Pollock is the better artist), but holding up Unix as a 'cathedral' is, at best, subjective.  Rob Pike, one of the guys who literally wrote the book on Unix, is often quoted as saying, "Not only is Unix dead, it's starting to smell really bad."  And that was in the early 1990s, well before both the dot-com boom and the Cathedral and the Bazaar.
  
  Yet there are still people telling us we should use pseudo-ttys and applications that use character addressing.  Are these the cathedrals we should aspire to?  I rather hope not.  In the early 1990s, there were still people pointing at Unix and Linux and deriding them as toys compared to systems like VMS and mainframes.  Much of the criticism was valid, even if those systems have largely disappeared (note that that has less to do with their quality and more to do with tying themselves to vendors and hardware platforms that have become increasingly irrelevant).  I remember VMS systems programers complaining about the "lost generation" of people trained on Unix, who didn't understand efficiency or the value of doing things like checking array boundaries!  Clearly, both are better than Windows, but despite that, there's never been any sort of golden age.

• Poul-Henning Kamp | Thu, 23 Aug 2012 19:39:19 UTC

  @Dan:
  
  I think we are in violent agreement here, I certainly agree that there's never been a golden age (-- yet!), and that no silver bullet is going to get us there.
  
  And that is the crux of the matter:  This column was inspired by the fact that the best book about the hard problem of software design I have read in a long time, is totally out of reach for the majority of people who most need to understand it.
  
  

• Vadim Goncharov | Thu, 23 Aug 2012 22:26:22 UTC

  Good article, and enough is said in discussion (especially on curiosity about the basics). I agree, however, that it is not directly Cathedral vs Bazaar (it's more effective not to attack this directly but better through flanks, though). It's rather social. Now IT is just a model of all other socium. First decades were just as in scientific community, then it became just like all other mankind, just a Bell curve. It is questionable, though, that IT could be better just like any other engineering dicippline - then we should like to the point about hubris and "Patches are welcome"-response.
  
  About autocrap: there is project for autoconf/automake replacement, http://sourceforge.net/projects/mk-configure/ - based on a different idea than autotools, rather than test about every other feature (like fortran compilers), just let the platform (and it's maintainers, you always have maintainers in distros these days) know what it supports. This leads to very compact and clean code, written in BSD make. Unfortunatelly it's NetBSD's bmake, which prevent it's spreading to at least FreeBSD's ports and then to being more popular. That's, BTW just another side of rant's main question and "Patches welcome"-response - author is just not interested in having it's child being more popular. It's not enough to me curious how to do better, it's also (and may be much more!) important to *push* better solutions to outside world. Or else mediocre solutions will dominate, just as in living nature's struggle for existence.
  
  P.S. If '...that quality happens only if somebody has the responsibility for it, and that "somebody" can be no more than one single person-with an exception for a dynamic duo.' - then... what about FreeBSD core@ vs single Linus? ;)

• Joseph | Thu, 23 Aug 2012 23:52:40 UTC

  I think a lot of the blame for the sorry state of things falls on our universities. All the CS classes I took in college in the early 00's either made us prove theorems or were auto-graded. This tends to foster a "if it works, it works" attitude. Sometimes the instructor claimed a part of the grade depended on coding "style", but this tended to be trivial things like having comments and descriptive variable names. The only software engineering course offered at my school taught OO with Java, which I think was simultaneously too general (it doesn't teach you how to solve real problems) and too specific (only one programming paradigm). So I feel like I never developed a systematic way to think about software design. I should mention that this school is considered a very prestigious technical university.
  
  Recently I've started to wonder why there aren't CS courses built around "reading the classics" e.g. the SysV source code. The only exception I know of is John Lion's course back in the day for which the Lion book was written. Without a good foundation of case studies to draw upon, is it any surprise that the prevalent problem-solving methodology of young programmers is to bang their heads against the wall until something gives?

• Ganesan Rajagopal | Fri, 24 Aug 2012 12:31:50 UTC

  Excellent rant :-). The comments about autoconf/libtool mess are spot on. Every other week I have the urge to sit down and rewrite libtool but don't have the energy to get into the cess pool. However, I think that the cathedral/bazaar metaphor is misdirected - especially looking at the number of comments which equate bazaar to the open source/free software movement. FreeBSD is open source but is typically thought of as the Cathedral model because historically very few people with commit access.
  
  I am sure many people would disagree but I consider Linux kernel also to be a cathedral in the sense that a single person or a small group steering the overall design/architecture with rigorous code review and emphasis on quality. What defines a "bazaar" is the lower barrier of entry with no private cliques. 
  
  Neither cathedral nor bazaar ensure quality. I find the rant a bit off track from that perspective. The festering mess that is autoconf/libtool is not a problem of the bazaar model. libtool made sense when HPUX/AIX/Tru64/Solaris all build shared libraries differently. It should just be thrown away today. But if you look at autoconf/libtool contributors over the years, you'll find one or two active contributors - certainly not an example for the bazaar model of development. autoconf/libtool are bad because they're outdated and need a replacement, period. It has absolutely nothing to do with Cathedral vs Bazaar.
  
  

• Dan Cross | Fri, 24 Aug 2012 19:18:48 UTC

  Poul-Henning:
  
  Yes, violent agreement in good ol' USENET fashion indeed.
  
  I agree with the crux of your argument, I just think it could be framed differently in a way that would make it more effective.  I hope you found my comments constructive.

• Zygo | Sat, 25 Aug 2012 07:09:04 UTC

  I think it is worth pointing out here that Firefox does, in fact, display TIFF images.

• Dan Haffey | Sat, 25 Aug 2012 23:39:06 UTC

  "and as an almost predictable application of the Peter Principle, rather than standardize Unix to eliminate the need for them, somebody wrote a program, autoconf, to write the configure scripts"
  
  Good rant, but a questionable invocation of the Peter Principle. It looks to me like regular old local reactions to unsolved global problems. I'm as repulsed by autoconf as the next guy, but I wouldn't accuse its creators of incompetence merely because they needed a solution and couldn't solve a big collective action problem like Unix standardization.

• Soren | Mon, 27 Aug 2012 09:02:10 UTC

  What's even more sad about the whole configure nonsense is that the amount of applications (or libraries or whatever is being configured) that actually use the information provided in config.h is miniscule. So not only is configure checking for a bunch of things that make no sense in modern day, but the output is almost invariably discarded.
  
  Most applications simply use configure to check if a specific set of libraries are installed and to allow users to enable/disable certain features. I'm fairly sure this could be accomplished with a fraction of the code and complexity of autoconf.

• Peter | Tue, 28 Aug 2012 02:49:26 UTC

  In my opinion, the problem is not if a piece of software stems from a cathedral or a bazaar. 
  It is also not about if it is configured with autotools (some claim "autohell" ;) ) or manually. 
  Configuring only happens only once, running some seldom used tests won't kill anybody, if the results are correct. (But it is indeed evil that no programmer can grasp autotools *easy*  - tools should be as easy to grasp like a hammer or a drilling machine )
  
  It is more about applications becoming overfeatured complexity hell, complexity which is not needed in 99.x% of all cases - like layers of wrapper libraries, using a complete library only for 1 or 2 functions, etc.
  
  It is also about things like "company standard" and "strategic alliance" where you are unfortunately forced to use that huge commercial database with the red trademark where a spreadsheet did work before (dept. leaders are ranked by burnt budget, you know).
  
  And it is about that cathedral project leaders as well as bazaar hackers are rewarded in money and fame by feature count and LOC and not by carefully asking for the real needs and leaving out unnecessary stuff.
  
  (sorry, if s/o calls the grammar police, I'm no native english speaker and it is late over here ;) )

• Martin | Tue, 28 Aug 2012 22:32:10 UTC

  Read the column and found myself nodding in agreement with some of the points that were being made. Also read some of the comments. Then went on with my business but apparently my subconscious went on thinking.
  
  Having considered the case, I guess my main issue with the rant is that there is an implicit appraisal; that the divine design of the cathedral is superior to the wasteful redundancy and haggling of the bazaar.
  
  But I think that is missing the point of the bazaar. It was never meant to be a cathedral. It serves different and far more mundane purposes. Blaming the bazaar for not being a cathedral is akin to blaming a workhorse for not being a Lipizzaner horse.
  
  Also, the wastefulness in the design of the bazaar is mirrored by the wastefulness of the cathedral. Figuratively speaking; the thick walls, the oversize windows, the architectural obsession with symmetries, the amount of energy it takes to heat up the place, the impracticality of the whole thing.
  
  In the real world, there are few bazaars and there are even fewer cathedrals. Most buildings fall somewhere in the middle between divinity and chaos. There are a lot of tradeoffs between usability, cost to manufacture, importance of maintenance, etc. I believe the same to be true for software. As developers, we don't want to fall into the pitfall of creating unmaintainable, fragile software, so we embrace modularity, test harnesses, separation of concerns, dependency injection, etc. On the other hand there is always a tradeoff between effort and reward, and although we may be professionally inclined to build divine masterpieces, we'll usually stop somewhere short of that. At a state where we can fix the worst inconsistencies and live with the remaining ones.

• Gregor | Fri, 31 Aug 2012 23:30:34 UTC

  For modern day attempt to recreate a modern computing environment as a 20,000loc cathedral see http://www.vpri.org/html/work/ifnct.htm
  
  Worth checking out.

• Charles | Wed, 05 Sep 2012 08:37:15 UTC

  Thanks for the thought-provoking article. I find it regrettable that folks turn things into a battle of ideologies and OS's. I don't think that was your point at all. But I can see why it can go that way. Still, the issues are more general than the article's examples.
  
  For example, serious contributors to TeX and friends tend to ask those in the know for the benefit and knowledge of their experience. Love it or hate it, one tends to get both good mentoring and swift bugfixes. There's a certain community cohesiveness and dedication. The output of the typesetter depends by nature on the elegance of design. Do people still see such necessity with programming, or has economics clouded that?
  
  By comparison, in spite of using paragraph/character styles, you can still get formatting anomalies in LibreOffice. One needs a plugin (!) to convert local formatting to named styles. It seems that emulating Office has resulted in inheriting problematic approaches from the design goals of Office. This is not the LO developers' fault; they inherited it. Can they hope to fix it?
  
  A more fundamental and general question involves whether economics is selecting for the brutish and ugly over the elegant and intelligent.
  
  I have to ask myself why I experience more problems with the latest Ubuntu LTS release than the previous one. Wasn't Linux supposed to free users from the frustrations of Windoze regression hell? So to get the broadest functionality, one tolerates frustration in Linux and begins to question the general development of software, open or closed. And for those who say to dump an OS if one doesn't like it, please. You are not going to dump your person-years of investment. Don't ask that of others.
  
  Maybe the barbarians are the cost accountants and executives who grab managers by the short hairs when enough financial gravitas is in play. I have seen the "enemy of done" argument used to justify questionable expectations for production deadlines and quality decisions. Maybe the suits trashed the cathedrals because today's computers hide the necessary complexity behind them. "When they're all supers, no one is super." Programmers and their craft have lost value as craftspeople and have become mere laborers. Open or closed, financial pressure and "instant experts" work against good design, fixing inadequacies, and correctness.

• Erik Nordstroem | Fri, 07 Sep 2012 01:36:25 UTC

  Having just finished reading the post and the comments (yes, all of them), I am left with a few questions.
  
  Like RORVI, I have a hard time finding good, reliable information and answers.
  
  I have read some books (I especially enjoyed many books published by O'Reilly), I am a bachelor student, and I have a part time job where I have responsibility over a few servers running Solaris 10. On my spare time, I use Ubuntu (at the student group of which I am a member), FreeBSD and Mac OS X.
  
  For many of the day-to-day things I do, the information I need is available in the relevant manual pages and user guides, and the assignments we get at the university are usually divided up in manageable "bite size" chunks such that they do not provide significant trouble (as long as i start in time; I've been known to postpone assignments untill the very end and not having enough time left to deliver when it is due).
  
  When it comes to larger projects, however, I have no real experience. I have tried to start a few projects on my own, but have rarely been able to get far with them (one exception was when an older and more experienced member of the student group I am in gave me some mentoring).
  
  One problem is that I don't know how much I should know.
  
  The reason that I write this comment, is because I was wondering if you could tell me what material I should read so that I get the foundation that I need to read and understand this book (The Design of Design).

• Tom Barrett | Mon, 10 Sep 2012 17:55:45 UTC

  Architecture is not an accident of nature, and neither is natural law. Disdain for one will bring about the downfall of the other.
  
  My own experience with open source has gone from, "This is cool!" (mid-90s) to, "This is...getting difficult to navigate," (early 2000s) to today's, "How the hell am I supposed to use this???"

• clockskew | Fri, 14 Sep 2012 11:42:06 UTC

  "Only open your mouth if it improves on silence."

• Ralph Dratman | Fri, 14 Sep 2012 17:49:23 UTC

  My belief is that we don't yet have computers powerful enough to put software development onto a rational footing. More precisely, even if we do already have computers marginally powerful enough, we are not yet routinely configuring them with the vast RAM spaces necessary to turn the existing base of open source code into a usable construction and optimization kit. The newly-architected computers I am discussing will cost significantly more per seat than this year's standard common-denominator system. Most hackers won't be able to afford them.
  
  All this is necessary because the problems being discussed here are so large and intertwined that, as a practical matter, they will only ever be fixed by a very smart computer program running on a very powerful computer. Here are some of the attributes this (currently imaginary) program will have: 
  
  First, the reconstruction program will be able to extract many algorithms and translate them to run in a new environment. It will be able to do this extraction from a combination of source code and fully linked running code. 
  
  Second, the program will be able to construct appropriately scaled emulation frameworks for many or most of the old environments, in order to deal with situations where the first-mentioned method fails. 
  
  Third, and perhaps most importantly, this advanced program-manipulation program must be able to study the ensemble of existing code statistically, as a series of random constructs, in order to follow the best-working existing implementations in the course of rebuilding things.
  
  The reconstruction program I describe is simultaneously simple in concept and stunningly complex in implementation detail. That is why the program will have to incrementally write itself in an evolutionary fashion, and why it will use techniques from the new bioinformatics world to manipulate all these program fragments created by all of us biological beings.
  
  For that to work, I don't need to tell anyone reading this that we will need -- at the very least -- significantly more powerful computing architectures and a hell of a lot of RAM space.

• Rod Grimes | Mon, 24 Sep 2012 10:19:25 UTC

  Your always full of such wonderfully flavored writing!!! 

• IGnatius T Foobar | Fri, 05 Oct 2012 12:28:54 UTC

  I call bullshit.  You're using FreeBSD as an example of excessive complexity and then declaring that all of Unix has this problem.
  
  I've got news for ya sunshine, there's this thing called Linux where it's easy.  Applications install with one command and all of the dependencies come on board automatically.  It's just as easy as Windows and Mac.

• Howard B. Golden | Sat, 03 Nov 2012 21:17:43 UTC

  Mr. Kamp, what you wrote needed to be said. Thank you for saying it.
  
  Along this line, I have been thinking about a "quality improvement institute, factory and service" that might help. Open source is well established now in industry, and the problems are apparent to all. However, the benefits are seen as even greater. This creates an opportunity to improve existing open source software and future development and maintenance processes. (Not surprisingly, this same opportunity exists for closed source software.) Therefore, there should be a willingness on the part of government and industry to fund quality improvement. (I don't attempt to specify whether such organization(s) are for-profit or not-for-profit, but their products would need to be open for use without compensation for them to benefit open source developers.)
  
  There are several approaches that should be pursued: (1) Improved training and education for developers; (2) Identification and promulgation of better or best practices (in choices of languages, tools and development processes); (3) Identification of existing software most in need of remediation or replacement; (4) Funding remediation/replacement of the most important needs; (5) Outreach to existing projects with offers to help adopt better practices; (6) Outreach to academia to encourage training in better practices; (7) Advice for starting new projects following best practices appropriate for the project size; (8) Funding development of tools to support better or best practices. (9) Review of existing tools and offering constructive suggestions for improvement, perhaps with funding.
  
  I would be happy to elaborate on these ideas with anyone interested.

• Daniel Dumitriu | Tue, 06 Nov 2012 06:07:47 UTC

  Yessss, for the "violent agreement" !
  It's just horribly disheartening how many of the "contributors" to the comments didn't get it. Most rushed to "contribute" without checking, first, who Don Knuth or Peter Naur, Per Brinch Hansen, Dijkstra, Fred Brooks and, even, Alan Turing, are/were. 
  Nor did they figure out why they should care...
  

• k | Wed, 07 Nov 2012 03:28:32 UTC

  fortunately for you people, configure has been replaced:  http://aiju.de/b/configure

• Cellar | Fri, 09 Nov 2012 11:56:05 UTC

  Apropos requiring perl AND python, last time I checked that was an artifact of the ports collection where an unexplained digotomy exists between running sanity tests and building the damn thing, since if you'd care to look up the original project page it only lists things like libiconv, and neither python nor perl. And lo and behold, it compiles fine if you kick both of them out, and add the documented requirements.
  
  
  Now, I'm not a fan in the least of the concept of glib or the people that wrote it, and there are many projects under the same banner where such requirements specifications are only inferrable through repeated runs of configure and figuring out what the crashes and error messages mean, but in this particular case it appears some of that unprofessionalism has seeped into the ports collection's fabric itself. Which is... not good, for that particular package is pretty widely needed.

• HELLYO protocol | Fri, 09 Nov 2012 13:29:20 UTC

  @Erik Nordstroem
  
  > When it comes to larger projects, however, I have no real experience.
  
  Do you mean "larger projects that still can be handled by one person"? I would say the most important point is, when starting out,  make abstraction of coding. Think about what problem you want to solve, what elements and objects should be "there", how they interact, what the architecture is, what the design is (go to the library and borrow Thimbleby's "Press On - Principles of Interaction Programming"). Read up on the principles behind the infrastructure you will be building on (OS, VMs, database, web interface). Study existing architecture and its philosophy (go to the library and borrow Eric Raymond's "Art of Unix Programming") but not religiously, search for exiting solutions to connect to, study the APIs you intend to use. Select the tools you will need, if necessary script your own. Learn good coding style (go to the library and borrow Steve McConnell's "Code Complete"). Take a look at other programming languages (like Erlang), read a book on "Programming Patterns" and a book on Object-Oriented Design, too (avoid UML if you can). Once your project connects many people, start reading about and applying an Agile Method like SCRUM, that will keep things under control and in movement. Fumble onwards from there.
  
  > One problem is that I don't know how much I should know. The reason that I write this comment, is because I was wondering if you could tell me what material I should read so that I get the foundation that I need to read and understand this book (The Design of Design).
  
  Don't get hang up on that. Understanding comes with time and experience, and failure, not with reading. But start reading anyway.
  
  I am currently at the point that I know that I will never know what I should know. Once you start getting Excel sheets of labeled and checkboxed requirements appearing on your desk while folders of commitee-designed standards block access to the fire exit, it's time to put your affairs in order. 
  

• Justin Wiley | Fri, 09 Nov 2012 15:51:51 UTC

  This is an old, pointless argument, and the fact the ACM has chosen to highlight it is another sign of their increasing irrelevance.  What Poul is really saying is: "remember the good old days?  Weren't they better than now?  Aren't we smarter than our children?"
  
  No, they were not, and no you are not.  I remember cathedral systems.  They used proprietary protocols.  They were based on proprietary hardware that couldn't evolve and couldn't be upgraded.  And only the chosen few knew how they worked.  ATT Unix had proprietary manuals for gods sake...you really want to go back to that?
  
  Progress in the 21rst century will not come from a giant ivory tower with the anointed few slowly building masterpieces.  There's too much demand, and too much complexity to tackle.  A decent solution and a working system today will always beat a perfect system 15 years from now.  I think 21rst century developers and architects understand and embrace that reality.
  
  If you can't Poul, please step aside...or by all means keep posting articles for the folks at the ACM retirement home to read.

• sed gawk | Fri, 09 Nov 2012 21:28:50 UTC

  Speaking as a younger *NIX coder,GNU/Linux (Suse) was my introduction to *NIX, I'd not seen BSD until a couple of years later, so in my mind the coreutils was the GNU core utils.
   
  This I think has something to do with why I use Linux over BSD as my workflow is available across all *NIX (gcc/vi/coreutils) - perhaps the same for other people.
  
  The autotools are a bit difficult to learn but I think they are worth the effort, M4 is a bit fugly but
  with a bit of practise and ruthless factoring into small macros its not that bad, (Top tip add banner lines to your macros so you can spot the output in the generated configure script)
  
  With the autotools,  I get cross-compilation/packaging/ cross-compiled unit-tests execute in a cross environment/transparent replacement of missing functions at link time/standardised argument handling which generates help messages/binutil access and ability to mung various assets (images/sql etc) in to my code with very little effort. 
  
  Mostly I copy my build-aux and m4 directories into a new project and write a simple configure.  My heart sinks when I have to work on project that doesn't use autotools. 
  
   So I think the autotools survived because when you take into account everything it provides, it's streets ahead of everything else, so @David MacKenzie - I do not accept your apology and I thank you for making the choices you did.
  
  

• Daniel Martinez | Sat, 10 Nov 2012 16:53:16 UTC

  Even if the article made some good points it should also admit that the lost generation is a result of a something better. Just to be practical let's atalk about possible solutions:
  
  How can you stop poor designs in the bazaar model? Education? (i.e. universal education) it would be great but, how? On the other hand, how can you prevent the uneducated and unexperienced programmers from contributing/reusing code in your project? Copyright??? (please no). Better designs? If I understood "The Cathedral and the Bazaar" well, the bazaar model did not override Brooks' work on design, it just enriched and complemented it. I think it is a bit unfair to blame the bazaar for everything.

• fishburger | Mon, 12 Nov 2012 11:06:21 UTC

  You do realize that autoconf was written before the "lost generation" was out of their nappies? Autoconf also fails as example of the lack of responsibility, we even had the person responsible showing up here to comment. As for an elegant solution to that particular problem - how could anyone have been responsible for standardizing the plethora of Unixen when their very existence spun on the premise of them being non-standard? And the Unix cataclysm itself: again brought to us by you the pre-lost-generation dinosaurs.
  
  Nobody has the incentive to optimize their software compilation times just so Mr. Unix Geek can build an entire software universe in less than a day on his laptop. I think it's a freaking miracle how little repetition you found, when you consider what's involved: a decentralized global software factory.
  
  Short of it, you can only have responsibility if you also have control. You don't control the people and projects making the software that you cram into your ports collection, so stop moaning.

• warren | Tue, 13 Nov 2012 23:50:37 UTC

  I learned from this article that CRUFT IS BAD.  But nobody takes time out to remove CRUFT until it breaks.
  
  Open source needs a Department of Cruft Removal. Because Raymond was right but so is Poul.  Design is overrated (Sorry Poul) but Architecture is 90% about Cruft Control (Point for Poul).
  
  W

• Farid Hajji | Sun, 02 Dec 2012 21:53:24 UTC

  As a long time user of all kinds of Unices in the Eighties, and FreeBSD aficionado since FreeBSD 1.x up until now, I totally agree with phk@. This said, there's a troubling point that didn't get enough attention yet, IMHO.
  
  The FreeBSD ports collection, just like full Linux-like (F)OS(S) distributions, are actually big meta-programs full of internal inter-dependencies and hidden couplings. That's the nature of the beast.
  
  While we have figured out how to refactor individual programs under one's control, even big ones, we have yet to figure out how to refactor such huge meta-programs like our FOSS ecosystem in order to clean them of their increasing number of meta-warts and meta-bitrot.
  
  Maybe, this is not a mere engineering task, but something bigger in need of fundamental methodologies. Methodologies we don't know about yet because they were never needed before. I guess this is an active area of research, or it ought to be.
  

• Ron Minnich | Wed, 05 Dec 2012 18:12:30 UTC

  A good example is worth much discussion. I'm working to start a web site where we link to good code and good articles about code. A link to this article would be the headline. Examples of high quality, portable code, written with good practices, not afflicted with the current set of mistakes we see (autotools, libtool, and so on) would include:
  - Go
  - Plan 9 ports
  - Inferno
  
  to start. These systems all build on many OSes and there is not one single bit of configure scripts in the group. All have many contributors and benefit from the bazaar, but are hardly chaotic. I'm sure we could find more. I've just gotten burned again (by autoconf, of course) and I feel we really need to start educating people. The current state of the art in most open source software is not very good.
  
  Some of you know me; if interested, let me know, I could use the help. I'm not that good at setting up web pages :-)

• Ranjit Singh | Fri, 21 Dec 2012 13:46:15 UTC

  For people asking "where do we start?" I'd highly recommend: "The Unix Programming Environment" (Kernighan & Pike, 1984) -- even today it is highly relevant and will open your eyes
  to the Unix philosophy simply by doing. Additionally you get to see what has been around since the
  beginning, which is useful for code portability.
  
  It also has the best yacc tutorial I've seen anywhere: I defy anyone to work through chapter 8 and not feel enlightened. (Making it work with ANSI C is a useful exercise, and not at all difficult.)
  The only caveat I'd mention is that he uses $* for shell parameters, where he should use "$@"; he mentions the latter after a while, but doesn't use it later on where it would be better.
  
  I've been coding for decades, and I wish I'd read it when it first came out, as it would have changed my life even more back then. It's available quite cheaply second-hand. You'll need a basic understanding of C, and should really have done the exercises in "The C Programming Language" (Kernighan & Ritchie 2nd ed) first. (Even if you haven't, buy both:)
  
  The best modern reference is: "Advanced Programming in the Unix Environment" [2nd ed.] (Stevens & Rago, 2005)  www.apuebook.com but it's a much heavier read. If you trace Stevens' work back, the original "Unix Network Programming" (one volume, from 1990) is in the same series as K&P(1984) and "The Design of the Unix Operating System" Bach (1986) both of which it refers to quite a bit. The latter is an excellent companion piece to "Lions' Commentary on UNIX 6th edition with Source Code" John Lions (Peer-to-Peer) which was mentioned above. Again, highly recommended, as are UNP volume 1: Sockets (3rd ed) and vol 2: IPC (2nd ed afaik.) www.unpbook.com
  
  Also, consider #friendly-coders on IRC chat.freenode.net if you want some helpful advice. They'll likely redirect you to language-specific channels, but they're there if you need an explanation of what they've said, advice on approaches or algorithms, or you just need some reassurance (especially given the prevalence of idiocy-elitism on IRC.) (Disclaimer: I work with one of the founders.)

• M. Simon | Wed, 26 Dec 2012 06:08:40 UTC

  The problem is "C". But people are so used to it that they don't even see it. 

• E. Sarmas | Wed, 26 Dec 2012 12:52:51 UTC

  I totally agree with above comment. The problem is in the use of "C".
  We need at last a new start (maybe Java or a cleaner C with Java influences but certainly not C++).
  

• M. Simon | Wed, 26 Dec 2012 22:11:51 UTC

  Let me add that I'm currently working with a guy who managed 5,000 programmers at AT&T and who is a crack programmer himself and he agrees with me. If all you have is a 75 tool pocket knife and what you really need is a hammer....

• David Ormand | Wed, 02 Jan 2013 19:03:16 UTC

  Good article, and controversial.  The takeaway I'm getting is that the root problem is the lack of a "proper" education, even from "accredited" public universities and tech schools, for CS/IT people.  Well, I'm not CS/IT, I'm just a relic who graduated from the TI-99/4A to doing embedded code (assembly, C, C++, Ada) in defense products (which for the most part do NOT have any formal software quality requirements!), and I'm very aware of my lack of training.  Given the shortcomings of regular school-based education (especially for those with full-time employment) and the difficulty of getting mentoring from old-timers, any suggestions for self-teaching?  Like a preferred reading list; I see suggestions for McConnell's "Code Complete", and I've heard of the "Mythical Man-Month" (and see its regular application - in the wrong way - in my company), and I've got the K&R books and the Humphrey's "Personal Software Process" book (my company is a CMM level 5 outfit, so I grok process) and some textbook for "Design Patterns" that is heavy on UML.  What I think would be sweet is a website with recommended reading lists for various software-related disciplines, such as system programming, network, crypto, embedded, etc.

• Dan Cross | Thu, 14 Feb 2013 21:22:03 UTC

  My list is here:  http://pub.gajendra.net/2012/09/stone_age

• Dave Wyland | Mon, 01 Apr 2013 18:55:21 UTC

  Great discussion. 
  
  One responder proposed an interesting idea, that Moore's law and advancing computer hardware performance indirectly allowed sloppy coding and software anarchy. An interesting idea. Moore's Law tended to improve performance by ~30%/year (CPU clock rate), year-after-year, inexorably. Hard to adjust to on a continuing basis. At some point, a new programmer can just assume that hardware performance is infinite, and that we should just go play in the new, digital ocean. 
  
  But that has quietly changed. Moore's Law went into the ICU in 2004. I had a 3.5 GHz Pentium 4 in 2004. There has never been a 4 GHz Pentium, to this day. Moore's law hit the scaling wall. Before 2004, each shrink meant cheaper silicon, lower power and higher speed. After 2004 (~90 nm), each shrink meant cheaper silicon, *higher* power and *lower* speed. Intel and others struggle with magic chemicals to make 2.5 GHz processors today. And no proven, near term solution to this problem has emerged. 
  
  I think this means we are in the clean up phase, where we get better results by cleaning up, reducing and streamlining code rather than adding code. This will be interesting to watch, whether you subscribe to the cathedral or bazaar model. 
  

• ZenBowman | Wed, 08 May 2013 01:31:53 UTC

  Great discussion, and it reminds me of an Alan Kay interview:
  
  """
  Binstock: You once referred to computing as pop culture.
  
  Kay: It is. Complete pop culture. I'm not against pop culture.... I consider jazz to be a developed part of high culture. Anything that's been worked on and developed and you [can] go to the next couple levels.
  
  Binstock: One thing about jazz aficionados is that they take deep pleasure in knowing the history of jazz.
  
  Kay: Yes! Classical music is like that, too. But pop culture holds a disdain for history. Pop culture is all about identity and feeling like you're participating. It has nothing to do with cooperation, the past or the future  it's living in the present. I think the same is true of most people who write code for money. They have no idea where [their culture came from]  and the Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs.
  """
  
  I'm a relative newbie, at the age of 30, but I found the vast majority of my CS curriculum (apart from a few courses on compilers, operating systems and AI) very superficial. This past year I studied SICP and received more illumination from those video lectures than I ever did in any coursework done during my PhD. I was lucky enough to work at a really good lab (USC/ISI) during my PhD, and the one thing I'm very thankful for is that I met some genuine programming wizards, who imbibed in me an appreciation for the history of our field. 
  
  Without an appreciation for our history, we will just continue to loop in cycles. The author is spot-on.