The Bike Shed

Security

  Download PDF version of this article

Quality Software Costs Money - Heartbleed Was Free

How to generate funding for FOSS


Poul-Henning Kamp


The world runs on free and open-source software, FOSS for short, and to some degree it has predictably infiltrated just about any software-based product anywhere in the world.

What's not to like about FOSS? Ready-to-run source code, ready to download, no license payments—just take it and run. There may be some fine print in the license to comply with but nothing too onerous or burdensome.

Your TV? It has a Linux or {Net|Free}BSD computer inside it. So does the copier-printer/multifunction machine at the office, as do the entertainment consoles in your car and in your kids' bedrooms. Code reuse has finally happened, but there is a footnote: you get what you pay for.

Earlier this year the OpenSSL Heartbleed bug laid waste to Internet security, and there are still hundreds of thousands of embedded devices of all kinds—probably your television among them—that have not been and will not ever be software-upgraded to fix it. The best way to prevent that from happening again is to avoid having bugs of that kind go undiscovered for several years, and the only way to avoid that is to have competent people paying attention to the software.

In the case of OpenSSL, it is painfully obvious that nobody was paying attention. There is a commercial entity called The OpenSSL Foundation, but as far as anyone can determine, it is a FIPS (Federal Information Processing Standard)-test consultancy and that is pretty much all it does.

According to an article in the Wall Street Journal,5 the OpenSSL Foundation received around $2,000 a year in donations to maintain the OpenSSL code. That amounts to approximately 20 hours of attention per year from a good programmer to maintain north of a half-million lines of code. That certainly explains why the OpenSSL bug tracker was write-only and why the code is still overcomplicated by support for vintage operating systems such as VMS and 16-bit Windows.

Predictably, money started pouring in after the Heartbleed bug caused havoc: IT security is a game you can win only by losing first. So OpenSSL maintenance, testing, and quality assurance is funded now, and we're good, right? No, we are not! Many other FOSS projects (in addition to OpenSSL) are badly understaffed because they are underfunded—and we need to fix that.

Why FOSS Needs Money

About the only thing GNU Project founder Richard Stallman and I can agree on when it comes to software freedom is that it's "Free as in free speech, not free beer."

I really hope the Heartbleed vulnerability helps bring home the message to other communities that FOSS does not materialize out of empty space; it is written by people. We love what we do, which is why I'm sitting here, way past midnight on a Saturday evening, writing about it; but we are also real people with kids, cars, mortgages, leaky roofs, sick pets, infirm parents, and all kinds of other perfectly normal worries.

The only way to improve the quality of FOSS is to make it possible for these perfectly normal people to spend time on it. They need time to review patch submissions carefully, to write and run test cases, to respond to and fix bug reports, to code, and most of all, time just to think about the code and what should happen to it.

It would not even be close to morally defensible to ask these people to forgo time to play with their kids or walk their dogs in order to develop and maintain the software that drives the profit in other people's companies. The right way to go—the moral way to go—and by far the most productive way to go is to pay the developers so they can make a living from the software they love. And not just for a month or two: good programmers, like most everyone else, prefer stable jobs so they can concentrate on the job rather than wasting time chasing the next gig or the next sponsor.

How to Fund FOSS

One way to fund FOSS is simply to hire the FOSS maintainers, with the understanding that they spend some amount of company time and resources on the project. Experience has shown that these people almost invariably have highly desirable brains, which employers love to throw at all sorts of interesting problems, and that tends to erode the "donated" company time. A lot of FOSS has been, and still is, developed and maintained this way, with or without written agreements or even company knowledge of this being the case.

A big thank you to those employers who do this deliberately!

These companies should add their support of FOSS to their lists of corporate social responsibilities, along with their sponsorships of local soccer teams and their funding of scholarships. They are doing something for the greater common good, which they should be proud of and get proper credit for.

Another way to fund FOSS is for software projects to set up foundations to collect money and hire developers. This is a relatively complex and expensive undertaking. You run straight into the murkiest corners of tax codes, so this approach is available only for larger projects. While several projects have gone this route, their ability to raise money varies significantly.

The Apache Foundation is probably the largest of all these FOSS foundations. A few entities "adopt" smaller projects inside their fields of interest. This generally works OK but cannot easily be generalized to different areas.

An easier and cheaper way, the one advocated here, is simply to throw money at the developers, the way the FreeBSD and Varnish communities have done with me. Forget about tax deductions and all that; simply have the developer send your company an invoice every so often and stuff it into some account in the IT department labeled "misc. software licenses" or "expert assistance" or whatever will fly under the radar in your company. It takes only a dozen companies worldwide to keep a top-notch programmer's nose in the code of a FOSS project full time—and this is probably code that is likely a lot more critical to a company's operations than anybody really likes to think about.

Here's my story...

FreeBSD Community Funding

My first crowdfunding of FOSS was in 2004 when I solicited the FreeBSD community for money3 so that I could devote three to six months of my time to the FreeBSD disk-I/O subsystem.

At the time I had spent 10 years as one of the central and key FreeBSD developers, so there were no questions about my ability or suitability for the task at hand. In 2004, however, crowdfunding was not yet "in," and I had to figure out how to do it myself. My parents raised me to believe that finances are a private matter, but I concluded that the only way you could reasonably ask strangers to throw money at you would be to run an open book where they could see what happened to the money. So I opened my books.

The next dilemma was my rate. Again, I had always perceived my rate to be a private matter between me and my customers. Because my rate is about half of what most people expect (as I won't work for most projects but only on things I really care about), I was concerned that publishing my rate would undercut friends and colleagues in the FreeBSD project who made a living consulting. There was no way around it, however, so I published my rate, making every attempt to distinguish it from a regular consulting rate, and I never heard any complaints.

Having agonized over the exact text and then tested it on a couple of close friends in the FreeBSD project, I threw the proposal out there and waited. I had a perfectly safe fallback plan—you have to when you have two kids and a mortgage—but I really had no idea what would happen next.

Worst case, I would cause the mother of all bikesheds (http://bikeshed.org/2) to get thrown out of the FreeBSD project, and I would be widely denounced for my "ideological impurity" with respect to FOSS. Best case, I expected to get maybe one or two months funded.

The FreeBSD community responded overwhelmingly. My company has never sent as many invoices as it did in 2004, and my accountant nearly blew a fuse. Suddenly I found myself in a situation I had never even considered: how to stop people from sending me money. I had set up a PayPal account, and at least at that time, there was no way to prevent people from dropping money into it. In the end, I managed to yell loud enough, so I was overfunded by only a few percent. I believe that my attempt to deflect the surplus to the FreeBSD Foundation gave that group a little boost that year, too.

Today people who do crowdfunding have "stretch goals" to soak up overfunding, but in 2004, I was in uncharted waters; whatever happened, I wanted to contain any fallout from my experiment in a single fiscal year. I also wasn't sure how it would work out in terms of the social dynamics of the project, so I didn't want it to extend past half a year.

In total, I made 27,000 euros, which kept my kids fed and my bank happy for the six months that I worked on the project.

And work I did.

I've never had a harsher boss than during those six months, and it surprised me how much it stressed me. I felt like I was working on a stage with the entire FreeBSD project in the audience wondering if I would deliver the goods or not. As a result, the 187 donors certainly got their money's worth, as most of that half year I worked 80-hour weeks, which made me decide not to continue, despite many donors indicating that they were perfectly willing to fund several more months.

Varnish Community Funding

Five years later, having developed Varnish HTTP Cache Version 1.0 for Norway's Verdens Gang newspaper, and having exploded into a vacuum in Web-content delivery, I decided to give community funding a go again.

Wiser from experience, I structured the VML (Varnish Moral License)4 to tackle the issues that had caused me grief the first time around: contact first, then send money, not the other way around; also focus on fewer, larger sponsors, rather than individuals sending me 10 or 15 euros, or even in one case one euro, which lingered in an unused PayPal account. I run even more-open books this time, and on the VML Web pages you can see how many hours I have worked, along with a terse one-line description of what I did for every single day I've been working under the VML since 2010.

I also decided to be honest with myself and my donors: one hour of work was one hour of work—nobody would benefit from me dying from stress. In practice it doesn't quite work like that: there is plenty of thinking in the shower, as well as e-mails and IRC (Internet Relay Chat) answers at all hours of the day and night, and a lot of "just checking a detail" that happens off the clock because I like my job, and nothing could stop me anyway.

This is the funding model I want to "sell" for other FOSS projects, because it works. In each of the years 2010, 2011, and 2013 I worked around 950 hours on Varnish (funded by the community) in addition to work I did for my other customers. In 2012 I worked only 589 hours, because I was building a prototype computer cluster to do adaptive optics real-time calculations for the European Southern Observatory's Extremely Large Telescope1 (there was just no way I could say no to that contract).

In 2014 I have hours available to do even more Varnish work, but despite my not-so-subtle hints, the current outlook is still for only 800 hours to be funded. I'm crossing my fingers that more sponsors will appear now that Varnish Version 4 has been released (nudge, nudge, wink, wink — He said knowingly).

The VML is not an ideal funding model in the sense that with a single exception, none of the big corporations that deliver massive amounts of HTTP traffic with Varnish has participated. A number of smaller concerns have kept the project alive and kicking. In a smaller company the CEO, or at least the CTO, is more likely to know what FOSS products the company uses, and quite likely keeps abreast of developments and announcements. In a larger organization, on the other hand, bigger issues drown out such "details" as a fundraising plea before they rise to a level where a decision can be made, or where the fear that the marketing department must be involved spikes the idea. I can vividly imagine how Dilbert would fail to convince his PHB (pointy-haired boss) that the company should give money away because it is using some free software.

The good news is that the PHB doesn't have to understand: if a dozen midsize companies each decide to spend $500 every month, that would do wonders for any piece of FOSS—if the money makes it into the hands of the right person(s). And there's that little detail: finding the Right Person.

There is no way to shortcut that: each FOSS project, each sponsoring company, each potential maintainer will have to look one another in the eye and decide if they think this is worth a shot or not. There will be failures and disappointments—that is unavoidable—but if we do not fund good people to maintain critical FOSS projects, there will be no end to the Heartbleed.

References

1. European Southern Observatory. The European Extremely Large Telescope; http://www.eso.org/public/teles-instr/e-elt/.

2. Kamp, P.-H. 1999. Why should I care what color the bikeshed is; http://bikeshed.org/.

3. Kamp, P.-H. 2004. Fundraising for FreeBSD development; http://people.freebsd.org/~phk/funding.html.

4. Kamp, P.-H. The Varnish Moral License; http://phk.freebsd.dk/VML.

5. Yadron, D. 2014. After Heartbleed bug, a race to plug Internet hole. Wall Street Journal (April 9); http://online.wsj.com/news/articles/SB10001424052702303873604579491350251315132 (login required).

LOVE IT, HATE IT? LET US KNOW

feedback@queue.acm.org

Poul-Henning Kamp (phk@FreeBSD.org) is one of the primary developers of the FreeBSD operating system, which he has worked on from the very beginning. He is widely unknown for his MD5-based password scrambler, which protects the passwords on Cisco routers, Juniper routers, and Linux and BSD systems. Some people have noticed that he wrote a memory allocator, a device file system, and a disk-encryption method that is actually usable. Kamp lives in Denmark with his wife, son, daughter, about a dozen FreeBSD computers, and one of the world's most precise NTP (Network Time Protocol) clocks. He makes a living as an independent contractor doing all sorts of stuff with computers and networks.

© 2014 ACM 1542-7730/14/0600 $10.00

acmqueue

Originally published in Queue vol. 12, no. 6
see this item in the ACM Digital Library


Tweet



Related:

Geetanjali Sampemane - Internal Access Controls
Trust, but Verify


Thomas Wadlow - Who Must You Trust?
You must have some trust if you want to get anything done.


Mike Bland - Finding More Than One Worm in the Apple
If you see something, say something.


Bob Toxen - The NSA and Snowden: Securing the All-Seeing Eye
How good security at the NSA could have stopped him



Comments

Displaying 10 most recent comments. Read the full list here

Logan | Sat, 21 Jun 2014 14:21:18 UTC

Your TV? It has a Linux or {Net|Free}BSD computer inside it. So does the copier-printer/multifunction machine at the office, as do the entertainment consoles in your car and in your kids' bedrooms. <- Quite a few printers also has OpenBSD inside: https://www.google.mu/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&cad=rja&uact=8&ved=0CDIQFjAE&url=http%3A%2F%2Fsupport.ricoh.com%2Fbb_v1oi%2Fpub_e%2Foi%2F0001027%2F0001027949%2FVB2227858%2FB2227858.pdf&ei=GoalU9aiC8iC4gSpv4CQCQ&usg=AFQjCNFNHKTv6O7ZOS5UY4VFi4BBW9pC3Q&bvm=bv.69411363,d.bGE That said, awesome article :-)

Stanny | Fri, 15 Aug 2014 20:37:04 UTC

I have never seen the amount of money spent have any effect whatsoever on software quality. Quality software is written by quality programmers, and it's a rare thing -- regardless of how much money you wave around.

Lodewijk | Fri, 15 Aug 2014 20:47:49 UTC

Don't you think the strict maintainer structure is the cause of much grief? Aside from practical and simple it's really anti-control-your-own-software. Shouldn't we fund features and forks more directly? And why should you give the software to these large companies for free? Do you realize that the PHB is just an honest manifestation of capitalism in this case? In Dilbert the guy usually makes bad choices, but if he says "No." to giving money away for no profit that seems about right to me. It's the right individual choice in a game theory problem. I see you think, but it would be much better if they /did/ fund it. And I agree. But gametheory doesn't work that way, and neither does reality. So, what can we do to keep software open but still require some sort of funding?

Joel A. Seely | Fri, 15 Aug 2014 20:49:09 UTC

The better analogy I've heard about Open Source Software is that it is "Free" as in "Puppy". Sure the little scamp is cute in the box at the grocery store, but don't forget you're going to need to train it, feed it, groom it, and take it to the vets. All of that takes time and money. It's the same for free software.

Pierre | Fri, 15 Aug 2014 22:29:18 UTC

I think it's more accurate to say that you don't get what you don't pay for. (Did the buyers of Enron, Bre-X or Worldcom stock "get what they paid for"?)

Alex Henzell | Fri, 15 Aug 2014 23:41:31 UTC

Tax redirection: Howabout the option for companies to pay, say, 10% of their tax to FOSS projects of their choice rather than the government.

David | Sat, 16 Aug 2014 08:49:33 UTC

This problem I think is a mirror of that faced by artists: viz commissions, grants and the open market. The importance of federal funding of the arts is clear, but (in these narrow contexts) FOSS is not art as it does not contribute to 'a richer cultural life'. It contributes to a richer software-based economy. Perhaps a tax on business to fund a department of FOSS that would disburse grants independently. We could call it an OpenLevy :)

Fellow Traveler | Sun, 17 Aug 2014 22:03:16 UTC

If you want the companies to fund the development, just use the GPL. This enforces a consortium-like arrangement where all the companies feel safe to contribute to open source, and are incentivized to do so. Notice that most of the pull requests to Linux are funded by large corporations who are otherwise in business competition with each other. The problem is not funding for open-source. Rather, the problem is funding for BSD-style open-source. Because when it's GPL, the problem ceases to exist.

Carlos | Mon, 18 Aug 2014 16:42:39 UTC

@Fellow Traveler. "If you want the companies to fund the development, just use the GPL" I don't think that's the problem. See the Apache Foundation. The truth, companies are takers, open-source is the mean, not the end for those companies. So, they usually contribute when they can use the software in their private business freely without having to open their own code and when that open-source software is a core part of their business, so they need to maintain it. The problem with most of those "take-a-lot-contribute-little" companies is a matter of *assumptions*, which is a very common problem in engineering: "it is open-source -> it is robust and tested -> we can trust".

CK Raju | Tue, 09 Dec 2014 06:51:29 UTC

Box Toxen and Poul-Henning Kamp make certain viewpoints with regard to the role of funds in secure software. Can anyone generalise and formulate a theory using aberrations as examples ? I think the method deployed by Poul-Henning is wrong. While funds may contribute to secureness of software, proprietary-ness of software will effectively prevent anyone from knowing what's really going behind the scene. When patches could be effectively used to alter the functionality of a given proprietary software, how could anyone verify truthfulness of a software to a function based on mere claims ? Access to software sources must be a extended in order to conclusively verify absence of mischievious code. Another perspective stems from the fact that proprietary software can only be improved at the developer's place. The case with Free Software is different, as any student could pick up a sample project and go on to complete the project as part of academic work - an exercise which is normally impossible with proprietary software.
Displaying 10 most recent comments. Read the full list here
Leave this field empty

Post a Comment:







© 2014 ACM, Inc. All Rights Reserved.