January/February 2018 issue of acmqueue

The January/February issue of acmqueue is out now

The Bike Shed


  Download PDF version of this article PDF

Please Put OpenSSL Out of Its Misery

OpenSSL must die, for it will never get any better.

Poul-Henning Kamp

The OpenSSL software package is around 300,000 lines of code, which means there are probably around 299 bugs still there, now that the Heartbleed bug — which allowed pretty much anybody to retrieve internal state to which they should normally not have access — has been fixed.

That's really all you need to know, but you also know that won't stop me, right?

Securing a computer network connection is not really hard in theory. First you let exceptionally skilled cryptographers design some cryptographic building blocks. You will need a good hash-function, a good symmetric block cipher, and a good asymmetric cipher. Next you get exceptionally skilled crypto-protocol designers to define how these building blocks should be tied together in a blow-by-blow fashion. Then an exceptionally skilled API designer defines how applications get access to the protocol, via a well-thought-out and error-resistant API with well-chosen and reliable default values and a good error reporting mechanism. Then exceptionally skilled programmers implement the algorithms and protocols according to the API in high-quality, fully-audited and analyzed library source code. And after that the application programmer — who's usually anything but exceptionally skilled — finally gets to write code to open a secure connection.

But we're not quite done.

We need to ensure that the compiler correctly translates the high-level language to machine instructions. And only exceptionally skilled compiler programmers will do! We also need to make sure that the computing environment is trustworthy — there can be no bugs, mistakes, backdoors, or malware in system libraries or in the operating system kernel. And written by? You got it , exceptionally skilled kernel programmers obviously! Of course this is all in vain if the CPU does not execute instructions faithfully, but exceptionally skilled CPU engineers will no doubt see to that.

Now — finally — we can securely transmit a picture of a cat from one computer to another.

That wasn't so bad, was it?

Specifying a dream team as I did is not without pitfalls, as OpenSSL contains this code:

* The aim of right-shifting md_size is so that the compiler
* doesn't figure out that it can remove div_spoiler as that
* would require it to prove that md_size is always even,
* which I hope is beyond it.
div_spoiler = md_size >> 1;
div_spoiler <<= (sizeof(div_spoiler)-1)*8;
rotate_offset =
   (div_spoiler + mac_start - scan_start) % md_size;

Be honest — would you have considered a proven, correct, compiler improvement a security risk before I showed you that?

Of course you wouldn't! It was proven correct, wasn't it?

And that's where our security model, if you can call it that, breaks down. There is so much code involved, that no one has a clue if it all works or not.

On my computer the numbers are roughly:

Operating system Kernel:        2.0 million lines of code
Compiler:                       2.0 --//--
C language runtime library:     0.5 --//--
Cryptolibrary:                  0.3 --//--

Each additional programming language involved will add about a million lines. Using a graphical user interface roughly doubles the number, and a browser almost doubles it again. A reasonable estimate is that the computer on which you read this has at least 100, but more likely 1,000 bugs through which your security can be compromised. If those bugs were unique to your computer, that wouldn't be too bad. But that's not the case, as they are the exact same bugs on millions and millions of computers, and, therefore, every bug has pandemic potential. And — as if the task were not difficult enough — there are people actively trying to sabotage the result, in order to make "intelligence gathering" easier and less expensive. Apparently some of those exceptionally skilled engineers and scientists on our dream-team have hidden agendas — otherwise the leadership of the NSA would be deeply incompetent.

Then there are Certificate Authorities. Their job is to act as "trust-anchors" so that once you have established a secure connection you also know who you are talking to.

CA's trustworthiness is a joke.

Do you trust:


to authenticate a connection to your bank?

You don't even know what "GÜVENLİĞİ HİZMETLERİ" means, right?

I certainly don't, and I don't trust them either!

In August 2012 they issued bogus certificates with which the holder could claim to be Google. TÜRKTRUST, of course, maintains that it was all "a one-time mistake," that "no foul play was involved" and that "it can never happen again."

Nobody believes a word of it.

Yet, your browser still trusts them by default, just as it trusts hundreds of other more or less suspect organizations to tell the truth. So we're connecting using bloated buggy software, trusting shady government-infiltrated CAs to assure us who we talk to?

There must be a better way, right?

Well there isn't.

We have never found a simpler way for two parties with no prior contact, to establish a secure telecommunications channel. Until the development of asymmetric cryptography (1973-1977) it was thought to be impossible. And if you want to know who is at the other end, the only way is to trust some third party who claims to know. No mathematical or cryptographic breakthrough will ever change that, given that our identities are social constructs, not physical realities.

So if we want e-commerce to work, we have to make the buggy code work and we should implement something more trustworthy than the "CA-Mafia."

And that brings me back to OpenSSL — which sucks. The code is a mess, the documentation is misleading, and the defaults are deceptive. Plus it is 300,000 lines of code that suffer from just about every software engineering ailment you can imagine:

and so on and so on.

And it's nobody's fault.

No one was ever truly in charge of OpenSSL, it just sort of became the default landfill for prototypes of cryptographic inventions, and since it had everything cryptographic under the sun (somewhere , if you could find out how to use it), it also became the default source of cryptographic functionality.

I'm sure more than one person has thought "Nobody ever got fired for using OpenSSL".

And that is why everybody is panicking on the Internet as I write this.

This bug was pretty bad, even as bugs in OpenSSL go, but my co-columnist at ACM Queue, Kode Vicious, managed to find a silver lining: "Because they used a 'short' integer, only 64 kilobytes worth of secrets are exposed."

And that is not the first nor will it be the last serious bug in OpenSSL, and, therefore, OpenSSL must die, for it will never get any better.

We need a well-designed API, as simple as possible to make it hard for people to use it incorrectly. And we need multiple independent quality implementations of that API, so that if one turns out to be crap, people can switch to a better one in a matter of hours.



[email protected]

Poul-Henning Kamp ([email protected]) is one of the primary developers of the FreeBSD operating system, which he has worked on from the very beginning. He is widely unknown for his MD5-based password scrambler, which protects the passwords on Cisco routers, Juniper routers, and Linux and BSD systems. Some people have noticed that he wrote a memory allocator, a device file system, and a disk encryption method that is actually usable. Kamp lives in Denmark with his wife, son, daughter, about a dozen FreeBSD computers, and one of the world's most precise NTP (Network Time Protocol) clocks. He makes a living as an independent contractor doing all sorts of stuff with computers and networks.

© 2014 ACM 1542-7730/14/0400 $10.00


Originally published in Queue vol. 12, no. 3
see this item in the ACM Digital Library



Arvind Narayanan, Jeremy Clark - Bitcoin's Academic Pedigree
The concept of cryptocurrencies is built from forgotten ideas in research literature.

Geetanjali Sampemane - Internal Access Controls
Trust, but Verify

Thomas Wadlow - Who Must You Trust?
You must have some trust if you want to get anything done.

Mike Bland - Finding More Than One Worm in the Apple
If you see something, say something.


(newest first)

Displaying 10 most recent comments. Read the full list here

Keith | Tue, 27 Oct 2015 05:13:20 UTC

It is refreshing to see that a reputable software engineer has openly criticized OpenSSL. OpenSSL is a terrible piece of software. I started using it in 2001.

The API makes no sense at all. The functions that you call, the way they're called and the sequence the functions are called in cannot be determined from reading the documentation. You have to rely on examples and look at the source code of accompanying programs.

Function names are made up from macros. This means you cannot easily find them in the source code. These names can change from one release to the next.

Upgrading to a newer version of OpenSSL will almost always break application code that uses the API. I have concluded that only the authors of OpenSSL can know how to use it. The only portable and reliable way to use OpenSSL is to call the application program with the appropriate command line arguments.

Alex | Wed, 06 Aug 2014 02:44:12 UTC


1. What is the difference between the heartbleed bug and other major problems/bugs with openssl and those of proprietary SSL implementations?

If you flip to the back of the book, you'll find answers to all odd numbered questions:

1. The bug was found, reported, and corrected.

I find it humorous things like this are highlighted when these problems exist in ALL software, not just open source software, and certainly not just openssl. When is the last time any proprietary SSL implementation was audited by a third party? If never is your guess, you're probably right. What makes you think any of their code is any better? Do you just trust their code is more secure? I don't -- I'm only a weekend hacker (and I wouldn't even call myself that level) and I wouldn't say that ~50% of the professional devs I meet are some combination under-educated, under-skilled, and extremely lacking in critical thinking/problem solving skills, and there are a lot of people in the industry who would agree with me. Surely some of those people must be working on SSL implementations, or other security-centric software.

Dan Cross | Fri, 09 May 2014 16:27:23 UTC

tal: it evaluates to neither. :-) It's 2^17 (arithmetic shift has lower precedence than addition).

Something I learned long ago is that if one is to successfully program in some language, one must write idiomatic code in that language. Unfortunately, I think that one of the problems with writing C code is that there is a lack of idiom around things like parenthesizing expressions that can lead to surprising problems if one is not familiar with the precedence rules of the language.

Martin Leiser | Fri, 25 Apr 2014 21:19:11 UTC

hmmm... I think it is all about Architecture. Software has bug. Period. On the other side: We know for 50 years how a MMU works. Hardware has bugs too, but much less. Why is to web server, openssl, the private key in one adress space? Use stunnel with PKCS11 to off load the private key. And the web server in a different process.

Two simpl rules: Divide and conquer. Least privilege.

Add a third rule: CPUs are cheap.. switch off optimizations for security relevant code.

But all that is not the trend of today is multithreading, throwing 100.000.000 Lines of code in one application server.....

Dont blame the openssl programmers, it is their bug, but our fault.

Sebastiaan | Thu, 17 Apr 2014 22:45:53 UTC

Must be because English is not my native language, but in my book these commits don't fall in the category 'typo'...




Sebastiaan | Thu, 17 Apr 2014 22:21:07 UTC

@willy just picked two random commits and yup, it's scary!

A new go to fail in the making? I'm no C programmer, but these two commits removed curly brackets and subsequently converted a single line of code inside an 'if' statement to two lines... Sure this code is correct, but *visually* harder a lot harder to spot?



Morten Jensen | Wed, 16 Apr 2014 20:37:41 UTC

Great article, was referred here from the version2.dk debate: http://www.version2.dk/blog/det-skandaleramte-sikkerhedslag-og-hvad-goer-vi-nu-57357

> /* > * The aim of right-shifting md_size is so that the compiler > * doesn't figure out that it can remove div_spoiler as that > * would require it to prove that md_size is always even, > * which I hope is beyond it. > */ > div_spoiler = md_size >> 1; > div_spoiler rotate_offset = > (div_spoiler + mac_start - scan_start) % md_size; > > Be honest  would you have considered a proven, correct, compiler improvement a security risk > before I showed you that? > > Of course you wouldn't! It was proven correct, wasn't it?

I do embedded development, and I sometimes experience that I have to qualify a variable volatile, to get the compiler to stop optimizing the code in ways that spoils the intended behavior. I would imagine the OpenSSL code could do the operation on a volatile-qualified variable and achieve the same effect without having to resort to nasty tricks ? From my K&R on the volatile qualifier: "The purpose of volatile is to force an implementation to suppress optimization that could otherwise occur."

Peter Kriens | Wed, 16 Apr 2014 14:39:18 UTC

Isn't this also a reason to not program this kind of code in C(++) instead of a managed language?

Kevin L | Wed, 16 Apr 2014 11:32:12 UTC

As an experienced engineer in both the process industries (MS in ChemE) and software (BS in CompSci), I think that data breaches at the SSL/TLS layer need to be treated in much the same way as pressure breaches in physical systems: you use multiple independent systems to verify each other **while online**, and if any one of those systems sees a problem you interrupt the flow (terminate the connection).

Applied to this case, that would be two or more SSL libraries executing in separate processes that would each perform the SSL/TLS protocol independently, with the HTTP process comparing their outputs and only sending bytes to the other side when the outputs fully agree. If they disagree, the HTTP daemon would log an error and terminate the connection. In this fashion each library would expose the bugs in the other, while protecting the users. Eventually both libraries would get better, only failing simultaneously for errors in the protocol design or the underlying crypto math. (This is also the approach used in aviation: 3 computers, 2 to check for hardware faults and a third in an independent implementation to check for logic faults.)

Outside of generating nonces and random numbers that would have to be passed between the libraries and the HTTP process, this doesn't seem like it would be too hard to do.

Javier | Wed, 16 Apr 2014 04:01:52 UTC

Very nice article

Aside from DANE, Convergence(http://convergence.io) is another alternative to CA. OpenBSD is cleaning OpenSSL(http://undeadly.org/cgi?action=article&sid=20140415093252)

Displaying 10 most recent comments. Read the full list here
Leave this field empty

Post a Comment:

© 2018 ACM, Inc. All Rights Reserved.