Getting Off the Mad Path

Debuggers and assertions

Dear KV,

I just spent the better part of a week debugging a problem in a piece of code that required the programmer—me—to rerun the same 100 lines of code, adding more and more debugging statements until the bug finally revealed itself. Since this particular code was in a serverless environment, a traditional debugger was not an option. Instead, I had to fire off the function, over and over and over and over again. Eventually, my eyes would glaze over, and I'd get up, walk around my home office, and sit down to start all over again. I somehow cannot believe that this is the right way to attack a problem like this, but when I asked other members of my team for help, they told me that tedium was to be expected. Clearly that cannot be so, can it?

Teed Off at Tedium

Dear Teed,

So many clichés relate to this topic, I feel I could just quote them all here, make myself a stiff drink, and call it a day. But that seems lazy and unfair, so I'll just make the drink and try to avoid the clichés.

Most of us understand that computers are great at doing the same thing over and over and over and over again. On bad days, KV feels that this is the only thing that computers are actually good for and that he is trapped inside a surreal, Dr. Seuss nightmare landscape, in which strangely drawn, brightly colored characters spew nonsense while causing all my code to crash in ways that are clearly not possible. Unlike some of my colleagues, I don't microdose, so I know that this is my imagination and I can pull the emergency brake on the nightmare and escape at any time without waiting for the dose to wear off.

Debugging code is often iterative, and iterative processes fall into the over-and-over category. You run the function, it gives the wrong output, you scratch your head and think, "Oh, it must be X," and you change whatever X is and run it again, and get another wrong result. Then you think, "Oh, it's not X, it's Y," and you change X back and then try Y, run the code again, and... . Down this path lies madness, with or without microdosing, but it is a well-worn path in our industry that we have all walked many times.

There are several strategies for leaving this path of madness.

You've already alluded to the first way off the mad path, and that is to use a debugger. Unfortunately, there are too few good debuggers in the world, and programmers have rarely been inclined to improve this situation, because—as any venture capitalist will tell you—there is very little money to be made in software tooling.

One might think that some of us who work in the industry would work on better debuggers just for our own sanity, but as KV has pointed out before, programmers are optimists: We always think that this time we've got it right and we can write more code rather than debugging what's in front of us now. This is not to say that there are no debuggers; there are, and some of them even work—a few even work well—but those that do are few and far between.

KV continues to grind his teeth as he sees code loaded with debugging statements that would be totally unnecessary if the programmers who wrote the code could be both confident in and proficient with their debuggers. If one is lucky enough to have access to a good debugger, one should give extreme thanks to whatever they normally give thanks to and use the damn thing! When KV was a much younger programmerette, he worked for a boss who insisted that all code be run in the debugger first—and when KV can, that's what he does. If the program works well in the debugger, then you can try it without the training wheels and see how it goes.

Another way off the mad path, and the one that's probably most relevant to your debugger-less environment, is to add some form of assertion to each and every line of the failing function. As the programmer, you certainly should know what the function is supposed to do, so assert that! A good language would let you do this and then let you hide the assertions from view, but I've rarely seen a language do that.

More often, there is an assertion library or 10 of them (because people are so lazy, they won't do a search for what they need before just hacking up their own) that you can use to do this. Do not go down the iterative path to madness by adding assertions only to the things you think are causing the bug. Assert every blasted line in the function and then—if the stars align—you will find the bug on the next run of the code, because you are checking every bit of the function line by line.

Finally, if you are fighting with a function that fails only one out of n times, you'll have to take advantage of the fact that computers really are good at doing things over and over and automate the execution of the failing function so that you can catch the failure. Step up a level, write a loop, make sure it can catch the failure, and then go back—as I said—and make a good, stiff drink. KV is a fan of the Vesper martini but will attest that if the failure he's looking for is intermittent, he can get through about six of these, and by then, well, debuggery is no longer what's on his mind. Cheers!

Related Kode Vicious articles

Outsourcing Responsibility
What do you do when your debugger fails you?
https://queue.acm.org/detail.cfm?id=2639483

Wanton Acts of Debuggery
Keep your debug messages clear, useful, and not annoying.
https://queue.acm.org/detail.cfm?id=2048938

Debugging on Live Systems
It's more of a social than a technical problem.
https://queue.acm.org/detail.cfm?id=2031677

Kode Vicious, known to mere mortals as George V. Neville-Neil, works on networking and operating-system code for fun and profit. He also teaches courses on various subjects related to programming. His areas of interest are code spelunking, operating systems, and rewriting your bad code (OK, maybe not that last one). He earned his bachelor's degree in computer science at Northeastern University in Boston, Massachusetts, and is a member of ACM, the Usenix Association, and IEEE. Neville-Neil is the co-author with Marshall Kirk McKusick and Robert N. M. Watson of The Design and Implementation of the FreeBSD Operating System (second edition). He is an avid bicyclist and traveler who currently lives in New York City.

Originally published in Queue vol. 19, no. 6—
Comment on this article in the ACM Digital Library

More related articles:

Charisma Chan, Beth Cooper - Debugging Incidents in Google’s Distributed Systems
This article covers the outcomes of research performed in 2019 on how engineers at Google debug production issues, including the types of tools, high-level strategies, and low-level tasks that engineers use in varying combinations to debug effectively. It examines the research approach used to capture data, summarizing the common engineering journeys for production investigations and sharing examples of how experts debug complex distributed systems. Finally, the article extends the Google specifics of this research to provide some practical strategies that you can apply in your organization.

Devon H. O'Dell - The Debugging Mindset
Software developers spend 35-50 percent of their time validating and debugging software. The cost of debugging, testing, and verification is estimated to account for 50-75 percent of the total budget of software development projects, amounting to more than $100 billion annually. While tools, languages, and environments have reduced the time spent on individual debugging tasks, they have not significantly reduced the total time spent debugging, nor the cost of doing so. Therefore, a hyperfocus on elimination of bugs during development is counterproductive; programmers should instead embrace debugging as an exercise in problem solving.

Peter Phillips - Enhanced Debugging with Traces
Creating an emulator to run old programs is a difficult task. You need a thorough understanding of the target hardware and the correct functioning of the original programs that the emulator is to execute. In addition to being functionally correct, the emulator must hit a performance target of running the programs at their original realtime speed. Reaching these goals inevitably requires a considerable amount of debugging. The bugs are often subtle errors in the emulator itself but could also be a misunderstanding of the target hardware or an actual known bug in the original program. (It is also possible the binary data for the original program has become subtly corrupted or is not the version expected.)

Queue Readers - Another Day, Another Bug
As part of this issue on programmer tools, we at Queue decided to conduct an informal Web poll on the topic of debugging. We asked you to tell us about the tools that you use and how you use them. We also collected stories about those hard-to-track-down bugs that sometimes make us think of taking up another profession.