Dear Diary

On keeping a laboratory notebook

Dear KV,

I recently joined a medical company as a data scientist—crunching numbers on its latest drugs—and one thing I’ve noticed is that biologists and other noncomputer people I work with keep notebooks. Every single experiment or idea is carefully recorded in a physical notebook that is then checked by someone they work with. The process seems laborious, but I’m told it’s required for their work. I can’t remember ever having to record experiments for my computer science courses in college, admittedly more than a few years ago, but I know you’ve written about keeping a debug log. Is that the same thing?

Noted

Dear Noted,

Congratulations on the new job, and I hope one of the side benefits is cheap drugs. Not that KV would take a gig just to get access to cheap drugs, but that might make my day job a little more palatable.

Yes, you have a good memory. I have written about using a debug log to figure out problems and apply the scientific method to debugging. While a debug log is helpful, it’s not the same thing as a laboratory notebook. Did you know that the first actual bug was a moth found in a relay of the Harvard Mark II computer in the late 1940s by Dr. (and, eventually, Rear Admiral) Grace Murray Hopper? She not only found the bug, but also taped it into the debug log. (You can find the log entry and image online at https://americanhistory.si.edu/collections/search/object/nmah_334663.) If only all our bugs were large enough to see!

I have to say I find it surprising that a data scientist would not know about laboratory notebooks. In the physical sciences such as physics and chemistry, as well as in the medical sciences, such notebooks are required. In fact, undergraduates in those classes must turn them in for grading as part of their studies. Most lab notebooks remain on paper, although there are also expensive online systems for keeping lab notes. The online systems are meant to protect companies from predatory patent trolls and to provide a digital way of proving that their invention, whatever that may be, was first.

We in computer science can learn from the long-standing processes in other sciences, and we should get ourselves up to date with the 18th century. Maintaining a proper lab notebook has been a thing in the noncomputer sciences for several hundred years, and it’s time computer science earned its science badge by going back to the future.

Most people who spend their days in front of computers would probably balk at keeping a paper notebook of experiments instead of a more convenient electronic form. Let’s see how we could keep an electronic laboratory notebook without paying through the nose.

Laboratory notebooks are different from other notebooks and logs in a few ways. A laboratory notebook should contain a series of experiments and experimental notes that work out a hypothesis. The hypothesis doesn’t need to be grand: “Light is both a wave and a particle, and, therefore, gravitation will divert the path of light.” It can be much like the debug log entries mentioned previously, or it can be a hypothesis about a particular change to the code: “Substituting a Fortran mathematical routine in this set of calculations for its C++ equivalent will reduce the time of the computation by a factor of 2.” The hypothesis is something that can be tested and should be stated simply enough that anyone else conversant in our scientific endeavor can understand it.

Each experiment should have a title and an unambiguous date. KV prefers Day - Month Name - Year, since even a non-English speaker can figure out that the thing in the middle isn’t a numerical day or a year. KV also keeps a state variable associated with the entry: Is it not-started (a mad idea KV came up with as he leapt from the shower), or is it in-progress (lots of head banging on the desk trying to build and run the experiment)? Once complete, it can be marked COMPLETE or FAILED. More experiments FAIL than COMPLETE for KV, but maybe you’re better than that. The goal isn’t to have a perfect record of COMPLETEs, but to have that one experiment that satisfies the hypothesis.

Once you have your hypothesis, you have to test it, but how? The next section of the notebook lays out the procedures, calculations, equipment, software libraries, and everything that is required for the experiment. The How section should include everything about the experiment you can think of. Many experimenters have failed because they didn’t include everything they used, and so they could not tell that, for example, running the experiment with different pieces of software on the same system would affect the outcome, or even mask an effect they were looking for.

Write down everything you can think of, and then ask a colleague if you’ve left out anything. In the physical sciences, this type of pair verification is common.

Now it’s time to run your experiment and record your observations, which is your next section. Much as you did in the How section, record everything you see (hear, taste, smell—whatever senses you can bring to bear on the problem will later serve you well when you do the analysis).

Note that observations are just that: Just observe, be as one with the experiment, be the observer. Later, you can analyze.

Eventually the observations will, ideally, complete, subject to the Turing limit, and it will be time to analyze the observations. This is where you summarize everything you thought you observed and try to discover if you have proven or disproven your hypothesis. You remember the hypothesis, right? It’s at the top of the entry! Go reread it. Did the work you did prove or disprove anything? If you had false assumptions or observations, cross them out (there are convenient fonts for this now) but leave them in the entry.

Finally, the experiment may have given you ideas for further experiments. As that great Irish scientist and comedian Dara Ó Briain said: “Science knows it doesn’t know everything; otherwise, it’d stop.” Write down what you might want to figure out next.

To be presumptuous, I’ve attached a template page from one of my own lab notebooks at the end of this column. It should be used as a starting point and not as gospel, because the gospel according to KV would contain a lot of blaspheming.

If more computer scientists (a group in which I include anyone who uses computers to solve problems) acted like scientists, maybe we wouldn’t have to have bar fights over the age-old question, “Is computing an art or a science?” And now, hold my beer, I have to go explain this to someone else in the bar

Recommended resources

For an excellent introduction to keeping a lab notebook, check out Howard M. Kanare’s Writing the Laboratory Notebook from 1985.

Here is the github repo for notebooks that I set up: https://github.com/gvnn3/labnotebook.

And here is the Org Mode Notebook file: https://raw.githubusercontent.com/gvnn3/labnotebook/main/samples/notebook.org.

Example Notebook Entry

1 Sep 2023 Getting the time in user space :in_progress:

Hypothesis

What are you trying to show? Be specific about what you are measuring: time, transactions, size of data, latency, etc.

How

(Procedures, calculations, equipment)

• The details of How

• Equipment::CPU, memory, disk, network features, and base performance characteristics

• Software::Name, version, options

• Scaffolding scripts, executables::Location, name, command arguments passed

• Commands typed::Use the script(1) command before you start any interactive session that will run test or other measurement code. This command captures all commands typed and their output for later use.

Observations

• Describe all that happens (planned or unplanned) during the experiment in narrative text.

• Raw experimental data::Tables and other data that are small enough to fit directly into the notes may be placed in this section.

• Large output files::Point to the files, signed and stored in a repo, that contain any relevant output.

Org mode has a way to strike out text, C-c C-x C-f +, which inserts plus-mark characters that make a strikeout as in the following line:

~~+- This idea did not work out and therefore is struck out.+~~

In a laboratory notebook we NEVER remove ideas; we leave them and strike them out. Yes, we can recover text via the version-control system but that’s not sufficient for our purpose. We need to have the strikeouts remain to work as a real lab notebook.

Data analysis

Processing of raw data, graphs, interpretations

• Summarize your interpretation of the results of the experiment in narrative text.

• Summary tables, graphs, images, etc. may be placed directly into the notes.

• Large tables, graphs, images, etc.::Point to the files, signed and stored in a repo, that contain any relevant output.

Ideas for future experiments

A list of further experiments that you believe would tease out new information.

George V. Neville-Neil works on networking and operating-system code for fun and profit. He also teaches courses on various subjects related to programming. His areas of interest are computer security, operating systems, networking, time protocols, and the care and feeding of large code bases. He is the author of The Kollected Kode Vicious and co-author with Marshall Kirk McKusick and Robert N. M. Watson of The Design and Implementation of the FreeBSD Operating System. For nearly 20 years, he has been the columnist better known as Kode Vicious. Since 2014, he has been an industrial visitor at the University of Cambridge, where he is involved in several projects relating to computer security. He earned his bachelor’s degree in computer science at Northeastern University in Boston, Massachusetts, and is a member of ACM, the Usenix Association, and IEEE. His software not only runs on Earth, but also has been deployed as part of VxWorks in NASA’s missions to Mars. He is an avid bicyclist and traveler who currently lives in New York City.

Originally published in Queue vol. 21, no. 5—
Comment on this article in the ACM Digital Library