March/April 2020 issue of acmqueue The March/April 2020 issue of acmqueue is out now

Subscribers and ACM Professional members login here

March/April 2020

Commit to Memory:
Power to the People

  Jessie Frazelle

Reducing datacenter carbon footprints

By designing rack-level architectures, huge improvements can be made for power efficiency over conventional servers, since PSUs will be less oversized, more consolidated, and redundant for the rack versus per server. While the hyperscalers have benefited from these gains in power efficiency, most of the industry is still waiting. The Open Compute Project was started as an effort to allow other companies running datacenters to benefit from the power efficiencies as well. If more organizations run rack-scale architectures in their datacenters, the wasted carbon emissions caused by conventional servers can be lessened.

Commit to Memory, Power,

Dark Patterns: Past, Present, and Future

  Arvind Narayanan, Arunesh Mathur, Marshini Chetty, Mihir Kshirsagar

The evolution of tricky user interfaces

Dark patterns are an abuse of the tremendous power that designers hold in their hands. As public awareness of dark patterns grows, so does the potential fallout. Journalists and academics have been scrutinizing dark patterns, and the backlash from these exposures can destroy brand reputations and bring companies under the lenses of regulators. Design is power. In the past decade, software engineers have had to confront the fact that the power they hold comes with responsibilities to users and to society. In this decade, it is time for designers to learn this lesson as well.

Privacy and Rights, Web Services

Is Persistent Memory Persistent?

  Terence Kelly

A simple and inexpensive test of failure-atomic update mechanisms

Power failures pose the most severe threat to application data integrity, and painful experience teaches that the integrity promises of failure-atomic update mechanisms can't be taken at face value. Diligent developers and operators insist on confirming integrity claims by extensive firsthand tests. This article presents a simple and inexpensive testbed capable of subjecting storage devices, system software, and application software to ten thousand sudden whole-system power-interruption tests per week.

Failure Testing


January/February 2020

The Morning Paper:
How Do Committees Invent? and Ironies of Automation

  Adrian Colyer

The formulation of Conway's law and the counterintuitive consequences of increasing levels of automation

My first choice, from 1968, is entitled "How Do Committees Invent?" This is the paper that gave us Conway's law, and while we all know that law today, author Melvin E. Conway provides a lot of great material that led up to the formulation of the law that bears his name.
For my second choice we go forward in time to 1983, with Lisanne Bainbridge's "Ironies of Automation." It's a classic treatise on the counterintuitive consequences of increasing levels of automation, and something oh-so-relevant to this forthcoming decade.

The Morning Paper

Kode Vicious
Kode Vicious Plays in Traffic

With increasing complexity comes increasing risk.

The first design principle of any safety-critical system must be simplicity. Systems such as Ethernet are known to be complex, and so it is a poor choice for use in a safety-critical system. But I hear accounting screaming about the cost of extra wiring in the harness of the car's control system. "Think how much money we can save if all the signals go over a single pair of wires instead of a harness with 10!"

Kode Vicious

Case Study
To Catch a Failure: The Record-and-Replay Approach to Debugging

A discussion with Robert O'Callahan, Kyle Huey, Devon O'Dell, and Terry Coatta

When work began at Mozilla on the record-and-replay debugging tool called rr, the goal was to produce a practical, cost-effective, resource-efficient means for capturing low-frequency nondeterministic test failures in the Firefox browser. Much of the engineering effort that followed was invested in making sure the tool could actually deliver on this promise with a minimum of overhead.

Case Studies, Debugging,

Escaping the Singularity
The Best Place to Build a Subway

  Pat Helland

Building projects despite (and because of) existing complex systems

Many engineering projects are big and complex. They require integrating into the existing environment to tie into stuff that precedes the new, big, complex thing. It is common to bemoan the challenges of dealing with the preexisting stuff. Many times, engineers don't realize that their projects (and their paychecks) exist only because of the preexisting and complex systems that impose constraints on the new work. This column looks at some sophisticated urban redevelopment projects that are very much part of daily life in San Francisco and compares them with the challenges inherent in building software.

Systems, Escaping the Singularity

Demystifying Stablecoins

  Jeremy Clark, Didem Demirag, and Seyedehmahsa Moosavi

Cryptography meets monetary policy

Self-sovereign stablecoins are interesting and probably here to stay; however, they face numerous regulatory hurdles from banking, financial tracking, and (likely) securities laws. For stablecoins backed by a governmental currency, the ultimate expression would be a centrally banked digital currency.

Networks, Security

Commit to Memory:
Chipping away at Moore's Law

  Jessie Frazelle

Modern CPUs are just chiplets connected together.

Smaller transistors can do more calculations without overheating, which makes them more power efficient. It also allows for smaller die sizes, which reduce costs and can increase density, allowing more cores per chip. The silicon wafers that chips are made of vary in purity, and none are perfect, which means every chip has a chance of having imperfections that differ in effect. Manufacturers can limit the effect of imperfections by using chiplets.

Commit to Memory, Computer architecture, Processors,

Everything Sysadmin:
Communicate Using the Numbers 1, 2, 3, and More

  Thomas A. Limoncelli

Leveraging expectations for better communication

The human brain reacts differently to lists of different lengths. When you align what you say with what the human brain expects, you communicate more effectively. In this column I'll explain how to leverage the way the brain reacts to various quantities to make your speaking and writing more effective.

People often use lists of various sizes when communicating. I might have 2 reasons for supporting the new company strategy. I might tell you my 3 favorite programming languages. I might make a presentation that describes 4 new features. There is 1 vegetable that I like more than any other.

The length of the list affects how the audience interprets what is being said. Not aligning with what the human brain expects is like swimming upstream. Given the choice, why would anyone do that?

Business and Management, Everything Sysadmin



November/December 2019

Special issue on the critical role of human perception in software

The Morning Paper:
The Way We Think About Data

  Adrian Colyer

Human inspection of black-box ML models; reclaiming ownership of data

In "Stop Explaining Black-box Machine-learning Models for High-stakes Decisions and Use Interpretable Models Instead," Cynthia Rudin makes the case for models that can be inspected and interpreted by human experts. And in "Local-first Software: You Own Your Data, in Spite of the Cloud," Martin Kleppmann describes how to retain sovereignty over your data.

Data and Databases, The Morning Paper

Kode Vicious
Master of Tickets

Valuing the quality, not the quantity, of work

Many silly metrics have been created to measure work, including the rate at which tickets are closed, the number of lines of code a programmer writes in a day, and the number of words an author can compose in an hour. All of these measures have one thing in common: They fail to take into account the quality of the output. If Alice writes 1,000 lines of impossible-to-read, buggy code in a day and Carol writes 100 lines of well-crafted, easy-to-use code in the same time, then who should be rewarded?

Kode Vicious

Commit to Memory:
Securing the Boot Process

  Jessie Frazelle

The hardware root of trust

The goal of a hardware root of trust is to verify that the software installed in every component of the hardware is the software that was intended. This way you can verify and know without a doubt whether a machine's hardware or software has been hacked or overwritten by an adversary. This is an introduction to a complicated topic, but the intention is to provide a full picture of the world of secure booting mechanisms.

Commit to Memory, Computer architecture, Processors, System evolution

Revealing the Critical Role of Human Performance in Software

  David D. Woods, John Allspaw

It's time to revise our appreciation of the human side of Internet-facing software systems.

Understanding, supporting, and sustaining the capabilities above the line of representation require all stakeholders to be able to continuously update and revise their models of how the system is messy and yet usually manages to work. This kind of openness to continually reexamine how the system really works requires expanding the efforts to learn from incidents.


Above the Line, Below the Line

  Richard I. Cook, M.D.

The resilience of Internet-facing systems relies on what is below the line of representation.

Knowledge and understanding of below-the-line structure and function are continuously in flux. Near-constant effort is required to calibrate and refresh the understanding of the workings, dependencies, limitations, and capabilities of what is present there. In this dynamic situation no individual or group can ever know the system state. Instead, individuals and groups must be content with partial, fragmented mental models that require more or less constant updating and adjustment if they are to be useful.

Development, Web Services

Cognitive Work of Hypothesis Exploration During Anomaly Response

  Marisa R. Grayson

A look at how we respond to the unexpected

Four incidents from web-based software companies reveal important aspects of anomaly response processes when incidents arise in web operations, two of which are discussed in this article. One particular cognitive function examined in detail is hypothesis generation and exploration, given the impact of obscure automation on engineers' development of coherent models of the systems they manage. Each case was analyzed using the techniques and concepts of cognitive systems engineering. The set of cases provides a window into the cognitive work "above the line" in incident management of complex web-operation systems.


Managing the Hidden Costs of Coordination

  Laura M.D. Maguire

Controlling coordination costs when multiple, distributed perspectives are essential

Some initial considerations to control cognitive costs for incident responders include: (1) assessing coordination strategies relative to the cognitive demands of the incident; (2) recognizing when adaptations represent a tension between multiple competing demands (coordination and cognitive work) and seeking to understand them better rather than unilaterally eliminating them; (3) widening the lens to study the joint cognition system (integration of human-machine capabilities) as the unit of analysis; and (4) viewing joint activity as an opportunity for enabling reciprocity across inter- and intra-organizational boundaries.

Debugging, Development

Beyond the "Fix-it" Treadmill

  J. Paul Reed

The Use of Post-Incident Artifacts in High-Performing Organizations

Given that humanity's study of the sociological factors in safety is almost a century old, the technology industry's post-incident analysis practices and how we create and use the artifacts those practices produce are all still in their infancy. So don't be surprised that many of these practices are so similar, that the cognitive and social models used to parse apart and understand incidents and outages are few and cemented in the operational ethos, and that the byproducts sought from post-incident analyses are far-and-away focused on remediation items and prevention.

Development, Quality Assurance



September/October 2019

Kode Vicious
Numbers Are for Computers, Strings Are for Humans

How and where software should translate data into a human-readable form

Unless what you are processing, storing, or transmitting are, quite literally, strings that come from and are meant to be shown to humans, you should avoid processing, storing, or transmitting that data as strings. Remember, numbers are for computers, strings are for humans. Let the computer do the work of presenting your data to the humans in a form they might find palatable. That's where those extra bytes and instructions should be spent, not doing the inverse.

Data and Databases, Kode Vicious

Commit to Memory:
Opening up the Baseboard Management Controller

  Jessie Frazelle

If the CPU is the brain of the board, the BMC is the brain stem.

In 2011 Facebook announced the Open Compute Project to form a community around open-source designs and specifications for data center hardware. Facebook and other hyperscalers provide their solutions to the problems that come with running data centers at scale. Since then, the project has expanded to all aspects of the open data center, including baseboard management controllers (BMCs), among many others. In this column, I focus on the BMC. It's an introduction to a complicated topic; some sections just touch the surface, but the intention is to provide a full picture of the world of the open-source BMC ecosystem, starting with a brief overview of the BMC's role in a system, touching on security concerns around the BMC, and then diving into some of the projects that have developed in the open-source ecosystem.

Commit to Memory, Open Source

Blockchain Technology: What Is It Good for?

  Scott Ruoti, Ben Kaiser, Arkady Yerukhimovich, Jeremy Clark, and Robert Cunningham

Industry's dreams and fears for this new technology

Business executives, government leaders, investors, and researchers frequently ask the following three questions: (1) What exactly is blockchain technology? (2) What capabilities does it provide? (3) What are good applications?

The goal of this article is to answer these questions thoroughly, provide a holistic overview of blockchain technology that separates hype from reality, and propose a useful lexicon for discussing the specifics of blockchain technology in the future.


Everything Sysadmin:
API Practices If You Hate Your Customers

  Thomas A. Limoncelli

APIs speak louder than words.

Do you have disdain for your customers? Do you wish they would go away? When you interact with customers are you silently fantasizing about them switching to your competitor's product? In short, do you hate your customers? In this article, I document a number of industry best practices designed to show customers how much you hate them. All of them are easy to implement. Heck, your company may be doing many of these already.

Business and Management, Everything Sysadmin

The Reliability of Enterprise Applications

  Sanjay Sha

Understanding enterprise reliability

This article describes a core set of principles and engineering methodologies that enterprises can apply to help them navigate the complex environment of enterprise reliability and deliver highly reliable and cost-efficient applications.

Quality Assurance

Escaping the Singularity
Space Time Discontinuum

  Pat Helland

Combining data from many sources may cause painful delays.

Back when you had only one database for an application to worry about, you didn't have to think about partial results. You also didn't have to think about data arriving after some other data. It was all simply there. Now, you can do so much more with big distributed systems, but you have to be more sophisticated in the tradeoff between timely answers and complete answers.

Data and Databases, Escaping the Singularity

Optimizations in C++ Compilers

  Matt Godbolt

A practical journey

There's a tradeoff to be made in giving the compiler more information: it can make compilation slower. Technologies such as link time optimization can give you the best of both worlds. Optimizations in compilers continue to improve, and upcoming improvements in indirect calls and virtual function dispatch might soon lead to even faster polymorphism.


The Morning Paper:
Back under a SQL Umbrella

  Adrian Colyer

Unifying serving and analytical data; using a database for distributed machine learning

Procella is the latest in a long line of data processing systems at Google. What's unique about it is that it's a single store handling reporting, embedded statistics, time series, and ad-hoc analysis workloads under one roof. It's SQL on top, cloud-native underneath, and it's serving billions of queries per day over tens of petabytes of data. There's one big data use case that Procella isn't handling today though, and that's machine learning. But in 'Declarative recursive computation on an RDBMS... or, why you should use a database for distributed machine learning,' Jankov et al. make the case for the database being the ideal place to handle the most demanding of distributed machine learning workloads.

Data and Databases, The Morning Paper



July/August 2019

The Morning Paper:
Putting Machine Learning into Production Systems

  Adrian Colyer

Data validation and software engineering for machine learning

In "Data Validation for Machine Learning," Breck et al. share details of the pipelines used at Google to validate petabytes of production data every day. With so many moving parts it's important to be able to detect and investigate changes in data distributions before they can impact model performance.

"Software Engineering for Machine Learning: A Case Study" shares lessons learned at Microsoft as machine learning started to pervade more and more of the company's systems, moving from specialized machine-learning products to simply being an integral part of many products and services.

AI, The Morning Paper

Hack for Hire

  Ariana Mirian

Investigating the emerging black market of retail email account hacking services

While targeted attacks are often thought of as requiring nation-state capabilities, there is an emerging black market for "hack-for-hire" services, which provide targeted attacks to anyone willing to pay a modest fee. These services purport to be able to break into the accounts of a variety of different email providers. As these services are just emerging, little is known about how they attack their victims and how much of a risk they pose.

Regardless of the behavior of the market, this study sheds light on the importance of security keys for populations who believe they are at risk, as only a security key can protect a user from the attacks viewed in this study. As the market evolves and defenses change, however, attacks might also change and shift from phishing to more persistent threats such as malware.

Privacy and Rights, Security

Escaping the Singularity
Write Amplification Versus Read Perspiration

  Pat Helland

The tradeoffs between write and read

In computing, there's an interesting trend where writing creates a need to do more work. You need to reorganize, merge, reindex, and more to make the stuff you wrote more useful. If you don't, you must search or do other work to support future reads.

Data and Databases, Escaping the Singularity

The Effects of Mixing Machine Learning and Human Judgment

  Michelle Vaccaro and Jim Waldo

Collaboration between humans and machines does not necessarily lead to better outcomes.

Based on the theoretical findings from the existing literature, some policymakers and software engineers contend that algorithmic risk assessments such as the COMPAS software can alleviate the incarceration epidemic and the occurrence of violent crimes by informing and improving decisions about policing, treatment, and sentencing.

Considered in tandem, these findings indicate that collaboration between humans and machines does not necessarily lead to better outcomes, and human supervision does not sufficiently address problems when algorithms err or demonstrate concerning biases. If machines are to improve outcomes in the criminal justice system and beyond, future research must further investigate their practical role: an input to human decision makers.

Artificial Intelligence

Kode Vicious
Koding Academies

A low-risk path to becoming a front-end plumber

Encourage your friend to pick a course that will introduce concepts that can be used into the future, rather than just a specific set of buzzword technologies that are hot this year. Most courses are based around Python. Encourage your friend to study that as a first computer language, as the concepts learned in Python can be applied in other languages and other fields. And make sure to be very direct in explaining that the certificate effectively makes its holder a front-end plumber, able to unclog the series of pipes that run between businesses and consumers' wallets, and that becoming a software engineer will take quite a bit more study and practice.

Education, Kode Vicious

Persistent Memory Programming on Conventional Hardware

  Terence Kelly

The persistent memory style of programming can dramatically simplify application software.

Driven by the advent of byte-addressable non-volatile memory, the persistent memory style of programming will gain traction among developers, taking its rightful place alongside existing paradigms for managing persistent application state. Until NVM becomes available on all computers, developers can use the techniques presented in this article to enjoy the benefits of persistent memory programming on conventional hardware.




May/June 2019

Case Study
DAML: The Contract Language of Distributed Ledgers

A discussion between Shaul Kfir and Camille Fournier

The how and why of Digital Asset's own distributed-ledger technology, DAML (Digital Asset Modeling Language).

Case Studies, Networks

Kode Vicious
What is a CSO Good For?

Security requires more than an off-the-shelf solution.

The CSO is not a security engineer, so let's contrast the two jobs to create a picture of what we should and should not see.

Business and Management, Kode Vicious, Security

Everything Sysadmin:
Demo Data as Code

  Thomas A. Limoncelli

Automation helps collaboration.

A casual request for a demo dataset may seem like a one-time thing that doesn't need to be automated, but the reality is that this is a collaborative process requiring multiple iterations and experimentation. There will undoubtedly be requests for revisions big and small, the need to match changing software, and to support new and revised demo stories. All of this makes automating the process worthwhile. Modern scripting languages make it easy to create ad hoc functions that act like a little language. A repeatable process helps collaboration, enables delegation, and saves time now and in the future.

Business and Management, Everything Sysadmin

Velocity in Software Engineering

  Tom Killalea

From tectonic plate to F-16

Software engineering occupies an increasingly critical role in companies across all sectors, but too many software initiatives end up both off target and over budget. A surer path is optimized for speed, open to experimentation and learning, agile, and subject to regular course correcting. Good ideas tend to be abundant, though execution at high velocity is elusive. The good news is that velocity is controllable; companies can invest systematically to increase it.


The Soft Side of Software:
The Evolution of Management

  Kate Matsudaira

Transitioning up the ladder

With each step up, the job changes - but not all of the changes are obvious. You have to shift your mindset, and focus on building new skills that are often very different from the skills that made you successful in your previous role.

Business and Management, The Soft Side of Software

Open-source Firmware

  Jessie Frazelle

Step into the world behind the kernel.

Open-source firmware can help bring computing to a more secure place by making the actions of firmware more visible and less likely to do harm. This article's goal is to make readers feel empowered to demand more from vendors who can help drive this change.

Open Source

The Morning Paper:
Time Protection in Operating Systems;
Speaker Legitimacy Detection

  Adrian Colyer

Operating system-based protection from timing-based side-channel attacks;
implications of voice-imitation software

Timing-based side-channel attacks are a particularly tricky class of attacks to deal with because the very thing you're often striving for can give you away. There are always more creative new instances of attacks to be found, so you need a principled way of thinking about defenses that address the class, not just a particular instantiation. That's what Ge et al. give us in "Time Protection, the Missing OS Abstraction." Just as operating systems prevent spatial inference through memory protection, so future operating systems will need to prevent temporal inference through time protection. It's going to be a long road to get there.

The second paper chosen for this edition comes from NDSS'19 (Network and Distributed System Security Symposium) and studies the physiological and social implications of the ever-improving abilities of voice-imitation software. It seems people may be especially vulnerable to being fooled by fake voices. "The crux of voice (in)security: a brain study of speaker legitimacy detection," by Neupane et al., is a fascinating study with implications far beyond just the technology.

Networks, Security


Older Issues