Departments

A "Perspectival" Mirror of the Elephant:
Investigating language bias on Google, ChatGPT, YouTube, and Wikipedia

Many people turn to Internet-based, software platforms such as Google, YouTube, Wikipedia, and more recently ChatGPT to find the answers to their questions. Most people tend to trust Google Search when it states that its mission is to deliver information from "many angles so you can form your own understanding of the world." Yet, our work finds that queries involving complex topics yield results focused on a narrow set of culturally dominant views, and these views are correlated with the language used in the search phrase. We call this phenomenon language bias, and this article shows how it occurs using the example of two complex topics: Buddhism and liberalism.

Software Drift:
Open source forking

Since the systems have a common parent, they probably work in the same technical domain, and therefore the features and fixes that are going to be added are probably similar. KV happens to have an example case at hand: two operating systems that diverged before they added SMP (symmetric multiprocessing) support. When an operating system adds SMP to an existing kernel, the first thing we think of is locks, those handy-dandy little performance killers that we've all been sprinkling around our code since the end of Dennard scaling.

Challenges in Adopting and Sustaining Microservice-based Software Development:
Organizational challenges can be more difficult than technical ones.

MS (microservice) has become the latest buzzword in software development. The MS approach to software development offers an alternative to the conventional monolith style. While benefits of MS-based development over monolith style are clear, industry experts agree that neither style provides an absolute advantage in all situations. Proponents contend that an MS approach to software development more readily facilitates mapping organizational changes manifesting from a more dynamic business environment to corresponding IT/IS (information technology/information systems) changes. This article identifies key challenges from the initial decision to adopt MSs to the ongoing task of sustaining the new paradigm over the long haul.

Free and Open Source Software - and Other Market Failures:
Open source is not a goal as much as a means to an end.

Open source was not so much the goal itself as a means to an end, which is freedom: freedom to fix broken things, freedom from people who thought they could clutch the source code tightly and wield our ignorance of it as a weapon to force us all to pay for and run Windows Vista. But the FOSS movement has won what it wanted, and no matter how much oldsters dream about their glorious days as young revolutionaries, it is not coming back, because the frustrations and anger of IT in 2024 are entirely different from those of 1991.

Give Your Project a Name:
It goes a long way toward creating a cohesive team with strong morale.

While some people are driven by infinite backlogs and iteration, others prefer launches and deadlines. Over the years, I have found certain milestones to be instrumental in creating a cohesive team with strong morale. When people have to work together to get through a challenging task, reaching those milestones brings them together.

From Open Access to Guarded Trust:
Experimenting responsibly in the age of data privacy

The last decade witnessed the emergence and strengthening of data protection regulations. For software engineers, this new era poses a unique challenge: How do you maintain the precision and efficacy of your platforms when complete data access, one of your most potent tools, is gradually being taken off the table? The mission is clear: Reinvent the toolkit. The way we perceive, handle, and experiment with data needs a drastic overhaul to navigate this brave new world.

Developer Ecosystems for Software Safety:
Continuous assurance at scale

How to design and implement information systems so that they are safe and secure is a complex topic. Both high-level design principles and implementation guidance for software safety and security are well established and broadly accepted. For example, Jerome Saltzer and Michael Schroeder's seminal overview of principles of secure design was published almost 50 years ago, and various community and governmental bodies have published comprehensive best practices about how to avoid common software weaknesses.

Programmer Job Interviews: The Hidden Agenda

Top tech interviews test coding and CS knowledge overtly, but they also evaluate a deeper technical instinct so subtly that candidates seldom notice the appraisal. We'll learn how interviewers create questions to covertly measure a skill that sets the best programmers above the rest. Equipped with empathy for the interviewer, you can prepare to shine on the job market by seizing camouflaged opportunities.

DevEx in Action:
A study of its tangible impacts

DevEx (developer experience) is garnering increased attention at many software organizations as leaders seek to optimize software delivery amid the backdrop of fiscal tightening and transformational technologies such as AI. Intuitively, there is acceptance among technical leaders that good developer experience enables more effective software delivery and developer happiness. Yet, at many organizations, proposed initiatives and investments to improve DevEx struggle to get buy-in as business stakeholders question the value proposition of improvements.

Resolving the Human-subjects Status of Machine Learning's Crowdworkers:
What ethical framework should govern the interaction of ML researchers and crowdworkers?

In recent years, machine learning (ML) has relied heavily on crowdworkers both for building datasets and for addressing research questions requiring human interaction or judgment. The diversity of both the tasks performed and the uses of the resulting data render it difficult to determine when crowdworkers are best thought of as workers versus human subjects. These difficulties are compounded by conflicting policies, with some institutions and researchers regarding all ML crowdworkers as human subjects and others holding that they rarely constitute human subjects. Notably few ML papers involving crowdwork mention IRB oversight, raising the prospect of non-compliance with ethical and regulatory requirements.

Is There Another System?:
Computer science is the study of what can be automated.

One of the easiest tests to determine if you are at risk is to look hard at what you do every day and see if you, yourself, could code yourself out of a job. Programming involves a lot of rote work: templating, boilerplate, and the like. If you can see a way to write a system to replace yourself, either do it, don't tell your bosses, and collect your salary while reading novels in your cubicle, or look for something more challenging to work on.

Automatically Testing Database Systems:
DBMS testing with test oracles, transaction history, and fuzzing

The automated testing of DBMS is an exciting, interdisciplinary effort that has seen many innovations in recent years. The examples addressed here represent different perspectives on this topic, reflecting strands of research from software engineering, (database) systems, and security angles. They give only a glimpse into these research strands, as many additional interesting and effective works have been proposed. Various approaches generate pairs of related tests to find both logic bugs and performance issues in a DBMS. Similarly, other isolation-level testing approaches have been proposed.

How to Design an ISA:
The popularity of RISC-V has led many to try designing instruction sets.

Over the past decade I've been involved in several projects that have designed either ISA (instruction set architecture) extensions or clean-slate ISAs for various kinds of processors (you'll even find my name in the acknowledgments for the RISC-V spec, right back to the first public version). When I started, I had very little idea about what makes a good ISA, and, as far as I can tell, this isn't formally taught anywhere.

What do Trains, Horses, and Home Internet Installation have in Common?:
Avoid changes mid-process

At first, I thought he was just trying to shirk his responsibilities and pass the buck on to someone else. His advice, however, made a lot of sense. The installation team probably generated configurations ahead of time, planned out how and when those changes need to be activated, and so on. The entire day is planned ahead. Bureaucracies usually have a happy path that works well, and any deviation requires who knows what? Managers getting involved? Error-prone manual steps? Ad hoc database queries? There's no way I could know.

Multiparty Computation: To Secure Privacy, Do the Math:
A discussion with Nigel Smart, Joshua W. Baron, Sanjay Saravanan, Jordan Brandt, and Atefeh Mashatan

Multiparty Computation is based on complex math, and over the past decade, MPC has been harnessed as one of the most powerful tools available for the protection of sensitive data. MPC now serves as the basis for protocols that let a set of parties interact and compute on a pool of private inputs without revealing any of the data contained within those inputs. In the end, only the results are revealed. The implications of this can often prove profound.

The Security Jawbreaker:
Access to a system should not imply authority to use it. Enter the principle of complete mediation.

When someone stands at the front door of your home, what are the steps to let them in? If it is a member of the family, they use their house key, unlocking the door using the authority the key confers. For others, a knock at the door or doorbell ring prompts you to make a decision. Once in your home, different individuals have differing authority based on who they are. Family members have access to your whole home. A close friend can roam around unsupervised, with a high level of trust. An appliance repair person is someone you might supervise for the duration of the job to be done.

Improving Testing of Deep-learning Systems:
A combination of differential and mutation testing results in better test data.

We used differential testing to generate test data to improve diversity of data points in the test dataset and then used mutation testing to check the quality of the test data in terms of diversity. Combining differential and mutation testing in this fashion improves mutation score, a test data quality metric, indicating overall improvement in testing effectiveness and quality of the test data when testing deep learning systems.

Dear Diary:
On keeping a laboratory notebook

While a debug log is helpful, it's not the same thing as a laboratory notebook. If more computer scientists acted like scientists, we wouldn't have to fight over whether computing is an art or a science.

Low-code Development Productivity:
"Is winter coming" for code-based technologies?

This article aims to provide new insights on the subject by presenting the results of laboratory experiments carried out with code-based, low-code, and extreme low-code technologies to study differences in productivity. Low-code technologies have clearly shown higher levels of productivity, providing strong arguments for low-code to dominate the software development mainstream in the short/medium term. The article reports the procedure and protocols, results, limitations, and opportunities for future research.

Software Managers' Guide to Operational Excellence:
The secret to being a great engineering leader? Setting up the right checks and balances.

Software engineering managers (or any senior technical leaders) have many responsibilities: the care and feeding of the team, delivering on business outcomes, and keeping the product/system/application up and running and in good order. Each of these areas can benefit from a systematic approach. The one I present here is setting up checks and balances for the team's operational excellence.

Use Cases are Essential:
Use cases provide a proven method to capture and explain the requirements of a system in a concise and easily understood format.

While the software industry is a fast-paced and exciting world in which new tools, technologies, and techniques are constantly being developed to serve business and society, it is also forgetful. In its haste for fast-forward motion, it is subject to the whims of fashion and can forget or ignore proven solutions to some of the eternal problems that it faces. Use cases, first introduced in 1986 and popularized later, are one of those proven solutions.

Device Onboarding using FDO and the Untrusted Installer Model:
FDO's untrusted model is contrasted with Wi-Fi Easy Connect to illustrate the advantages of each mechanism.

Automatic onboarding of devices is an important technique to handle the increasing number of "edge" and IoT devices being installed. Onboarding of devices is different from most device-management functions because the device's trust transitions from the factory and supply chain to the target application. To speed the process with automatic onboarding, the trust relationship in the supply chain must be formalized in the device to allow the transition to be automated.

Knowing What You Need to Know:
Personal, team, and organizational effectiveness can be improved with a little preparation

Blockers can take a tiny task and stretch it over days or weeks. Taking a moment at the beginning of a project to look for and prevent possible blockers can improve productivity. These examples of personal, team, and organizational levels show how gathering the right information and performing preflight checks can save hours of wasted time later.

Halfway Around the World:
Learn the language, meet the people, eat the food

Not only do different cultures treat different features differently, but they also treat each other differently. How people act with respect to each other is a topic that can, and does, fill volumes of books that, as nerds, we probably have never read, but finding out a bit about where you're heading is a good idea. You can try to ask the locals, although people generally are so enmeshed in their own cultures that they have a hard time explaining them to others.

Protecting Secrets from Computers

Bob is in prison and Alice is dead; they trusted computers with secrets. Review time-tested tricks that can help you avoid the grim fate of the old crypto couple.

Creating the First Confidential GPUs:
The team at NVIDIA brings confidentiality and integrity to user code and data for accelerated computing.

Today's datacenter GPU has a long and storied 3D graphics heritage. In the 1990s, graphics chips for PCs and consoles had fixed pipelines for geometry, rasterization, and pixels using integer and fixed-point arithmetic. In 1999, NVIDIA invented the modern GPU, which put a set of programmable cores at the heart of the chip, enabling rich 3D scene generation with great efficiency.

Why Should I Trust Your Code?:
Confidential computing enables users to authenticate code running in TEEs, but users also need evidence this code is trustworthy.

For Confidential Computing to become ubiquitous in the cloud, in the same way that HTTPS became the default for networking, a different, more flexible approach is needed. Although there is no guarantee that every malicious code behavior will be caught upfront, precise auditability can be guaranteed: Anyone who suspects that trust has been broken by a confidential service should be able to audit any part of its attested code base, including all updates, dependencies, policies, and tools. To achieve this, we propose an architecture to track code provenance and to hold code providers accountable. At its core, a new Code Transparency Service (CTS) maintains a public, append-only ledger that records all code deployed for confidential services.

Hardware VM Isolation in the Cloud:
Enabling confidential computing with AMD SEV-SNP technology

Confidential computing is a security model that fits well with the public cloud. It enables customers to rent VMs while enjoying hardware-based isolation that ensures that a cloud provider cannot purposefully or accidentally see or corrupt their data. SEV-SNP was the first commercially available x86 technology to offer VM isolation for the cloud and is deployed in Microsoft Azure, AWS, and Google Cloud. As confidential computing technologies such as SEV-SNP develop, confidential computing is likely to simply become the default trust model for the cloud.

Confidential Computing: Elevating Cloud Security and Privacy:
Working toward a more secure and innovative future

Confidential Computing (CC) fundamentally improves our security posture by drastically reducing the attack surface of systems. While traditional systems encrypt data at rest and in transit, CC extends this protection to data in use. It provides a novel, clearly defined security boundary, isolating sensitive data within trusted execution environments during computation. This means services can be designed that segment data based on least-privilege access principles, while all other code in the system sees only encrypted data. Crucially, the isolation is rooted in novel hardware primitives, effectively rendering even the cloud-hosting infrastructure and its administrators incapable of accessing the data.

Stone Knives and Bear Skins

If you look at the software tooling landscape, you see that the majority of developers work with either open-source tools; or tools from the recently reformed home of proprietary software, Microsoft, which has figured out that its Visual Studio Code system is a good way to sucker people into working with its platforms; or finally Apple, whose tools are meant only for its platform. In specialized markets, such as deeply embedded, military, and aerospace, there are proprietary tools that are often far worse than their open-source cousins, because the market for such tools is small but lucrative.

Pointers in Far Memory:
A rethink of how data and computations should be organized

Effectively exploiting emerging far-memory technology requires consideration of operating on richly connected data outside the context of the parent process. Operating-system technology in development offers help by exposing abstractions such as memory objects and globally invariant pointers that can be traversed by devices and newly instantiated compute. Such ideas will allow applications running on future heterogeneous distributed systems with disaggregated memory nodes to exploit near-memory processing for higher performance and to independently scale their memory and compute resources for lower cost.

Don't "Think of the Internet!":
No human right is absolute.

I cannot help but notice few women subscribe to absolutist views of electronic privacy and anonymity. Can it be that only people who play life on the easiest setting find unrestricted privacy and anonymity a great idea?

How Flexible is CXL's Memory Protection?:
Replacing a sledgehammer with a scalpel

CXL, a new interconnect standard for cache-coherent memory sharing, is becoming a reality - but its security leaves something to be desired. Decentralized capabilities are flexible and resilient against malicious actors, and should be considered while CXL is under active development.

Security Mismatch:
Security must be a business enabler, not a hinderer.

Information security teams that say 'no' need to change. Hiding behind a moat makes repelling attacks easy, but bridges allow you to replenish supplies and foster relationships with customers? castles. Remember, a security team's role is to empower their business to move forward with confidence, not to hinder progress.

Managing Hybrid Teams:
The combination of on-site and remote workers takes extra effort from team leaders.

After three years of working remotely, many companies are asking their people to return to the office. Not everyone is coming back, however. With some people in the office and some still working from home, leaders must get this transition to hybrid work right. Hybrid is the worst of both worlds in some ways. You can easily end up creating two experiences?one for the people in the office and one for the remote workers?which can lead to problems that will compound over time and have long-term damaging effects on your team.

Echoes of Intelligence:
Textual interpretation and large language models

We are now in the presence of a new medium disguised as good old text, but that text has been generated by an LLM, without authorial intention—an aspect that, if known beforehand, completely changes the expectations and response a human should have from a piece of text. Should our interpretation capabilities be engaged? If yes, under what conditions? The rules of the language game should be spelled out; they should not be passed over in silence.

You Don't know Jack about Application Performance:
Knowing whether you're doomed to fail is important when starting a project.

You don't need to do a full-scale benchmark any time you have a performance or capacity planning problem. A simple measurement will provide the bottleneck point of your system: This example program will get significantly slower after eight requests per second per CPU. That's often enough to tell you the most important thing: if you're going to fail.

The Human Touch:
There is no substitute for good, direct, honest training.

The challenge of providing a safe communications environment in the face of such social engineering attacks isn't just the technology; it's also people. As anyone who has done serious work in computer security knows, the biggest problems are between the keyboard and the chair. Most people by default trust other people and are willing to give them the benefit of the doubt.

OS Scheduling:
Better scheduling policies for modern computing systems

In any system that multiplexes resources, the problem of scheduling what computations run where and when is perhaps the most fundamental. Yet, like many other essential problems in computing (e.g., query optimization in databases), academic research in scheduling moves like a pendulum, with periods of intense activity followed by periods of dormancy when it is considered a "solved" problem. These three papers make significant contributions to an ongoing effort to develop better scheduling policies for modern computing systems.

Cargo Cult AI:
Is the ability to think scientifically the defining essence of intelligence?

Evidence abounds that the human brain does not innately think scientifically; however, it can be taught to do so. The same species that forms cargo cults around widespread and unfounded beliefs in UFOs, ESP, and anything read on social media also produces scientific luminaries such as Sagan and Feynman. Today's cutting-edge LLMs are also not innately scientific. But unlike the human brain, there is good reason to believe they never will be unless new algorithmic paradigms are developed.

Beyond the Repository:
Best practices for open source ecosystems researchers

Much of the existing research about open source elects to study software repositories instead of ecosystems. An open source repository most often refers to the artifacts recorded in a version control system and occasionally includes interactions around the repository itself. An open source ecosystem refers to a collection of repositories, the community, their interactions, incentives, behavioral norms, and culture. The decentralized nature of open source makes holistic analysis of the ecosystem an arduous task, with communities and identities intersecting in organic and evolving ways. Despite these complexities, the increased scrutiny on software security and supply chains makes it of the utmost importance to take an ecosystem-based approach when performing research about open source.

DevEx: What Actually Drives Productivity:
The developer-centric approach to measuring and improving productivity

Developer experience focuses on the lived experience of developers and the points of friction they encounter in their everyday work. In addition to improving productivity, DevEx drives business performance through increased efficiency, product quality, and employee retention. This paper provides a practical framework for understanding DevEx, and presents a measurement framework that combines feedback from developers with data about the engineering systems they interact with. These two frameworks provide leaders with clear, actionable insights into what to measure and where to focus in order to improve developer productivity.

Designing a Framework for Conversational Interfaces:
Combining the latest advances in machine learning with earlier approaches

Wherever possible, business logic should be described by code rather than training data. This keeps our system's behavior principled, predictable, and easy to change. Our approach to conversational interfaces allows them to be built much like any other application, using familiar tools, conventions, and processes, while still taking advantage of cutting-edge machine-learning techniques.

The Parchment Path?:
Is there ever a time when learning is not of value - for its own sake?

The greater the risk, the greater the reward, and if you do succeed, it will be an achievement that you can look back on and smile wryly about. Postdocs never laugh because postdocs are post-laughter. However, there are some things to consider before plunking down your application fee and writing all those essays.

Opportunity Cost and Missed Chances in Optimizing Cybersecurity:
The loss of potential gain from other alternatives when one alternative is chosen

Opportunity cost should not be an afterthought when making security decisions. One way to ease into considering complex alternatives is to consider the null baseline of doing nothing instead of the choice at hand. Opportunity cost can feel abstract, elusive, and imprecise, but it can be understood by everyone, given the right introduction and framing. Using the approach presented here will make it natural and accessible.

Improvement on End-to-End Encryption May Lead to Silent Revolution:
Researchers are on a brink of what could be the next big improvement in communication privacy.

Privacy is an increasing concern, whether you are texting with a business associate or transmitting volumes of data over the Internet. Over the past few decades, cryptographic techniques have enabled privacy improvements in chat apps and other electronic forms of communication. Now researchers are on the brink of what could be the next big improvement in communication privacy: E2EEEE (End-to-End Encryption with Endpoint Elimination). This article is based on interviews with researchers who plan on presenting at a symposium on the topic scheduled for April 1, 2023.

Catch-23: The New C Standard Sets the World on Fire

A new major revision of the C programming language standard is nearly upon us. C23 introduces pleasant conveniences, retains venerable traps for the unwary, and innovates a gratuitous catastrophe. A few steps forward, much sideways shuffling, and a drunken backward stumble into the fireplace come together in the official dance of C standardization, the Whiskey Tango Foxtrot.

Sharpening Your Tools:
Updating bulk_extractor for the 2020s

This article presents our experience updating the high-performance Digital forensics tool BE (bulk_extractor) a decade after its initial release. Between 2018 and 2022, we updated the program from C++98 to C++17. We also performed a complete code refactoring and adopted a unit test framework. DF tools must be frequently updated to keep up with changes in the ways they are used. A description of updates to the bulk_extractor tool serves as an example of what can and should be done.

More Than Just Algorithms:
A discussion with Alfred Spector, Peter Norvig, Chris Wiggins, Jeannette Wing, Ben Fried, and Michael Tingley

Dramatic advances in the ability to gather, store, and process data have led to the rapid growth of data science and its mushrooming impact on nearly all aspects of the economy and society. Data science has also had a huge effect on academic disciplines with new research agendas, new degrees, and organizational entities. The authors of a new textbook, Data Science in Context: Foundations, Challenges, Opportunities, share their ideas about the impact of the field on nearly all aspects of the economy and society.

All Sliders to the Right:
Hardware Overkill

There are many reasons why this year's model isn't any better than last year's, and many reasons why performance fails to scale, some of which KV has covered in these pages. It is true that the days of upgrading every year and getting a free performance boost are long gone, as we're not really getting single cores that are faster than about 4GHz. One thing that many software developers fail to understand is the hardware on which their software runs at a sufficiently deep level.

Three-part Harmony for Program Managers Who Just Don't Get It, Yet:
Open-source software, open standards, and agile software development

This article examines three tools in the system acquisitions toolbox that can work to expedite development and procurement while mitigating programmatic risk: OSS, open standards, and the Agile/Scrum software development processes are all powerful additions to the DoD acquisition program management toolbox.

The Fun in Fuzzing:
The debugging technique comes into its own.

Stefan Nagy, an assistant professor in the Kahlert School of Computing at the University of Utah, takes us on a tour of recent research in software fuzzing, or the systematic testing of programs via the generation of novel or unexpected inputs. The first paper he discusses extends the state of the art in coverage-guided fuzzing with the semantic notion of "likely invariants," inferred via techniques from property-based testing. The second explores encoding domain-specific knowledge about certain bug classes into test-case generation.

To PiM or Not to PiM:
The case for in-memory inferencing of quantized CNNs at the edge

As artificial intelligence becomes a pervasive tool for the billions of IoT (Internet of things) devices at the edge, the data movement bottleneck imposes severe limitations on the performance and autonomy of these systems. PiM (processing-in-memory) is emerging as a way of mitigating the data movement bottleneck while satisfying the stringent performance, energy efficiency, and accuracy requirements of edge imaging applications that rely on CNNs (convolutional neural networks).

Taking Flight with Copilot:
Early insights and opportunities of AI-powered pair-programming tools

Over the next five years, AI-powered tools likely will be helping developers in many diverse tasks. For example, such models may be used to improve code review, directing reviewers to parts of a change where review is most needed or even directly providing feedback on changes. Models such as Codex may suggest fixes for defects in code, build failures, or failing tests. These models are able to write tests automatically, helping to improve code quality and downstream reliability of distributed systems. This study of Copilot shows that developers spend more time reviewing code than actually writing code.

Reinventing Backend Subsetting at Google:
Designing an algorithm with reduced connection churn that could replace deterministic subsetting

Backend subsetting is useful for reducing costs and may even be necessary for operating within the system limits. For more than a decade, Google used deterministic subsetting as its default backend subsetting algorithm, but although this algorithm balances the number of connections per backend task, deterministic subsetting has a high level of connection churn. Our goal at Google was to design an algorithm with reduced connection churn that could replace deterministic subsetting as the default backend subsetting algorithm.

The Elephant in the Room:
It's time to get the POSIX elephant off our necks.

By writing code for the elephant that is Posix, we lose the chance to take advantage of modern hardware.

OCCAM-v2: Combining Static and Dynamic Analysis for Effective and Efficient Whole-program Specialization:
Leveraging scalable pointer analysis, value analysis, and dynamic analysis

OCCAM-v2 leverages scalable pointer analysis, value analysis, and dynamic analysis to create an effective and efficient tool for specializing LLVM bitcode. The extent of the code-size reduction achieved depends on the specific deployment configuration. Each application that is to be specialized is accompanied by a manifest that specifies concrete arguments that are known a priori, as well as a count of residual arguments that will be provided at runtime. The best case for partial evaluation occurs when the arguments are completely concretely specified. OCCAM-v2 uses a pointer analysis to devirtualize calls, allowing it to eliminate the entire body of functions that are not reachable by any direct calls.

OSS Supply-chain Security: What Will It Take?

While enterprise security teams naturally tend to turn their focus primarily to direct attacks on their own infrastructure, cybercrime exploits now are increasingly aimed at easier targets upstream. This has led to a perfect storm, since virtually all significant codebase repositories at this point include at least some amount of open-source software. But opportunities also abound there for the authors of malware. The broader cybercrime world, meanwhile, has noted that open-source supply chains are generally easy to penetrate. What's being done at this point to address the apparent risks?

Literate Executables

Literate executables redefine the relationship between compiled binaries and source code to be that of chicken and egg, so it's easy to derive either from the other. This episode of Drill Bits provides a general-purpose literacy tool and showcases the advantages of literacy by retrofitting it onto everyone's favorite command-line utility.

Split Your Overwhelmed Teams:
Two Teams of Five is Not the Same as One Team of Ten

This team?s low morale and high stress were a result of the members feeling overwhelmed by too many responsibilities. The 10-by-10 communication structure made it difficult to achieve consensus, there were too many meetings, and everyone was suffering from the high cognitive load. By splitting into two teams, each can be more nimble, which the manager likes, and have a lower cognitive load, which the team likes. There is more opportunity for repetition, which lets people develop skills and demonstrate them. Altogether, this helps reduce stress and improve morale.

The Rise of Fully Homomorphic Encryption:
Often called the Holy Grail of cryptography, commercial FHE is near.

Once commercial FHE is achieved, data access will become completely separated from unrestricted data processing, and provably secure storage and computation on untrusted platforms will become both relatively inexpensive and widely accessible. In ways similar to the impact of the database, cloud computing, PKE, and AI, FHE will invoke a sea change in how confidential information is protected, processed, and shared, and will fundamentally change the course of computing at a foundational level.

Mapping the Privacy Landscape for Central Bank Digital Currencies:
Now is the time to shape what future payment flows will reveal about you.

As central banks all over the world move to digitize cash, the issue of privacy needs to move to the forefront. The path taken may depend on the needs of each stakeholder group: privacy-conscious users, data holders, and law enforcement.

From Zero to One Hundred:
Demystifying zero trust and its implications on enterprise people, process, and technology

Changing network landscapes and rising security threats have imparted a sense of urgency for new approaches to security. Zero trust has been proposed as a solution to these problems, but some regard it as a marketing tool to sell existing best practice while others praise it as a new cybersecurity standard. This article discusses the history and development of zero trust and why the changing threat landscape has led to a new discourse in cybersecurity.

The Arrival of Zero Trust: What Does it Mean?

It used to be that enterprise cybersecurity was all castle and moat. First, secure the perimeter and then, in terms of what went on inside that, Trust, but verify. The perimeter, of course, was the corporate network. But what does that even mean at this point? With most employees now working from home at least some of the time and organizations relying increasingly on cloud computing, there is no such thing as a single, enterprise-wide perimeter anymore. And, with corporate security breaches having become a regular news item over the past two decades, trust has essentially evaporated as well.

Crash Consistency:
Keeping data safe in the presence of crashes is a fundamental problem.

Keeping data safe in the presence of crashes is a fundamental problem in storage systems. Although the high-level ideas for crash consistency are relatively well understood, realizing them in practice is surprisingly complex and full of challenges. The systems research community is actively working on solving this challenge, and the papers examined here offer three solutions.

The Four Horsemen of an Ailing Software Project:
Don't let the pale rider catch you with an exception.

KV has talked about various measures of software quality in past columns, but perhaps falling software quality is one of the most objective measures that a team is failing. This Pestilence, brought about by the low morale engendered in the team by War and Famine, is a clear sign that something is wrong. In the real world, a diseased animal can be culled so that disease does not spread and become a pestilence over the land. Increasing bug counts, especially in the absence of increased functionality, is a sure sign of a coming project apocalypse.

CSRB's Opus One:
Comments on the Cyber Safety Review Board Log4j Event Report

We in FOSS need to become much better at documenting design decisions in a way and a place where the right people will find it, read it, and understand it, before they do something ill-advised or downright stupid with our code.

Privacy of Personal Information:
Going incog in a goldfish bowl

Each online interaction with an external service creates data about the user that is digitally recorded and stored. These external services may be credit card transactions, medical consultations, census data collection, voter registration, etc. Although the data is ostensibly collected to provide citizens with better services, the privacy of the individual is inevitably put at risk. With the growing reach of the Internet and the volume of data being generated, data protection and, specifically, preserving the privacy of individuals, have become particularly important.

Securing the Company Jewels:
GitHub and runbook security

Often the problem with a runbook isn't the runbook itself, it's the runner of the runbook that matters. A runbook, or a checklist, is supposed to be an aid to memory and not a replacement for careful and independent thought. But our industry being what it is, we now see people take these things to their illogical extremes, and I think this is the problem you are running into with your local runbook runner.

I'm Probably Less Deterministic Than I Used to Be:
Embracing randomness is necessary in cloud environments.

In my youth, I thought the universe was ruled by cause and effect like a big clock. In this light, computing made sense. Now I see that both life and computing can be a crapshoot, and that has given me a new peace.

The Challenges of IoT, TLS, and Random Number Generators in the Real World:
Bad random numbers are still with us and are proliferating in modern systems.

Many in the cryptographic community scoff at the mistakes made in implementing RNGs. Many cryptographers and members of the IETF resist the call to make TLS more resilient to this class of failures. This article discusses the history, current state, and fragility of the TLS protocol, and it closes with an example of how to improve the protocol. The goal is not to suggest a solution but to start a dialog to make TLS more resilient by proving that the security of TLS without the assumption of perfect random numbers is possible.

Convergence:
Research for Practice reboot

It is with great pride and no small amount of excitement that I announce the reboot of acmqueue's Research for Practice column. For three years, beginning at its inception in 2016, Research for Practice brought both seminal and cutting-edge research - via careful curation by experts in academia - within easy reach for practitioners who are too busy building things to manage the deluge of scholarly publications. We believe the series succeeded in its stated goal of sharing "the joy and utility of reading computer science research" between academics and their counterparts in industry. We know our readers have missed it, and we are delighted to rekindle the flame after a three-year hiatus.

I Unplugged What?

Linear Address Spaces:
Unsafe at any speed

The linear address space as a concept is unsafe at any speed, and it badly needs mandatory CHERI seat belts. But even better would be to get rid of linear address spaces entirely and go back to the future, as successfully implemented in the Rational R1000 computer 30-plus years ago.

When Should a Black Box Be Transparent?:
When is a replacement not a replacement?

The right answer in these cases is to ask the vendor for as much information as possible to reduce the risk in accepting this so-called replacement. First, ask for the test plans and test output so you can understand whether they tested the component in a way that relates to your use case. Just because they tested the thing doesn't mean they tested all the parts your product cares about. In fact, it's unlikely they did.

Walk a Mile in Their Shoes:
The Covid pandemic through the lens of four tech workers

Covid has changed how people work in many ways, but many of the outcomes have been paradoxical in nature. What works for one person may not work for the next (or even the same person the next day), and we have yet to figure out how to predict exactly what will work for everyone. As you saw in the composite personas described here, some people struggle with isolation and loneliness, have a hard time connecting socially with their teams, or find the time pressures of hybrid work with remote teams to be overwhelming. Others relish this newfound way of working, enjoying more time with family, greater flexibility to exercise during the day, a better work/life balance, and a stronger desire to contribute to the world.

FHIR: Reducing Friction in the Exchange of Healthcare Data:
A discussion with James Agnew, Pat Helland, and Adam Cole

With the full clout of the Centers for Medicare and Medicaid Services currently being brought to bear on healthcare providers to meet high standards for patient data interoperability and accessibility, it would be easy to assume the only reason this goal wasn't accomplished long ago is simply a lack of will. Interoperable data? How hard can that be? Much harder than you think, it turns out. To dig into why this is the case, we asked Pat Helland, a principal architect at Salesforce, to speak with James Agnew (CTO) and Adam Cole (senior solutions architect) of Smile CDR, a Toronto, Ontario-based provider of a leading platform used by healthcare organizations to achieve FHIR (Fast Healthcare Interoperability Resources) compliance.

Long Live Software Easter Eggs!:
They are as old as software.

It's a period of unrest. Rebel developers, striking from continuous deployment servers, have won their first victory. During the battle, rebel spies managed to push an epic commit in the HTML code of https://pro.sony. Pursued by sinister agents, the rebels are hiding in commits, buttons, tooltips, API, HTTP headers, and configuration screens.

Persistent Memory Allocation:
Leverage to move a world of software

A lever multiplies the force of a light touch, and the right software interfaces provide formidable leverage in multiple layers of code: A familiar interface enables a new persistent memory allocator to breathe new life into an enormous installed base of software and hardware. Compatibility allows a persistent heap to slide easily beneath a widely used scripting-language interpreter, thereby endowing all scripts with effortless on-demand persistence.

Autonomous Computing:
We frequently compute across autonomous boundaries but the implications of the patterns to ensure independence are rarely discussed.

Autonomous computing is a pattern for business work using collaborations to connect fiefdoms and their emissaries. This pattern, based on paper forms, has been used for centuries. Here, we explain fiefdoms, collaborations, and emissaries. We examine how emissaries work outside the autonomous boundary and are convenient while remaining an outsider. And we examine how work across different fiefdoms can be initiated, run for long periods of time, and eventually be completed.

Distributed Latency Profiling through Critical Path Tracing:
CPT can provide actionable and precise latency analysis.

Low latency is an important feature for many Google applications such as Search, and latency-analysis tools play a critical role in sustaining low latency at scale. For complex distributed systems that include services that constantly evolve in functionality and data, keeping overall latency to a minimum is a challenging task. In large, real-world distributed systems, existing tools such as RPC telemetry, CPU profiling, and distributed tracing are valuable to understand the subcomponents of the overall system, but are insufficient to perform end-to-end latency analyses in practice.

The Planning and Care of Data:
Rearranging buckets for no good reason

Questions such as, "How do we secure this data?" work only if you ask them at the start, and not when some lawyers or government officials are sitting in a conference room, rooting through your data and logs, and making threatening noises under their breath. All the things we care about with our data require forethought, but it seems in our rush to create "stakeholder value" we are willing to sacrifice these important attributes and just act like data gourmands, until, like Mr.

Middleware 101:
What to know now and for the future

Whether segregating a sophisticated software component into smaller services, transferring data between computers, or creating a general gateway for seamless communication, you can rely on middleware to achieve communication between different devices, applications, and software layers. Following the increasing agile movement, the tech industry has adopted the use of fast waterfall models to create stacks of layers for each structural need, including integration, communication, data, and security. Given this scope, emphasis must now be on endpoint connection and agile development. This means that middleware should not serve solely as an object-oriented solution to execute simple request-response commands.

Persistence Programming:
Are we doing this right?

A few years ago, my team was working on a commercial Java development project for Enhanced 911 (E911) emergency call centers. We were frustrated by trying to meet the data-storage requirements of this project using the traditional model of Java over an SQL database. After some reflection about the particular requirements (and nonrequirements) of the project, we took a deep breath and decided to create our own custom persistence layer from scratch.

Surveillance Too Cheap to Meter:
Stopping Big Brother would require an expensive overhaul of the entire system.

IT nerds tend to find technological solutions for all sorts of problems?economic, political, sociological, and so on. Most of the time, these solutions don't make the problems that much worse, but when a problem is of a purely economic nature, only solutions that affect the economics of the situation can possibly work. Neither cryptography nor smart programming will be able to move the needle even a little bit when the fundamental problem is that surveillance is too cheap to meter.

FPGAs in Client Compute Hardware:
Despite certain challenges, FPGAs provide security and performance benefits over ASICs.

FPGAs (field-programmable gate arrays) are remarkably versatile. They are used in a wide variety of applications and industries where use of ASICs (application-specific integrated circuits) is less economically feasible. Despite the area, cost, and power challenges designers face when integrating FPGAs into devices, they provide significant security and performance benefits. Many of these benefits can be realized in client compute hardware such as laptops, tablets, and smartphones.

Getting Off the Mad Path:
Debuggers and assertions

KV continues to grind his teeth as he sees code loaded with debugging statements that would be totally unnecessary if the programmers who wrote the code could be both confident in and proficient with their debuggers. If one is lucky enough to have access to a good debugger, one should give extreme thanks to whatever they normally give thanks to and use the damn thing!

The Keys to the Kingdom:
A deleted private key, a looming deadline, and a last chance to patch a new static root of trust into the bootloader

An unlucky fat-fingering precipitated the current crisis: The client had accidentally deleted the private key needed to sign new firmware updates. They had some exciting new features to ship, along with the usual host of reliability improvements. Their customers were growing impatient, but my client had to stall when asked for a release date. How could they come up with a meaningful date? They had lost the ability to sign a new firmware release.

Steampunk Machine Learning:
Victorian contrivances for modern data science

Fitting models to data is all the rage nowadays but has long been an essential skill of engineers. Veterans know that real-world systems foil textbook techniques by interleaving routine operating conditions with bouts of overload and failure; to be practical, a method must model the former without distortion by the latter. Surprisingly effective aid comes from an unlikely quarter: a simple and intuitive model-fitting approach that predates the Babbage Engine. The foundation of industrial-strength decision support and anomaly detection for production datacenters, this approach yields accurate yet intelligible models without hand-holding or fuss.

Interpretable Machine Learning:
Moving from mythos to diagnostics

The emergence of machine learning as a society-changing technology in the past decade has triggered concerns about people's inability to understand the reasoning of increasingly complex models. The field of IML (interpretable machine learning) grew out of these concerns, with the goal of empowering various stakeholders to tackle use cases, such as building trust in models, performing model debugging, and generally informing real human decision-making.

A Conversation with Margo Seltzer and Mike Olson:
The history of Berkeley DB

Kirk McKusick sat down with Margo Seltzer and Mike Olson to discuss the history of Berkeley DB, for which they won the ACM Software System Award in 2021. Kirk McKusick has spent his career as a BSD and FreeBSD developer. Margo Seltzer has spent her career as a professor of computer science and as an entrepreneur of database software companies. Mike Olson started his career as a software developer and later started and managed several open-source software companies. Berkeley DB is a production-quality, scalable, NoSQL, Open Source platform for embedded transactional data management.

It Takes a Community:
The Open-source Challenge

Of the many challenges faced by open-source developers, among the most daunting are some that other programmers scarcely ever think about. Building a successful open-source community depends on many different elements, some of which are familiar to any developer. Just as important are the skills to recruit, to inspire, to mentor, to manage, and to mediate disputes. But what exactly does it take to pull all that off?

Federated Learning and Privacy:
Building privacy-preserving systems for machine learning and data science on decentralized data

Centralized data collection can expose individuals to privacy risks and organizations to legal risks if data is not properly managed. Federated learning is a machine learning setting where multiple entities collaborate in solving a machine learning problem, under the coordination of a central server or service provider. Each client's raw data is stored locally and not exchanged or transferred; instead, focused updates intended for immediate aggregation are used to achieve the learning objective.

Chip Measuring Contest:
The benefits of purpose-built chips

Alan Kay once said, "People who are really serious about software should make their own hardware." We are now seeing product companies genuinely live up to this value. It is exciting when the incumbents known as the chip vendors are being outdone, in the very technology that is their bread and butter, by their previous customers. Let's dive into some of the interesting bits of these purpose-built chips: the benefits of economics, user experience, and performance for the companies building them.

Meaning and Context in Computer Programs:
Sharing domain knowledge among programmers using the source code as the medium

When you look at a function program's source code, how do you know what it means? Is the meaning found in the return values of the function, or is it located inside the function body? What about the function name? Answering these questions is important to understanding how to share domain knowledge among programmers using the source code as the medium. The program is the medium of communication among programmers to share their solutions.

Lamboozling Attackers: A New Generation of Deception:
Software engineering teams can exploit attackers' human nature by building deception environments.

The goal of this article is to educate software leaders, engineers, and architects on the potential of deception for systems resilience and the practical considerations for building deception environments. By examining the inadequacy and stagnancy of historical deception efforts by the information security community, the article also demonstrates why engineering teams are now poised to become significantly more successful owners of deception systems.

Patent Absurdity:
A case when ignorance is the best policy

The main reason a lawyer will give for not reading a software patent is that, if you run afoul of the patent and it can be shown that you had knowledge of it, your company will incur triple the damages that they would have, had you not had knowledge of the patent. That seems like reason enough to avoid reading them, but there is an even better reason, and that is, as design or technical documents, software patents suck.

The Software Industry IS STILL the Problem:
The time is (also) way overdue for IT professional liability

The time is way overdue for IT engineers to be subject to professional liability, like almost every other engineering profession. Before you tell me that is impossible, please study how the very same thing happened with electricity, planes, cranes, trains, ships, automobiles, lifts, food processing, buildings, and, for that matter, driving a car.

Crashproofing the Original NoSQL Key-Value Store

Fortifying software to protect persistent data from crashes can be remarkably easy if a modern file system handles the heavy lifting. This episode of Drill Bits unveils a new crash-tolerance mechanism that vaults the venerable gdbm database into the league of transactional NoSQL data stores. We'll motivate this upgrade by tracing gdbm's history. We'll survey the subtle science of crashproofing, navigating a minefield of traps for the unwary. We'll arrive at a compact and rugged design that leverages modern file-system features, and we'll tour the production-ready implementation of this design and its ergonomic interface.

Designing UIs for Static Analysis Tools:
Evaluating tool design guidelines with SWAN

Static-analysis tools suffer from usability issues such as a high rate of false positives, lack of responsiveness, and unclear warning descriptions and classifications. Here, we explore the effect of applying user-centered approach and design guidelines to SWAN, a security-focused static-analysis tool for the Swift programming language. SWAN is an interesting case study for exploring static-analysis tool usability because of its large target audience, its potential to integrate easily into developers' workflows, and its independence from existing analysis platforms.

Human-Centered Approach to Static-Analysis-Driven Developer Tools:
The future depends on good HCI

Complex and opaque systems do not scale easily. A human-centered approach for evolving tools and practices is essential to ensuring that software is scaled safely and securely. Static analysis can unveil information about program behavior, but the goal of deriving this information should not be to accumulate hairsplitting detail. HCI can help direct static-analysis techniques into developer-facing systems that structure information and embody relationships in representations that closely mirror a programmer's thought. The survival of great software depends on programming languages that support, rather than inhibit, communicating, reasoning, and abstract thinking.

Static Analysis at GitHub:
An experience report

The Semantic Code team at GitHub builds and operates a suite of technologies that power symbolic code navigation on github.com. We learned that scale is about adoption, user behavior, incremental improvement, and utility. Static analysis in particular is difficult to scale with respect to human behavior; we often think of complex analysis tools working to find potentially problematic patterns in code and then trying to convince the humans to fix them.

Static Analysis: An Introduction:
The fundamental challenge of software engineering is one of complexity.

Modern static-analysis tools provide powerful and specific insights into codebases. The Linux kernel team, for example, developed Coccinelle, a powerful tool for searching, analyzing, and rewriting C source code; because the Linux kernel contains more than 27 million lines of code, a static-analysis tool is essential both for finding bugs and for making automated changes across its many libraries and modules. Another tool targeted at the C family of languages is Clang scan-build, which comes with many useful analyses and provides an API for programmers to write their own analyses. Like so many things in computer science, the utility of static analysis is self-referential: To write reliable programs, we must also write programs for our programs.

Don't Get Stuck in the "Con" Game:
Consistency, convergence, and confluence are not the same! Eventual consistency and eventual convergence aren't the same as confluence, either.

"Eventual consistency" is a popular phrase with a fuzzy definition. People are even inconsistent in their use of consistency. But two other terms, "convergence" and "confluence", that have crisper definitions and are more easily understood.

Declarative Machine Learning Systems:
The future of machine learning will depend on it being in the hands of the rest of us.

The people training and using ML models now are typically experienced developers with years of study working within large organizations, but the next wave of ML systems should allow a substantially larger number of people, potentially without any coding skills, to perform the same tasks. These new ML systems will not require users to fully understand all the details of how models are trained and used for obtaining predictions, but will provide them a more abstract interface that is less demanding and more familiar.

Real-world String Comparison:
How to handle Unicode sequences correctly

In many languages a string comparison is a pitfall for beginners. With any Unicode string as input, a comparison often causes problems even for advanced users. The semantic equivalence of different characters in Unicode requires a normalization of the strings before comparing them. This article shows how to handle Unicode sequences correctly. The comparison of two strings for equality often raises questions concerning the difference between comparison by value, comparison of object references, strict equality, and loose equality. The most important aspect is semantic equivalence.

Divide and Conquer:
The use and limits of bisection

Bisection is of no use if you have a heisenbug that fails only from time to time. These subtle bugs are the hardest to fix and the ones that cause us to think critically about what we are doing. Timing bugs, bugs in distributed systems, and all the difficult problems we face in building increasingly complex software systems can't yet be addressed by simple bisection. It's often the case that it would take longer to write a usable bisection test for a complex problem than it would to analyze the problem whilst at the tip of the tree.

When Curation Becomes Creation:
Algorithms, microcontent, and the vanishing distinction between platforms and creators

Media platforms today benefit from: (1) discretion to organize content, (2) algorithms for curating user-posted content, and (3) absolution from liability. This favorable regulatory environment results from the current legal framework, which distinguishes between intermediaries and content providers. This distinction is ill-adapted to the modern social media landscape, where platforms deploy powerful data-driven algorithms to play an increasingly active role in shaping what people see, and where users supply disconnected bits of raw content as fodder.

Digging into Big Provenance (with SPADE):
A user interface for querying provenance

Several interfaces exist for querying provenance. Many are not flexible in allowing users to select a database type of their choice. Some provide query functionality in a data model that is different from the graph-oriented one that is natural for provenance. Others have intuitive constructs for finding results but have limited support for efficiently chaining responses, as needed for faceted search. This article presents a user interface for querying provenance that addresses these concerns and is agnostic to the underlying database being used.

What Went Wrong?:
Why we need an IT accident investigation board

Governments should create IT accident investigation boards for the exact same reasons they have done so for ships, railroads, planes, and in many cases, automobiles. Denmark got its Railroad Accident Investigation Board because too many people were maimed and killed by steam trains. The UK's Air Accidents Investigation Branch was created for pretty much the same reasons, but, specifically, because when the airlines investigated themselves, nobody was any the wiser. Does that sound slightly familiar in any way?

A New Era for Mechanical CAD:
Time to move forward from decades-old design

The hardware industry is desperate for a modern way to do mechanical design. A new CAD program created for the modern world would lower the barrier to building hardware, decrease the time of development, and usher in a new era of building. The tools used to build with today are supported on the shoulders of giants, but a lot could be done to make them even better. At some point, mechanical CAD lost some of its roots of innovation.

ACID: My Personal:
How could I miss such a simple thing?

I had a chance recently to chat with my old friend, Andreas Reuter, the inventor of ACID. He and his Ph.D. advisor, Theo Härder, coined the term in their famous 1983 paper, Principles of Transaction-Oriented Database Recovery. I had blinders on after almost four decades of seeing C based on my assumptions. One big lesson for me is to work hard to ALWAYS question your assumptions. Try hard to surround yourself with curious and passionate people, both young and old, who will challenge you and try to dislodge your blinders.

In Praise of the Disassembler:
There's much to be learned from the lower-level details of hardware.

When you're starting out you want to be able to hold the entire program in your head if at all possible. Once you're conversant with your first, simple assembly language and the machine architecture you're working with, it will be completely possible to look at a page or two of your assembly and know not only what it is supposed to do but also what the machine will do for you step by step. When you look at a high-level language, you should be able to understand what you mean it to do, but often you have no idea just how your intent will be translated into action.

Schrödinger's Code:
Undefined behavior in theory and practice

Undefined behavior ranks among the most baffling and perilous aspects of popular programming languages. This installment of Drill Bits clears up widespread misconceptions and presents practical techniques to banish undefined behavior from your own code and pinpoint meaningless operations in any software—techniques that reveal alarming faults in software supporting business-critical applications at Fortune 500 companies.

Quantum-safe Trust for Vehicles:
The race is already on

In the automotive industry, cars now coming off assembly lines are sometimes referred to as "rolling data centers" in acknowledgment of all the entertainment and communications capabilities they contain. The fact that autonomous driving systems are also well along in development does nothing to allay concerns about security. Indeed, it would seem the stakes of automobile cybersecurity are about to become immeasurably higher just as some of the underpinnings of contemporary cybersecurity are rendered moot.

The Complex Path to Quantum Resistance:
Is your organization prepared?

There is a new technology on the horizon that will forever change the information security and privacy industry landscape. Quantum computing, together with quantum communication, will have many beneficial applications but will also be capable of breaking many of today's most popular cryptographic techniques that help ensure data protection?in particular, confidentiality and integrity of sensitive information. These techniques are ubiquitously embedded in today's digital fabric and implemented by many industries such as finance, health care, utilities, and the broader information communication technology (ICT) community.

Biases in AI Systems:
A survey for practitioners

This article provides an organization of various kinds of biases that can occur in the AI pipeline starting from dataset creation and problem formulation to data analysis and evaluation. It highlights the challenges associated with the design of bias-mitigation strategies, and it outlines some best practices suggested by researchers. Finally, a set of guidelines is presented that could aid ML developers in identifying potential sources of bias, as well as avoiding the introduction of unwanted biases. The work is meant to serve as an educational resource for ML developers in handling and addressing issues related to bias in AI systems.

Fail-fast Is Failing... Fast!:
Changes in compute environments are placing pressure on tried-and-true distributed-systems solutions.

For more than 40 years, fail-fast has been the dominant way of achieving fault tolerance. In this approach, some mechanism is responsible for ensuring that each component is up, functioning, and responding to work. As the industry moves to leverage cloud computing, this is getting more challenging. The way we create robust solutions is under pressure as the individual components don't fail fast but instead, starts running slow, which is far worse The slow component may be healthy enough to say, "I'm still here!" but slow enough to clog up all the work. This makes fail-fast schemes vulnerable.

Software Development in Disruptive Times:
Creating a software solution with fast decision capability, agile project management, and extreme low-code technology

In this project, the challenge was to "deploy software faster than the coronavirus spread." In a project with such peculiar characteristics, several factors can influence success, but some clearly stand out: top management support, agility, understanding and commitment of the project team, and the technology used. Conventional development approaches and technologies would simply not be able to meet the requirements promptly.

Aversion to Versions:
Resolving code-dependency issues

One should never hardcode a version or a path inside the code itself. Code needs to be flexible so that it can be installed anywhere and run anywhere so long as the necessary dependencies can be resolved, either at build time for statically compiled code or at runtime for interpreted code or code with dynamically linked libraries. There are current, good ways to get this right, so it's a shame that so many people continue to get it wrong.

WebRTC - Realtime Communication for the Open Web Platform:
What was once a way to bring audio and video to the web has expanded into more use cases we could ever imagine.

In this time of pandemic, the world has turned to Internet-based, RTC (realtime communication) as never before. The number of RTC products has, over the past decade, exploded in large part because of cheaper high-speed network access and more powerful devices, but also because of an open, royalty-free platform called WebRTC. WebRTC is growing from enabling useful experiences to being essential in allowing billions to continue their work and education, and keep vital human contact during a pandemic. The opportunities and impact that lie ahead for WebRTC are intriguing indeed.

Toward Confidential Cloud Computing:
Extending hardware-enforced cryptographic protection to data while in use

Although largely driven by economies of scale, the development of the modern cloud also enables increased security. Large data centers provide aggregate availability, reliability, and security assurances. The operational cost of ensuring that operating systems, databases, and other services have secure configurations can be amortized among all tenants, allowing the cloud provider to employ experts who are responsible for security; this is often unfeasible for smaller businesses, where the role of systems administrator is often conflated with many others.

The SPACE of Developer Productivity:
There's more to it than you think.

Developer productivity is about more than an individual's activity levels or the efficiency of the engineering systems relied on to ship software, and it cannot be measured by a single metric or dimension. The SPACE framework captures different dimensions of productivity, and here we demonstrate how this framework can be used to understand productivity in practice and why using it will help teams better understand developer productivity and create better measures to inform their work and teams.

Offline Algorithms in Low-Frequency Trading:
Clearing Combinatorial Auctions

Expectations run high for software that makes real-world decisions, particularly when money hangs in the balance. This third episode of the Drill Bits column shows how well-designed software can effectively create wealth by optimizing gains from trade in combinatorial auctions. We'll unveil a deep connection between auctions and a classic textbook problem, we'll see that clearing an auction resembles a high-stakes mutant Tetris, we'll learn to stop worrying and love an NP-hard problem that's far from intractable in practice, and we'll contrast the deliberative business of combinatorial auctions with the near-real-time hustle of high-frequency trading.

Enclaves in the Clouds:
Legal considerations and broader implications

With organizational data practices coming under increasing scrutiny, demand is growing for mechanisms that can assist organizations in meeting their data-management obligations. TEEs (trusted execution environments) provide hardware-based mechanisms with various security properties for assisting computation and data management. TEEs are concerned with the confidentiality and integrity of data, code, and the corresponding computation. Because the main security properties come from hardware, certain protections and guarantees can be offered even if the host privileged software stack is vulnerable.

Let's Play Global Thermonuclear Energy:
It's important to know where your power comes from.

For us to grow and progress as a civilization, we need more investment in providing electricity to the world through clean, safe, and efficient processes. Thermonuclear energy is a huge step forward. This article is mostly focused on the use cases around grid-scale reactors. It's hard to see a future without some sort of thermonuclear energy powering all sorts of things around us.

Best Practice: Application Frameworks:
While powerful, frameworks are not for everyone.

While frameworks can be a powerful tool, they have some disadvantages and may not make sense for all organizations. Framework maintainers need to provide standardization and well-defined behavior while not being overly prescriptive. When frameworks strike the right balance, however, they can offer large developer productivity gains. The consistency provided by widespread use of frameworks is a boon for other teams such as SRE and security that have a vested interest in the quality of applications. Additionally, the structure of frameworks provides a foundation for building higher-level abstractions such as microservices platforms, which unlock new opportunities for system architecture and automation.

The Non-psychopath's Guide to Managing an Open-source Project:
Respect your staff, learn from others, and know when to let go.

Transitioning from one of the technical faithful to one of the hated PHBs (pointy-haired bosses), whether in the corporate or the open-source world, is truly a difficult transition. Unless you are a type who has always been meant for the C-suite?, it's going to take a lot of work and a lot of patience, mostly with yourself, to make this transition.

Baleen Analytics:
Large-scale filtering of data provides serendipitous surprises.

Data analytics hoovers up anything it can find and we are finding patterns and insights that weren't available before, with implications for both data analytics and for messaging between services and microservices. It seems that a pretty good understanding among many different sources allows more flexibility and interconnectivity. Increasingly, flexibility dominates perfection.

Always-on Time-series Database: Keeping Up Where There's No Way to Catch Up:
A discussion with Theo Schlossnagle, Justin Sheehy, and Chris McCubbin

What if you found you needed to provide for the capture of data from disconnected operations, such that updates might be made by different parties at the same time without conflicts? And what if your service called for you to receive massive volumes of data almost continuously throughout the day, such that you couldn't really afford to interrupt data ingest at any point for fear of finding yourself so far behind present state that there would be almost no way to catch up?

Everything VPN is New Again:
The 24-year-old security model has found a second wind.

The VPN (virtual private network) is 24 years old. The concept was created for a radically different Internet from the one we know today. As the Internet grew and changed, so did VPN users and applications. The VPN had an awkward adolescence in the Internet of the 2000s, interacting poorly with other widely popular abstractions. In the past decade the Internet has changed again, and this new Internet offers new uses for VPNs. The development of a radically new protocol, WireGuard, provides a technology on which to build these new VPNs.

Battery Day:
A closer look at the technology that makes portable electronics possible

Tesla held its first Battery Day on September 22, 2020. The Tesla team didn't just look at one angle but all the angles: cell design, manufacturing, vehicle integration, and materials. If Tesla were to achieve 400 watt-hours per kilogram, a zero-emissions jet just might be on the horizon.

Differential Privacy: The Pursuit of Protections by Default:
A discussion with Miguel Guevara, Damien Desfontaines, Jim Waldo, and Terry Coatta

First formalized in 2006, differential privacy is an approach based on a mathematically rigorous definition of privacy that allows formalization and proof of the guarantees against re-identification offered by a system. While differential privacy has been accepted by theorists for some time, its implementation has turned out to be subtle and tricky, with practical applications only now starting to become available. To date, differential privacy has been adopted by the U.S. Census Bureau, along with a number of technology companies, but what this means and how these organizations have implemented their systems remains a mystery to many.

Kabin Fever:
KV's guidelines for KFH (koding from home)

Let me invite you to my next Zoom meeting on how to host Zoom meetings! As a devotee of mobile computing and remote work from my earliest days at university, I have, over time, developed a number of useful habits for maintaining a good and productive working rhythm, and I've found that many of these apply well to those of you who are newly working from home.

Decentralized Computing

Feeding all relevant inputs to a central solver is the obvious way to tackle a problem, but it's not always the only way. Decentralized methods that make do with only local communication and local computation are sometimes the best way. This episode of Drill Bits reviews an elegant protocol for self-organizing wireless networks that can also solve a seemingly impossible social networking problem. The protocol preserves privacy among participants and is so simple that it can be implemented with pencil, paper, and postcards. Example software implements both the decentralized protocol and a centralized solver.

The Time I Stole $10,000 from Bell Labs:
Or why DevOps encourages us to celebrate outages

If IT workers fear they will be punished for outages, they will adopt behavior that leads to even larger outages. Instead, we should celebrate our outages: Document them blamelessly, discuss what we've learned from them openly, and spread that knowledge generously. An outage is not an expense. It is an investment in the people who have learned from it. We can maximize that investment through management practices that maximize learning for those involved and by spreading that knowledge across the organization. Managed correctly, every outage makes the organization smarter.

A Second Conversation with Werner Vogels:
The Amazon CTO sits with Tom Killalea to discuss designing for evolution at scale.

When I joined Amazon in 1998, the company had a single US-based website selling only books and running a monolithic C application on five servers, a handful of Berkeley DBs for key/value data, and a relational database. That database was called "ACB" which stood for "Amazon.Com Books," a name that failed to reflect the range of our ambition. In 2006 acmqueue published a conversation between Jim Gray and Werner Vogels, Amazon's CTO, in which Werner explained that Amazon should be viewed not just as an online bookstore but as a technology company. In the intervening 14 years, Amazon's distributed systems, and the patterns used to build and operate them, have grown in influence.

The Die is Cast:
Hardware Security is Not Assured

The future of hardware security will evolve with hardware. As packaging advances and focus moves to beyond Moore's law technologies, hardware security experts will need to keep ahead of changing security paradigms, including system and process vulnerabilities. Research focused on quantum hacking is emblematic of the translation of principles of security on the physical attack plane for emerging communications and computing technologies. Perhaps the commercial market will evolve such that the GAO will run a study on compromised quantum technologies in the not-too-distant future.

Out-of-this-World Additive Manufacturing:
From thingamabobs to rockets, 3D printing takes many forms.

Popular culture uses the term 3D printing as a synonym for additive manufacturing processes. In 2010, the American Society for Testing and Materials group came up with a set of standards to classify additive manufacturing processes into seven categories. Each process uses different materials and machine technology, which affects the use cases and applications, as well as the economics. I went down a rabbit hole researching the various processes in my hunt to buy the best 3D printer.

The Identity in Everyone's Pocket:
Keeping users secure through their smartphones

Newer phones use security features in many different ways and combinations. As with any security technology, however, using a feature incorrectly can create a false sense of security. As such, many app developers and service providers today do not use any of the secure identity-management facilities that modern phones offer. For those of you who fall into this camp, this article is meant to leave you with ideas about how to bring a hardware-backed and biometrics-based concept of user identity into your ecosystem.

Removing Kode:
Dead functions and dead features

Removing dead code from systems is one of KV's favorite koding pastimes because there is nothing quite like that feeling you get when you get rid of something you know wasn't being used. Code removal is like cleaning house, only sometimes you clean house with a flame thrower, which, honestly, is very satisfying. Since you're using a version-control system (you had better be using a VCS!), it's very easy to remove code without worry. If you ever need the code you removed, you can retrieve it from the VCS at will.

Security Analysis of SMS as a Second Factor of Authentication:
The challenges of multifactor authentication based on SMS, including cellular security deficiencies, SS7 exploits, and SIM swapping

Despite their popularity and ease of use, SMS-based authentication tokens are arguably one of the least secure forms of two-factor authentication. This does not imply, however, that it is an invalid method for securing an online account. The current security landscape is very different from that of two decades ago. Regardless of the critical nature of an online account or the individual who owns it, using a second form of authentication should always be the default option, regardless of the method chosen.

Efficient Graph Search

Welcome to Drill Bits, a new column about programming. This inaugural episode shows how graph search algorithms can avoid unnecessary work. A simple modification to classic breadth-first search improves the lower bound on its running time: Whereas classic BFS always requires time proportional to the number of vertices plus the number of edges, the improved "Efficient BFS" sometimes runs in time proportional to the number of vertices alone. Both asymptotic analysis and experiments show that Efficient BFS can be much faster than classic BFS.

The Life of a Data Byte:
Be kind and rewind.

One thing that remains true is the storing of 0s and 1s. The means by which that is done vary greatly. I hope the next time you burn a CD-RW with a mix of songs for a friend, or store home videos in an optical disc archive, you think about how the nonreflective bumps translate to a 0 and the reflective lands of the disk translate to a 1. If you are creating a mixtape on a cassette, remember that those are closely related to the Datasette used in the Commodore PET. Lastly, remember to be kind and rewind.

Scrum Essentials Cards:
Experiences of Scrum Teams Improving with Essence

This article presents a series of examples and case studies on how people have used the Scrum Essentials cards to benefit their teams and improve how they work. Scrum is one of the most popular agile frameworks used successfully all over the world. It has been taught and used for 15-plus years. It is by far the most-used practice when developing software, and it has been generalized to be applicable for not just software but all kinds of products. It has been taught to millions of developers, all based on the Scrum Guide.

Five Nonobvious Remote Work Techniques:
Emulating the efficiency of in-person conversations

The physical world has social conventions around conversations and communication that we use without even thinking. As we move to a remote-work world, we have to be more intentional to create such conventions. Developing these social norms is an ongoing commitment that outlasts initial technical details of VPN and desktop videoconference software configuration. Companies that previously forbade remote work can no longer deny its benefits. Once the pandemic-related lockdowns are over, many people will continue working remotely. Those who return to the office will need to work in ways that are compatible with their remotely working associates.

Data on the Outside vs. Data on the Inside:
Data kept outside SQL has different characteristics from data kept inside.

This article describes the impact of services and trust on the treatment of data. It introduces the notions of inside data as distinct from outside data. After discussing the temporal implications of not sharing transactions across the boundaries of services, the article considers the need for immutability and stability in outside data. This leads to a depiction of outside data as a DAG of data items being independently generated by disparate services.

Sanity vs. Invisible Markings:
Tabs vs. spaces

Invisible and near-invisible markings bring us to the human part of the problem?not that code editor authors aren’t human, but most of us will not write new editors, though all of us will use editors. As we all know, once upon a time computers had small memories and the difference between a tab, which is a single byte, and a corresponding number of spaces (8) could be a significant difference between the size of source code stored on a precious disk, and also transferred, over whatever primitive and slow bus, from storage into memory.

The History, Status, and Future of FPGAs:
Hitting a nerve with field-programmable gate arrays

This article is a summary of a three-hour discussion at Stanford University in September 2019 among the authors. It has been written with combined experiences at and with organizations such as Zilog, Altera, Xilinx, Achronix, Intel, IBM, Stanford, MIT, Berkeley, University of Wisconsin, the Technion, Fairchild, Bell Labs, Bigstream, Google, DIGITAL (DEC), SUN, Nokia, SRI, Hitachi, Silicom, Maxeler Technologies, VMware, Xerox PARC, Cisco, and many others. These organizations are not responsible for the content, but may have inspired the authors in some ways, to arrive at the colorful ride through FPGA space described above.

Broken Hearts and Coffee Mugs:
The ordeal of security reviews

Overall, there are two broad types of security review: white box and black box. A white-box review is one in which the attackers have nearly full access to information such as code, design documents, and other information that will make it easier for them to design and carry out a successful attack. A black-box review, or test, is one in which the attackers can see the system only in the same way that a normal user or consumer would.

Debugging Incidents in Google’s Distributed Systems:
How experts debug production issues in complex distributed systems

This article covers the outcomes of research performed in 2019 on how engineers at Google debug production issues, including the types of tools, high-level strategies, and low-level tasks that engineers use in varying combinations to debug effectively. It examines the research approach used to capture data, summarizing the common engineering journeys for production investigations and sharing examples of how experts debug complex distributed systems. Finally, the article extends the Google specifics of this research to provide some practical strategies that you can apply in your organization.

Power to the People:
Reducing datacenter carbon footprints

By designing rack-level architectures, huge improvements can be made for power efficiency over conventional servers, since PSUs will be less oversized, more consolidated, and redundant for the rack versus per server. While the hyperscalers have benefited from these gains in power efficiency, most of the industry is still waiting. The Open Compute Project was started as an effort to allow other companies running datacenters to benefit from the power efficiencies as well. If more organizations run rack-scale architectures in their datacenters, the wasted carbon emissions caused by conventional servers can be lessened.

Is Persistent Memory Persistent?:
A simple and inexpensive test of failure-atomic update mechanisms

Power failures pose the most severe threat to application data integrity, and painful experience teaches that the integrity promises of failure-atomic update mechanisms can’t be taken at face value. Diligent developers and operators insist on confirming integrity claims by extensive firsthand tests. This article presents a simple and inexpensive testbed capable of subjecting storage devices, system software, and application software to ten thousand sudden whole-system power-interruption tests per week.

Dark Patterns: Past, Present, and Future:
The evolution of tricky user interfaces

Dark patterns are an abuse of the tremendous power that designers hold in their hands. As public awareness of dark patterns grows, so does the potential fallout. Journalists and academics have been scrutinizing dark patterns, and the backlash from these exposures can destroy brand reputations and bring companies under the lenses of regulators. Design is power. In the past decade, software engineers have had to confront the fact that the power they hold comes with responsibilities to users and to society. In this decade, it is time for designers to learn this lesson as well.

How Do Committees Invent? and Ironies of Automation:
The formulation of Conway’s law and the counterintuitive consequences of increasing levels of automation

The Lindy effect tells us that if a paper has been highly relevant for a long time, it’s likely to continue being so for a long time to come as well. My first choice is "How Do Committees Invent?" Author Melvin E. Conway provides a lot of great material that led up to the formulation of the law that bears his name. My second choice is Lisanne Bainbridge’s "Ironies of Automation." It’s a classic treatise on the counterintuitive consequences of increasing levels of automation.

Kode Vicious Plays in Traffic:
With increasing complexity comes increasing risk.

There is no single answer to the question of how to apply software to systems that can, literally, kill us, but there are models to follow that may help ameliorate the risk. The risks involved in these systems come from three major areas: marketing, accounting, and management. It is not that it is impossible to engineer such systems safely, but the history of automated systems shows us that it is difficult to do so cheaply and quickly.

To Catch a Failure: The Record-and-Replay Approach to Debugging:
A discussion with Robert O’Callahan, Kyle Huey, Devon O’Dell, and Terry Coatta

When work began at Mozilla on the record-and-replay debugging tool called rr, the goal was to produce a practical, cost-effective, resource-efficient means for capturing low-frequency nondeterministic test failures in the Firefox browser. Much of the engineering effort that followed was invested in making sure the tool could actually deliver on this promise with a minimum of overhead. What was not anticipated, though, was that rr would come to be widely used outside of Mozilla?and not just for sleuthing out elusive failures, but also for regular debugging.

The Best Place to Build a Subway:
Building projects despite (and because of) existing complex systems

Many engineering projects are big and complex. They require integrating into the existing environment to tie into stuff that precedes the new, big, complex thing. It is common to bemoan the challenges of dealing with the preexisting stuff. Many times, engineers don’t realize that their projects (and their paychecks) exist only because of the preexisting and complex systems that impose constraints on the new work. This column looks at some sophisticated urban redevelopment projects that are very much part of daily life in San Francisco and compares them with the challenges inherent in building software.

Demystifying Stablecoins:
Cryptography meets monetary policy

Self-sovereign stablecoins are interesting and probably here to stay; however, they face numerous regulatory hurdles from banking, financial tracking, and securities laws. For stablecoins backed by a governmental currency, the ultimate expression would be a CBDC. Since paper currency has been in steady decline (and disproportionately for legitimate transactions), a CBDC could reintroduce cash with technological advantages and efficient settlement while minimizing user fees.

Chipping Away at Moore’s Law:
Modern CPUs are just chiplets connected together.

Smaller transistors can do more calculations without overheating, which makes them more power efficient. It also allows for smaller die sizes, which reduce costs and can increase density, allowing more cores per chip. The silicon wafers that chips are made of vary in purity, and none are perfect, which means every chip has a chance of having imperfections that differ in effect. Manufacturers can limit the effect of imperfections by using chiplets.

Communicate Using the Numbers 1, 2, 3, and More:
Leveraging expectations for better communication

People often use lists of various sizes when communicating. I might have 2 reasons for supporting the new company strategy. I might tell you my 3 favorite programming languages. I might make a presentation that describes 4 new features. There is 1 vegetable that I like more than any other. The length of the list affects how the audience interprets what is being said. Not aligning with what the human brain expects is like swimming upstream. Given the choice, why would anyone do that?

The Way We Think About Data:
Human inspection of black-box ML models; reclaiming ownership of data

The two papers I’ve chosen for this issue of acmqueue both challenge the way we think about and use data, though in very different ways. In "Stop Explaining Black-box Machine-learning Models for High-stakes Decisions and Use Interpretable Models Instead," Cynthia Rudin makes the case for models that can be inspected and interpreted by human experts. The second paper, "Local-first Software: You Own Your Data, in Spite of the Cloud," describes how to retain sovereignty over your data.

Master of Tickets:
Valuing the quality, not the quantity, of work

Many silly metrics have been created to measure work, including the rate at which tickets are closed, the number of lines of code a programmer writes in a day, and the number of words an author can compose in an hour. All of these measures have one thing in common: They fail to take into account the quality of the output. If Alice writes 1,000 lines of impossible-to-read, buggy code in a day and Carol writes 100 lines of well-crafted, easy-to-use code in the same time, then who should be rewarded?

Securing the Boot Process:
The hardware root of trust

The goal of a hardware root of trust is to verify that the software installed in every component of the hardware is the software that was intended. This way you can verify and know without a doubt whether a machine’s hardware or software has been hacked or overwritten by an adversary. In a world of modchips, supply chain attacks, evil maid attacks, cloud provider vulnerabilities in hardware components, and other attack vectors it has become more and more necessary to ensure hardware and software integrity.

Beyond the Fix-it Treadmill:
The Use of Post-Incident Artifacts in High-Performing Organizations

Given that humanity’s study of the sociological factors in safety is almost a century old, the technology industry’s post-incident analysis practices and how we create and use the artifacts those practices produce are all still in their infancy. So don’t be surprised that many of these practices are so similar, that the cognitive and social models used to parse apart and understand incidents and outages are few and cemented in the operational ethos, and that the byproducts sought from post-incident analyses are far-and-away focused on remediation items and prevention.

Managing the Hidden Costs of Coordination:
Controlling coordination costs when multiple, distributed perspectives are essential

Some initial considerations to control cognitive costs for incident responders include: (1) assessing coordination strategies relative to the cognitive demands of the incident; (2) recognizing when adaptations represent a tension between multiple competing demands (coordination and cognitive work) and seeking to understand them better rather than unilaterally eliminating them; (3) widening the lens to study the joint cognition system (integration of human-machine capabilities) as the unit of analysis; and (4) viewing joint activity as an opportunity for enabling reciprocity across inter- and intra-organizational boundaries.

Cognitive Work of Hypothesis Exploration During Anomaly Response:
A look at how we respond to the unexpected

Four incidents from web-based software companies reveal important aspects of anomaly response processes when incidents arise in web operations, two of which are discussed in this article. One particular cognitive function examined in detail is hypothesis generation and exploration, given the impact of obscure automation on engineers’ development of coherent models of the systems they manage. Each case was analyzed using the techniques and concepts of cognitive systems engineering. The set of cases provides a window into the cognitive work "above the line" in incident management of complex web-operation systems.

Above the Line, Below the Line:
The resilience of Internet-facing systems relies on what is below the line of representation.

Knowledge and understanding of below-the-line structure and function are continuously in flux. Near-constant effort is required to calibrate and refresh the understanding of the workings, dependencies, limitations, and capabilities of what is present there. In this dynamic situation no individual or group can ever know the system state. Instead, individuals and groups must be content with partial, fragmented mental models that require more or less constant updating and adjustment if they are to be useful.

Revealing the Critical Role of Human Performance in Software:
It’s time to revise our appreciation of the human side of Internet-facing software systems.

Understanding, supporting, and sustaining the capabilities above the line of representation require all stakeholders to be able to continuously update and revise their models of how the system is messy and yet usually manages to work. This kind of openness to continually reexamine how the system really works requires expanding the efforts to learn from incidents.

Numbers Are for Computers, Strings Are for Humans:
How and where software should translate data into a human-readable form

Unless what you are processing, storing, or transmitting are, quite literally, strings that come from and are meant to be shown to humans, you should avoid processing, storing, or transmitting that data as strings. Remember, numbers are for computers, strings are for humans. Let the computer do the work of presenting your data to the humans in a form they might find palatable. That’s where those extra bytes and instructions should be spent, not doing the inverse.

Opening up the Baseboard Management Controller:
If the CPU is the brain of the board, the BMC is the brain stem.

In 2011 Facebook announced the Open Compute Project to form a community around open-source designs and specifications for data center hardware. Since then, the project has expanded to all aspects of the open data center. This column focuses on the BMC and is an introduction to a complicated topic. The intention is to provide a full picture of the world of the open-source BMC ecosystem, starting with a brief overview of the BMC’s role in a system, touching on security concerns around the BMC, and then diving into some of the projects that have developed in the open-source ecosystem.

Blockchain Technology: What Is It Good for?:
Industry’s dreams and fears for this new technology

Business executives, government leaders, investors, and researchers frequently ask the following three questions: (1) What exactly is blockchain technology? (2) What capabilities does it provide? (3) What are good applications? Here we answer these questions thoroughly, provide a holistic overview of blockchain technology that separates hype from reality, and propose a useful lexicon for discussing the specifics of blockchain technology in the future.

API Practices If You Hate Your Customers:
APIs speak louder than words.

Do you have disdain for your customers? Do you wish they would go away? When you interact with customers are you silently fantasizing about them switching to your competitor’s product? In short, do you hate your customers? In this article, I document a number of industry best practices designed to show customers how much you hate them. All of them are easy to implement. Heck, your company may be doing many of these already.

The Reliability of Enterprise Applications:
Understanding enterprise reliability

Enterprise reliability is a discipline that ensures applications will deliver the required business functionality in a consistent, predictable, and cost-effective manner without compromising core aspects such as availability, performance, and maintainability. This article describes a core set of principles and engineering methodologies that enterprises can apply to help them navigate the complex environment of enterprise reliability and deliver highly reliable and cost-efficient applications.

Space Time Discontinuum:
Combining data from many sources may cause painful delays.

Back when you had only one database for an application to worry about, you didn’t have to think about partial results. You also didn’t have to think about data arriving after some other data. It was all simply there. Now, you can do so much more with big distributed systems, but you have to be more sophisticated in the tradeoff between timely answers and complete answers.

Optimizations in C++ Compilers:
A practical journey

There’s a tradeoff to be made in giving the compiler more information: it can make compilation slower. Technologies such as link time optimization can give you the best of both worlds. Optimizations in compilers continue to improve, and upcoming improvements in indirect calls and virtual function dispatch might soon lead to even faster polymorphism.

Back under a SQL Umbrella:
Unifying serving and analytical data; using a database for distributed machine learning

Procella is the latest in a long line of data processing systems at Google. What’s unique about it is that it’s a single store handling reporting, embedded statistics, time series, and ad-hoc analysis workloads under one roof. It’s SQL on top, cloud-native underneath, and it’s serving billions of queries per day over tens of petabytes of data. There’s one big data use case that Procella isn’t handling today though, and that’s machine learning. But in ’Declarative recursive computation on an RDBMS... or, why you should use a database for distributed machine learning,’ Jankov et al.

Putting Machine Learning into Production Systems:
Data validation and software engineering for machine learning

Breck et al. share details of the pipelines used at Google to validate petabytes of production data every day. With so many moving parts it’s important to be able to detect and investigate changes in data distributions before they can impact model performance. "Software Engineering for Machine Learning: A Case Study" shares lessons learned at Microsoft as machine learning started to pervade more and more of the company’s systems, moving from specialized machine-learning products to simply being an integral part of many products and services.

Hack for Hire:
Investigating the emerging black market of retail email account hacking services

Hack-for-hire services charging $100-$400 per contract were found to produce sophisticated, persistent, and personalized attacks that were able to bypass 2FA via phishing. The demand for these services, however, appears to be limited to a niche market, as evidenced by the small number of discoverable services, an even smaller number of successful services, and the fact that these attackers target only about one in a million Google users.

Write Amplification Versus Read Perspiration:
The tradeoffs between write and read

In computing, there’s an interesting trend where writing creates a need to do more work. You need to reorganize, merge, reindex, and more to make the stuff you wrote more useful. If you don’t, you must search or do other work to support future reads.

The Effects of Mixing Machine Learning and Human Judgment:
Collaboration between humans and machines does not necessarily lead to better outcomes.

Based on the theoretical findings from the existing literature, some policymakers and software engineers contend that algorithmic risk assessments such as the COMPAS software can alleviate the incarceration epidemic and the occurrence of violent crimes by informing and improving decisions about policing, treatment, and sentencing. Considered in tandem, these findings indicate that collaboration between humans and machines does not necessarily lead to better outcomes, and human supervision does not sufficiently address problems when algorithms err or demonstrate concerning biases.

Koding Academies:
A low-risk path to becoming a front-end plumber

Encourage your friend to pick a course that will introduce concepts that can be used into the future, rather than just a specific set of buzzword technologies that are hot this year. Most courses are based around Python. Encourage your friend to study that as a first computer language, as the concepts learned in Python can be applied in other languages and other fields.

Persistent Memory Programming on Conventional Hardware:
The persistent memory style of programming can dramatically simplify application software.

Driven by the advent of byte-addressable non-volatile memory, the persistent memory style of programming will gain traction among developers, taking its rightful place alongside existing paradigms for managing persistent application state. Until NVM becomes available on all computers, developers can use the techniques presented in this article to enjoy the benefits of persistent memory programming on conventional hardware.

DAML: The Contract Language of Distributed Ledgers:
A discussion between Shaul Kfir and Camille Fournier

We’ll see the same kind of Cambrian explosion we witnessed in the web world once we started using mutualized infrastructure in public clouds and frameworks. It took only three weeks to learn enough Ruby on Rails and Heroku to push out the first version of a management system for that brokerage. And that’s because I had to think only about the models, the views, and the controllers. The hardest part, of course, had to do with building a secure wallet.

What is a CSO Good For?:
Security requires more than an off-the-shelf solution.

The CSO is not a security engineer, so let’s contrast the two jobs to create a picture of what we should and should not see.

Demo Data as Code:
Automation helps collaboration.

A casual request for a demo dataset may seem like a one-time thing that doesn’t need to be automated, but the reality is that this is a collaborative process requiring multiple iterations and experimentation. There will undoubtedly be requests for revisions big and small, the need to match changing software, and to support new and revised demo stories. All of this makes automating the process worthwhile. Modern scripting languages make it easy to create ad hoc functions that act like a little language. A repeatable process helps collaboration, enables delegation, and saves time now and in the future.

Velocity in Software Engineering:
From tectonic plate to F-16

Software engineering occupies an increasingly critical role in companies across all sectors, but too many software initiatives end up both off target and over budget. A surer path is optimized for speed, open to experimentation and learning, agile, and subject to regular course correcting. Good ideas tend to be abundant, though execution at high velocity is elusive. The good news is that velocity is controllable; companies can invest systematically to increase it.

The Evolution of Management:
Transitioning up the ladder

With each step up, the job changes - but not all of the changes are obvious. You have to shift your mindset, and focus on building new skills that are often very different from the skills that made you successful in your previous role.

Open-source Firmware:
Step into the world behind the kernel.

Open-source firmware can help bring computing to a more secure place by making the actions of firmware more visible and less likely to do harm. This article’s goal is to make readers feel empowered to demand more from vendors who can help drive this change.

Time Protection in Operating Systems and Speaker Legitimacy Detection:
Operating system-based protection from timing-based side-channel attacks; implications of voice-imitation software

Timing-based side-channel attacks are a particularly tricky class of attacks to deal with because the very thing you’re often striving for can give you away. There are always more creative new instances of attacks to be found, so you need a principled way of thinking about defenses that address the class, not just a particular instantiation. That’s what Ge et al. give us in "Time Protection, the Missing OS Abstraction." Just as operating systems prevent spatial inference through memory protection, so future operating systems will need to prevent temporal inference through time protection. It’s going to be a long road to get there.

Surviving Software Dependencies:
Software reuse is finally here but comes with risks.

Software reuse is finally here, and its benefits should not be understated, but we’ve accepted this transformation without completely thinking through the potential consequences. The Copay and Equifax attacks are clear warnings of real problems in the way software dependencies are consumed today. There’s a lot of good software out there. Let’s work together to find out how to reuse it safely.

MUST and MUST NOT:
On writing documentation

Pronouncements without background or explanatory material are useless to those who are not also deeply steeped in the art and science of computer security or security in general. It takes a particular bend of mind to think like an attacker and a defender all at once, and most people are incapable of doing this; so, if you want the people reading the document to follow your guidance, then you must take them on a journey from ignorance to knowledge.

Extract, Shoehorn, and Load:
Data doesn’t always fit nicely into a new home.

It turns out that the business value of ill-fitting data is extremely high. The process of taking the input data, discarding what doesn’t fit, adding default or null values for missing stuff, and generally shoehorning it to the prescribed shape is important. The prescribed shape is usually one that is amenable to analysis for deeper meaning.

Access Controls and Health Care Records: Who Owns the Data?:
A discussion with David Evans, Richard McDonald, and Terry Coatta

What if health care records were handled in more of a patient-centric manner, using systems and networks that allow data to be readily shared by all the physicians, clinics, hospitals, and pharmacies a person might choose to share them with or have occasion to visit? And, more radically, what if it was the patients who owned the data?

The DevOps Phenomenon:
An executive crash course

Stressful emergency releases are a thing of the past for companies that subscribe to the DevOps method of software development and delivery. New releases are frequent. Bugs are fixed rapidly. New business opportunities are sought with gusto and confidence. New features are released, revised, and improved with rapid iterations. DevOps presents a strategic advantage for organizations when compared with traditional software-development methods. Leadership plays an important role during that transformation. DevOps is about providing guidelines for faster time to market of new software features and achieving a higher level of stability. Implementing cross-functional, product-oriented teams helps bridge the gaps between software development and operations.

Overly Attached:
Know when to let go of emotional attachment to your work.

A smart, senior engineer couldn’t make logical decisions if it meant deprecating the system he and his team had worked on for a number of years. Even though the best thing would have been to help another team create the replacement system, they didn’t want to entertain the idea because it would mean putting an end to something they had invested so much in. It is good to have strong ownership, but what happens when you get too attached?

Industry-scale Knowledge Graphs: Lessons and Challenges:
Five diverse technology companies show how it’s done

This article looks at the knowledge graphs of five diverse tech companies, comparing the similarities and differences in their respective experiences of building and using the graphs, and discussing the challenges that all knowledge-driven enterprises face today. The collection of knowledge graphs discussed here covers the breadth of applications, from search, to product descriptions, to social networks.

GAN Dissection and Datacenter RPCs:
Visualizing and understanding generative adversarial networks; datacenter RPCs can be general and fast.

Image generation using GANs (generative adversarial networks) has made astonishing progress over the past few years. While staring in wonder at some of the incredible images, it’s natural to ask how such feats are possible. "GAN Dissection: Visualizing and Understanding Generative Adversarial Networks" gives us a look under the hood to see what kinds of things are being learned by GAN units, and how manipulating those units can affect the generated images. February saw the 16th edition of the Usenix Symposium on Networked Systems Design and Implementation. Kalia et al. blew me away with their work on fast RPCs (remote procedure calls) in the datacenter.

Troubling Trends in Machine Learning Scholarship:
Some ML papers suffer from flaws that could mislead the public and stymie future research.

Flawed scholarship threatens to mislead the public and stymie future research by compromising ML’s intellectual foundations. Indeed, many of these problems have recurred cyclically throughout the history of AI and, more broadly, in scientific research. In 1976, Drew McDermott chastised the AI community for abandoning self-discipline, warning prophetically that "if we can’t criticize ourselves, someone else will save us the trouble." The current strength of machine learning owes to a large body of rigorous research to date, both theoretical and empirical. By promoting clear scientific thinking and communication, our community can sustain the trust and investment it currently enjoys.

Tom’s Top Ten Things Executives Should Know About Software:
Software acumen is the new norm.

Software is eating the world. To do their jobs well, executives and managers outside of technology will benefit from understanding some fundamentals of software and the software-delivery process.

Garbage Collection as a Joint Venture:
A collaborative approach to reclaiming memory in heterogeneous software systems

Cross-component tracing is a way to solve the problem of reference cycles across component boundaries. This problem appears as soon as components can form arbitrary object graphs with nontrivial ownership across API boundaries. An incremental version of CCT is implemented in V8 and Blink, enabling effective and efficient reclamation of memory in a safe manner.

How to Create a Great Team Culture (and Why It Matters):
Build safety, share vulnerability, establish purpose.

As leader of the team, you have significant influence over your team’s culture. You can institute policies and procedures that help make your team happy and productive, monitor team successes, and continually improve the team. Another important part of team culture, however, is helping people feel they are a part of creating it. How can you expand the job of creating a culture to other team members?

Online Event Processing:
Achieving consistency where distributed transactions have failed

Support for distributed transactions across heterogeneous storage technologies is either nonexistent or suffers from poor operational and performance characteristics. In contrast, OLEP is increasingly used to provide good performance and strong consistency guarantees in such settings. In data systems it is very common for logs to be used as internal implementation details. The OLEP approach is different: it uses event logs, rather than transactions, as the primary application programming model for data management. Traditional databases are still used, but their writes come from a log rather than directly from the application. The use of OLEP is not simply pragmatism on the part of developers, but rather it offers a number of advantages.

The Worst Idea of All Time:
Revelations at 100!

In February 2004, with the other members of the Queue editorial board, I was at our monthly in-person dinner meeting, where we gather to come up with interesting discussion topics that will result in practitioner-oriented articles (and the best authors to write them) for publication in Queue. It was only our second year in business, and although we had published some successful and widely read articles, Queue still had no regular columnists. I was initially invited to board meetings by another editorial board member, Eric Allman, and had written a couple of articles for the publication. I was also co-authoring my first book but had never been a columnist.

Net Neutrality: Unexpected Solution to Blockchain Scaling:
Cloud-delivery networks could dramatically improve blockchains’ scalability, but clouds must be provably neutral first.

Provably neutral clouds are undoubtedly a viable solution to blockchain scaling. By optimizing the transport layer, not only can the throughput be fundamentally scaled up, but the latency could be dramatically reduced. Indeed, the latency distribution in today’s data centers is already biased toward microsecond timescales for most of the flows, with millisecond timescales residing only at the tail of the distribution. There is no reason why a BDN point of presence would not be able to achieve a similar performance. Adding dedicated optical infrastructure among such BDN points of presence would further alleviate throughput and reduce latency, creating the backbone of an advanced BDN.

SageDB and NetAccel:
Learned models within the database system; network-accelerated query processing

The CIDR (Conference on Innovative Data Systems Research) runs once every two years, and luckily for us 2019 is one of those years. I’ve selected two papers from this year’s conference that highlight bold and exciting directions for data systems.

Identity by Any Other Name:
The complex cacophony of intertwined systems

New emerging systems and protocols both tighten and loosen our notions of identity, and that’s good! They make it easier to get stuff done. REST, IoT, big data, and machine learning all revolve around notions of identity that are deliberately kept flexible and sometimes ambiguous. Notions of identity underlie our basic mechanisms of distributed systems, including interchangeability, idempotence, and immutability.

Edge Computing:
Scaling resources within multiple administrative domains

Creating edge computing infrastructures and applications encompasses quite a breadth of systems research. Let’s take a look at the academic view of edge computing and a sample of existing research that will be relevant in the coming years.

Achieving Digital Permanence:
The many challenges to maintaining stored information and ways to overcome them

Today’s Information Age is creating new uses for and new ways to steward the data that the world depends on. The world is moving away from familiar, physical artifacts to new means of representation that are closer to information in its essence. We need processes to ensure both the integrity and accessibility of knowledge in order to guarantee that history will be known and true.

Know Your Algorithms:
Stop using hardware to solve software problems.

Knowing that your CPU is in use 100 percent of the time doesn’t tell you much about the overall system other than it’s busy, but busy with what? Maybe it’s sitting in a tight loop, or some clown added a bunch of delay loops during testing that are no longer necessary. Until you profile your system, you have no idea why the CPU is busy. All systems provide some form of profiling so that you can track down where the bottlenecks are, and it’s your responsibility to apply these tools before you spend money on brand new hardware.

Metrics That Matter:
Critical but oft-neglected service metrics that every SRE and product owner should care about

Measure your site reliability metrics, set the right targets, and go through the work to measure the metrics accurately. Then, you’ll find that your service runs better, with fewer outages, and much more user adoption.

Design Patterns for Managing Up:
Four challenging work situations and how to handle them

Challenges come up all the time at work. Spend time now thinking about how you want to be seen at work, and then think about how that version of you would respond to the challenges that you could encounter. When you have a plan in place, you are much more likely to succeed.

A Hitchhiker’s Guide to the Blockchain Universe:
Blockchain remains a mystery, despite its growing acceptance.

It is difficult these days to avoid hearing about blockchain. Despite the significant potential of blockchain, it is also difficult to find a consistent description of what it really is. This article looks at the basics of blockchain: the individual components, how those components fit together, and what changes might be made to solve some of the problems with blockchain technology.

Tear Down the Method Prisons! Set Free the Practices!:
Essence: a new way of thinking that promises to liberate the practices and enable true learning organizations

This article explains why we need to break out of this repetitive dysfunctional behavior, and it introduces Essence, a new way of thinking that promises to free the practices from their method prisons and thus enable true learning organizations.

Security for the Modern Age:
Securely running processes that require the entire syscall interface

Giving operators a usable means of securing the methods they use to deploy and run applications is a win for everyone. Keeping the usability-focused abstractions provided by containers, while finding new ways to automate security and defend against attacks, is a great path forward.

SQL is No Excuse to Avoid DevOps:
Automation and a little discipline allow better testing, shorter release cycles, and reduced business risk.

Using SQL databases is not an impediment to doing DevOps. Automating schema management and a little developer discipline enables more vigorous and repeatable testing, shorter release cycles, and reduced business risk. When you can confidently deploy new releases, you do it more frequently. New features that previously sat unreleased for weeks or months now reach users sooner. Bugs are fixed faster. Security holes are closed sooner. It enables the company to provide better value to customers.

Understanding Database Reconstruction Attacks on Public Data:
These attacks on statistical databases are no longer a theoretical danger.

With the dramatic improvement in both computer speeds and the efficiency of SAT and other NP-hard solvers in the last decade, DRAs on statistical databases are no longer just a theoretical danger. The vast quantity of data products published by statistical agencies each year may give a determined attacker more than enough information to reconstruct some or all of a target database and breach the privacy of millions of people. Traditional disclosure-avoidance techniques are not designed to protect against this kind of attack.

Writing a Test Plan:
Establish your hypotheses, methodologies, and expected results.

If you can think of each of your tests as an experiment with a hypothesis, a test methodology, and a test result, it should all fall into place rather than falling through the cracks.

The Importance of a Great Finish:
You have to finish strong, every time.

How can you make sure that you are recognized as a valuable member of your team, whose work is seen as critical to the team’s success? You have to finish strong, every time. Here is how to keep your momentum up and make the right moves to be a visible contributor to the final success of every project.

CodeFlow: Improving the Code Review Process at Microsoft:
A discussion with Jacek Czerwonka, Michaela Greiler, Christian Bird, Lucas Panjer, and Terry Coatta

Delivering a new set of capabilities for managing and improving Microsoft’s code-review process was the primary goal right from the start. In the course of accomplishing that, much was also learned about certain general code-review principles. In fact, subsequent research has offered surprising evidence of just how similar the impact can be when many of these principles are followed at companies other than Microsoft.

Benchmarking "Hello, World!":
Six different views of the execution of "Hello, World!" show what is often missing in today’s tools

As more and more software moves off the desktop and into data centers, and more and more cell phones use server requests as the other half of apps, observation tools for large-scale distributed transaction systems are not keeping up. This makes it tempting to look under the lamppost using simpler tools. You will waste a lot of high-pressure time following that path when you have a sudden complex performance crisis.

Using Remote Cache Service for Bazel:
Save time by sharing and reusing build and test output

Remote cache service is a new development that significantly saves time in running builds and tests. It is particularly useful for a large code base and any size of development team. Bazel is an actively developed open-source build and test system that aims to increase productivity in software development. It has a growing number of optimizations to improve the performance of daily development tasks.

A Chance Gardener:
Harvesting open-source products and planting the next crop

It is a very natural progression for a company to go from being a pure consumer of open source, to interacting with the project via patch submission, and then becoming a direct contributor. No one would expect a company to be a direct contributor to all the open-source projects it consumes, as most companies consume far more software than they would ever produce, which is the bounty of the open-source garden. It ought to be the goal of every company consuming open source to contribute something back, however, so that its garden continues to bear fruit, instead of rotting vegetables.

Why SRE Documents Matter:
How documentation enables SRE teams to manage new and existing services

SRE (site reliability engineering) is a job function, a mindset, and a set of engineering approaches for making web products and services run reliably. SREs operate at the intersection of software development and systems engineering to solve operational problems and engineer solutions to design, build, and run large-scale distributed systems scalably, reliably, and efficiently. A mature SRE team likely has well-defined bodies of documentation associated with many SRE functions.

How to Live in a Post-Meltdown and -Spectre World:
Learn from the past to prepare for the next battle.

Spectre and Meltdown create a risk landscape that has more questions than answers. This article addresses how these vulnerabilities were triaged when they were announced and the practical defenses that are available. Ultimately, these vulnerabilities present a unique set of circumstances, but for the vulnerability management program at Goldman Sachs, the response was just another day at the office.

How to Get Things Done When You Don’t Feel Like It:
Five strategies for pushing through

If you want to be successful, then it serves you better to rise to the occasion no matter what. That means learning how to push through challenges and deliver valuable results.

Tracking and Controlling Microservice Dependencies:
Dependency management is a crucial part of system and software design.

Dependency cycles will be familiar to you if you have ever locked your keys inside your house or car. You can’t open the lock without the key, but you can’t get the key without opening the lock. Some cycles are obvious, but more complex dependency cycles can be challenging to find before they lead to outages. Strategies for tracking and controlling dependencies are necessary for maintaining reliable systems.

The Obscene Coupling Known as Spaghetti Code:
Teach your junior programmers how to read code

Since you both are working on the same code base, you also have ample opportunity for leadership by showing this person how you code. You must do this carefully or the junior programmer will think you’re pulling rank, but, with a bit of gentle show and tell, you can get your Padawan to see what you’re driving at. This human interaction is often difficult for those of us who prefer to spend our days with seemingly logical machines.

Corp to Cloud: Google’s Virtual Desktops:
How Google moved its virtual desktops to the cloud

Over one-fourth of Googlers use internal, data-center-hosted virtual desktops. This on-premises offering sits in the corporate network and allows users to develop code, access internal resources, and use GUI tools remotely from anywhere in the world. Among its most notable features, a virtual desktop instance can be sized according to the task at hand, has persistent user storage, and can be moved between corporate data centers to follow traveling Googlers. Until recently, our virtual desktops were hosted on commercially available hardware on Google’s corporate network using a homegrown open-source virtual cluster-management system called Ganeti. Today, this substantial and Google-critical workload runs on GCP (Google Compute Platform).

Knowledge Base Construction in the Machine-learning Era:
Three critical design points: Joint-learning, weak supervision, and new representations

More information is accessible today than at any other time in human history. From a software perspective, however, the vast majority of this data is unusable, as it is locked away in unstructured formats such as text, PDFs, web pages, images, and other hard-to-parse formats. The goal of knowledge base construction is to extract structured information automatically from this "dark data," so that it can be used in downstream applications for search, question-answering, link prediction, visualization, modeling and much more.

The Secret Formula for Choosing the Right Next Role:
The best careers are not defined by titles or resume bullet points.

Focus on factors that will increase your career capital and make you a more valuable hire in your next role, and the one after that, and the one after that. When you are looking at the options for your next role, there are smarter choices that you can make.

The Mythos of Model Interpretability:
In machine learning, the concept of interpretability is both important and slippery.

Supervised machine-learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world?

GitOps: A Path to More Self-service IT:
IaC + PR = GitOps

GitOps lowers the bar for creating self-service versions of common IT processes, making it easier to meet the return in the ROI calculation. GitOps not only achieves this, but also encourages desired behaviors in IT systems: better testing, reduction of bus factor, reduced wait time, more infrastructure logic being handled programmatically with IaC, and directing time away from manual toil toward creating and maintaining automation.

Mind Your State for Your State of Mind:
The interactions between storage and applications can be complex and subtle.

Applications have had an interesting evolution as they have moved into the distributed and scalable world. Similarly, storage and its cousin databases have changed side by side with applications. Many times, the semantics, performance, and failure models of storage and applications do a subtle dance as they change in support of changing business requirements and environmental challenges. Adding scale to the mix has really stirred things up. This article looks at some of these issues and their impact on systems.

FPGAs in Data Centers:
FPGAs are slowly leaving the niche space they have occupied for decades.

This installment of Research for Practice features a curated selection from Gustavo Alonso, who provides an overview of recent developments utilizing FPGAs (field-programmable gate arrays) in datacenters. As Moore’s Law has slowed and the computational overheads of datacenter workloads such as model serving and data processing have continued to rise, FPGAs offer an increasingly attractive point in the trade-off between power and performance. Gustavo’s selections highlight early successes and practical deployment considerations that inform the ongoing, high-stakes debate about the future of datacenter- and cloud-based computation substrates.

Workload Frequency Scaling Law - Derivation and Verification:
Workload scalability has a cascade relation via the scale factor.

This article presents equations that relate to workload utilization scaling at a per-DVFS subsystem level. A relation between frequency, utilization, and scale factor (which itself varies with frequency) is established. The verification of these equations turns out to be tricky, since inherent to workload, the utilization also varies seemingly in an unspecified manner at the granularity of governance samples. Thus, a novel approach called histogram ridge trace is applied. Quantifying the scaling impact is critical when treating DVFS as a building block. Typical application includes DVFS governors and or other layers that influence utilization, power, and performance of the system.

Consistently Eventual:
For many data items, the work never settles on a value.

Applications are no longer islands. Not only do they frequently run distributed and replicated over many cloud-based computers, but they also run over many hand-held computers. This makes it challenging to talk about a single truth at a single place or time. In addition, most modern applications interact with other applications. These interactions settle out to impact understanding. Over time, a shared opinion emerges just as new interactions add increasing uncertainty. Many business, personal, and computational "facts" are, in fact, uncertain. As some changes settle, others meander from place to place. With all the regular, irregular, and uncleared checks, my understanding of our personal joint checking account is a bit hazy.

Algorithms Behind Modern Storage Systems:
Different uses for read-optimized B-trees and write-optimized LSM-trees

This article takes a closer look at two storage system design approaches used in a majority of modern databases (read-optimized B-trees and write-optimized LSM (log-structured merge)-trees) and describes their use cases and tradeoffs.

Every Silver Lining Has a Cloud:
Cache is king. And if your cache is cut, you’re going to feel it.

Clearly, your management has never heard the phrase, "You get what you pay for." Or perhaps they heard it and didn’t realize it applied to them. The savings in cloud computing comes at the expense of a loss of control over your systems, which is summed up best in the popular nerd sticker that says, "The Cloud is Just Other People’s Computers." Some providers now have something called Metal-as-a-Service, which I really think ought to mean that an ’80s metal band shows up at your office, plays a gig, smashes the furniture, and urinates on the carpet, but alas, it’s just the cloud providers’ way of finally admitting that cloud computing isn’t really the right answer for all applications.

C Is Not a Low-level Language:
Your computer is not a fast PDP-11.

In the wake of the recent Meltdown and Spectre vulnerabilities, it’s worth spending some time looking at root causes. Both of these vulnerabilities involved processors speculatively executing instructions past some kind of access check and allowing the attacker to observe the results via a side channel. The features that led to these vulnerabilities, along with several others, were added to let C programmers continue to believe they were programming in a low-level language, when this hasn’t been the case for decades.

Prediction-Serving Systems:
What happens when we wish to actually deploy a machine learning model to production?

This installment of Research for Practice features a curated selection from Dan Crankshaw and Joey Gonzalez, who provide an overview of machine learning serving systems. What happens when we wish to actually deploy a machine learning model to production, and how do we serve predictions with high accuracy and high computational efficiency? Dan and Joey’s selection provides a thoughtful selection of cutting-edge techniques spanning database-level integration, video processing, and prediction middleware.

Watchdogs vs. Snowflakes:
Taking wild-ass guesses

That a system can randomly jam doesn’t just indicate a serious bug in the system; it is also a major source of risk. You don’t say what your distributed job-control system controls, but let’s just say I hope it’s not something with significant, real-world side effects, like a power station, jet aircraft, or financial trading system. The risk, of course, is that the system will jam, not when it’s convenient for someone to add a dummy job to clear the jam, but during some operation that could cause data loss or return incorrect results.

Thou Shalt Not Depend on Me:
A look at JavaScript libraries in the wild

Most websites use JavaScript libraries, and many of them are known to be vulnerable. Understanding the scope of the problem, and the many unexpected ways that libraries are included, are only the first steps toward improving the situation. The goal here is that the information included in this article will help inform better tooling, development practices, and educational efforts for the community.

How to Come up with Great Ideas:
Think like an entrepreneur.

No matter what your profession, learning to think more innovatively and spark new ideas can help you. I have included some points and inspiration that have helped me, but the real key is changing your behavior and taking action.

Designing Cluster Schedulers for Internet-Scale Services:
Embracing failures for improving availability

Engineers looking to build scheduling systems should consider all failure modes of the underlying infrastructure they use and consider how operators of scheduling systems can configure remediation strategies, while aiding in keeping tenant systems as stable as possible during periods of troubleshooting by the owners of the tenant systems.

Manual Work is a Bug:
A.B.A: always be automating

Every IT team should have a culture of constant improvement - or movement along the path toward the goal of automating whatever the team feels confident in automating, in ways that are easy to change as conditions change. As the needle moves to the right, the team learns from each other’s experiences, and the system becomes easier to create and safer to operate. A good team has a structure in place that makes the process frictionless and collaborative

Canary Analysis Service:
Automated canarying quickens development, improves production safety, and helps prevent outages.

It is unreasonable to expect engineers working on product development or reliability to have statistical knowledge; removing this hurdle led to widespread CAS adoption. CAS has proven useful even for basic cases that don’t need configuration, and has significantly improved Google’s rollout reliability. Impact analysis shows that CAS has likely prevented hundreds of postmortem-worthy outages, and the rate of postmortems among groups that do not use CAS is noticeably higher.

Continuous Delivery Sounds Great, but Will It Work Here?:
It’s not magic, it just requires continuous, daily improvement at all levels.

Continuous delivery is a set of principles, patterns, and practices designed to make deployments predictable, routine affairs that can be performed on demand at any time. This article introduces continuous delivery, presents both common objections and actual obstacles to implementing it, and describes how to overcome them using real-life examples. Continuous delivery is not magic. It’s about continuous, daily improvement at all levels of the organization.

Toward a Network of Connected Things:
A look into the future of IoT deployments and their usability

While the scale of data presents new avenues for improvement, the key challenges for the everyday adoption of IoT systems revolve around managing this data. First, we need to consider where the data is being processed and stored and what the privacy and systems implications of these policies are. Second, we need to develop systems that generate actionable insights from this diverse, hard-to-interpret data for non-tech users. Solving these challenges will allow IoT systems to deliver maximum value to end users.

Containers Will Not Fix Your Broken Culture (and Other Hard Truths):
Complex socio-technical systems are hard; film at 11.

We focus so often on technical anti-patterns, neglecting similar problems inside our social structures. Spoiler alert: the solutions to many difficulties that seem technical can be found by examining our interactions with others. Let’s talk about five things you’ll want to know when working with those pesky creatures known as humans.

How Is Your Week Going So Far?:
Praise matters just as much as money.

None of us hears "thank you" or "awesome job" enough at work. Being the person who praises other people is an amazing person to be, especially when you follow this formula for making your praise ridiculously effective.

DevOps Metrics:
Your biggest mistake might be collecting the wrong data.

Delivering value to the business through software requires processes and coordination that often span multiple teams across complex systems, and involves developing and delivering software with both quality and resiliency. As practitioners and professionals, we know that software development and delivery is an increasingly difficult art and practice, and that managing and improving any process or system requires insights into that system. Therefore, measurement is paramount to creating an effective software value stream. Yet accurate measurement is no easy feat.

Popping Kernels:
Choosing between programming in the kernel or in user space

In a world in which high-performance code continues to be written in a fancy assembler, a.k.a. C, with no memory safety and plenty of other risks, the only recourse is to stick to software engineering basics. Reduce the amount of code in harm’s way, keep coupling between subsystems efficient and explicit, and work to provide better tools for the job, such as static code checkers and large suites of runtime tests.

Monitoring in a DevOps World:
Perfect should never be the enemy of better.

Monitoring can seem quite overwhelming. The most important thing to remember is that perfect should never be the enemy of better. DevOps enables highly iterative improvement within organizations. If you have no monitoring, get something; get anything. Something is better than nothing, and if you’ve embraced DevOps, you’ve already signed up for making it better over time.

Cluster Scheduling for Data Centers:
Expert-curated Guides to the Best of CS Research: Distributed Cluster Scheduling

This installment of Research for Practice features a curated selection from Malte Schwarzkopf, who takes us on a tour of distributed cluster scheduling, from research to practice, and back again. With the rise of elastic compute resources, cluster management has become an increasingly hot topic in systems R&D, and a number of competing cluster managers including Kubernetes, Mesos, and Docker are currently jockeying for the crown in this space.

Operational Excellence in April Fools’ Pranks:
Being funny is serious work.

Successful pranks require care and planning. Write a design proposal and a project plan. Involve operations early. If this is a technical change to your website, perform load testing, preferably including a "dark launch" or hidden launch test. Hide the prank behind a feature flag rather than requiring a new software release. Perform a retrospective and publish the results widely. Remember that some of the best pranks require little or no technical changes at all. For example, one could simply summarize the best practices for launching any new feature but write it under the guise of how to launch an April Fools’ prank.

Bitcoin’s Underlying Incentives:
The unseen economic forces that govern the Bitcoin protocol

Incentives are crucial for the Bitcoin protocol’s security and effectively drive its daily operation. Miners go to extreme lengths to maximize their revenue and often find creative ways to do so that are sometimes at odds with the protocol. Cryptocurrency protocols should be placed on stronger foundations of incentives. There are many areas left to improve, ranging from the very basics of mining rewards and how they interact with the consensus mechanism, through the rewards in mining pools, and all the way to the transaction fee market itself.

Reducing the Attack Surface:
Sometimes you can give the monkey a less dangerous club.

The best way to reduce the attack surface of a piece of software is to remove any unnecessary code. Since you now have two teams demanding that you leave in the code, it’s probably time to think about making two different versions of your binary. The application sounds like it’s an embedded system, so I’ll guess that it’s written in C and take it from there.

Titus: Introducing Containers to the Netflix Cloud:
Approaching container adoption in an already cloud-native infrastructure

We believe our approach has enabled Netflix to quickly adopt and benefit from containers. Though the details may be Netflix-specific, the approach of providing low-friction container adoption by integrating with existing infrastructure and working with the right early adopters can be a successful strategy for any organization looking to adopt containers.

Views from the Top:
Try to see things from a manager’s perspective.

Leadership is hard. None of us comes to work to do a bad job, and there are always ways we can be better. So, when you have a leader who isn’t meeting your expectations, maybe try reframing the situation and looking at things a little differently from the top down.

Abstracting the Geniuses Away from Failure Testing:
Ordinary users need tools that automate the selection of custom-tailored faults to inject.

This article presents a call to arms for the distributed systems research community to improve the state of the art in fault tolerance testing. Ordinary users need tools that automate the selection of custom-tailored faults to inject. We conjecture that the process by which superusers select experiments can be effectively modeled in software. The article describes a prototype validating this conjecture, presents early results from the lab and the field, and identifies new research directions that can make this vision a reality.

Private Online Communication; Highlights in Systems Verification:
The importance of private communication will continue to grow. We need techniques to build larger verified systems from verified components.

First, Albert Kwon provides an overview of recent systems for secure and private communication. Second, James Wilcox takes us on a tour of recent advances in verified systems design.

Network Applications Are Interactive:
The network era requires new models, with interactions instead of algorithms.

The miniaturization of devices and the prolific interconnectedness of these devices over high-speed wireless networks is completely changing how commerce is conducted. These changes (a.k.a. digital) will profoundly change how enterprises operate. Software is at the heart of this digital world, but the software toolsets and languages were conceived for the host-based era. The issues that already plague software practice (such as high defects, poor software productivity, information vulnerability, poor software project success rates, etc.) will be more profound with such an approach. It is time for software to be made simpler, secure, and reliable.

XML and JSON Are Like Cardboard:
Cardboard surrounds and protects stuff as it crosses boundaries.

In cardboard, the safety and care for stuff is the important reason for its existence. Similarly, in XML and JSON the safety and care of the data, both in transit and in storage, are why we bother.

Breadth and Depth:
We all wear many hats, but make sure you have one that fits well.

When people ask me the question of where they should focus their time I ask them what is the one thing you could be the best in the world at? The answer might be going deep or going wide. The important thing is to spend your time on building the skills that will move you to where to you want to go.

Cache Me If You Can:
Building a decentralized web-delivery model

The world is more connected than it ever has been before, and with our pocket supercomputers and IoT (Internet of Things) future, the next generation of the web might just be delivered in a peer-to-peer model. It’s a giant problem space, but the necessary tools and technology are here today. We just need to define the problem a little better.

Bitcoin’s Academic Pedigree:
The concept of cryptocurrencies is built from forgotten ideas in research literature.

We’ve seen repeatedly that ideas in the research literature can be gradually forgotten or lie unappreciated, especially if they are ahead of their time, even in popular areas of research. Both practitioners and academics would do well to revisit old ideas to glean insights for present systems. Bitcoin was unusual and successful not because it was on the cutting edge of research on any of its components, but because it combined old ideas from many previously unrelated fields. This is not easy to do, as it requires bridging disparate terminology, assumptions, etc., but it is a valuable blueprint for innovation.

Cold, Hard Cache:
On the implementation and maintenance of caches

Dear KV, Our latest project at work requires a large number of slightly different software stacks to deploy within our cloud infrastructure. With modern hardware, I can test this deployment on a laptop. The problem I keep running up against is that our deployment system seems to secretly cache some of my files and settings and not clear them, even when I repeatedly issue the command to do so. I’ve resorted to repeatedly using the find command so that I can blow away the offending files. What I’ve found is that the system caches data in many places so I’ve started a list.

Vigorous Public Debates in Academic Computer Science:
Expert-curated Guides to the Best of CS Research

This installment of Research for Practice features a special curated selection from John Regehr, who takes us on a tour of great debates in academic computer science research. In case you thought flame wars were reserved for Usenet mailing lists and Twitter, think again: the academic literature is full of dramatic, spectacular, and vigorous debates spanning file systems, operating system kernel design, and formal verification.

Hootsuite: In Pursuit of Reactive Systems:
A discussion with Edward Steel, Yanik Berube, Jonas Bonér, Ken Britton, and Terry Coatta

It has become apparent how critical frameworks and standards are for development teams when using microservices. People often mistake the flexibility microservices provide with a requirement to use different technologies for each service. Like all development teams, we still need to keep the number of technologies we use to a minimum so we can easily train new people, maintain our code, support moves between teams, and the like.

Four Ways to Make CS & IT Curricula More Immersive:
Why the Bell Curve Hasn’t Transformed into a Hockey Stick

Our first experiences cement what becomes normal for us. Students should start off seeing a well-run system, dissect it, learn its parts, progressively dig down into the details. Don’t let them see what a badly run system looks like until they have experienced one that is well run. A badly run system should then disgust them.

Metaphors We Compute By:
Code is a story that explains how to solve a particular problem.

Programmers must be able to tell a story with their code, explaining how they solved a particular problem. Like writers, programmers must know their metaphors. Many metaphors will be able to explain a concept, but you must have enough skill to choose the right one that’s able to convey your ideas to future programmers who will read the code. Thus, you cannot use every metaphor you know. You must master the art of metaphor selection, of meaning amplification. You must know when to add and when to subtract. You will learn to revise and rewrite code as a writer does. Once there’s nothing else to add or remove, you have finished your work.

10 Ways to Be a Better Interviewer:
Plan ahead to make the interview a successful one.

Of course, there is no right way to do an interview, but you can always be better. Make an effort to make your candidates as comfortable as possible so they have the greatest chance for success.

Is There a Single Method for the Internet of Things?:
Essence can keep software development for the IoT from becoming unwieldy.

The Industrial Internet Consortium predicts the IoT (Internet of Things) will become the third technological revolution after the Industrial Revolution and the Internet Revolution. Its impact across all industries and businesses can hardly be imagined. Existing software (business, telecom, aerospace, defense, etc.) is expected to be modified or redesigned, and a huge amount of new software, solving new problems, will have to be developed. As a consequence, the software industry should welcome new and better methods.

IoT: The Internet of Terror:
If it seems like the sky is falling, that’s because it is.

It is true that many security-focused engineers can sound like Chicken Little, running around announcing that the sky is falling, but, unless you’ve been living under a rock, you will notice that, indeed, the sky IS falling. Not a day goes by without a significant attack against networked systems making the news, and the Internet of Terror is leading the charge in taking distributed systems down the road to hell - a road that you wish to pave with your good intentions.

Research for Practice: Technology for UnderservedCommunities; Personal Fabrication:
Expert-curated Guides to the Best of CS Research

This installment of Research for Practice provides curated reading guides to technology for underserved communities and to new developments in personal fabrication. First, Tawanna Dillahunt describes design considerations and technology for underserved and impoverished communities. Designing for the more than 1.6 billion impoverished individuals worldwide requires special consideration of community needs, constraints, and context. Tawanna’s selections span protocols for poor-quality communication networks, community-driven content generation, and resource and public service discovery. Second, Stefanie Mueller and Patrick Baudisch provide an overview of recent advances in personal fabrication (e.g., 3D printers). Their selection covers new techniques for fabricating (and emulating) complex materials (e.g., by manipulating the internal structure of an object), for more easily specifying object shape and behavior, and for human-in-the-loop rapid prototyping.

Data Sketching:
The approximate approach is often faster and more efficient.

Do you ever feel overwhelmed by an unending stream of information? It can seem like a barrage of new email and text messages demands constant attention, and there are also phone calls to pick up, articles to read, and knocks on the door to answer. Putting these pieces together to keep track of what’s important can be a real challenge. In response to this challenge, the model of streaming data processing has grown in popularity. The aim is no longer to capture, store, and index every minute event, but rather to process each observation quickly in order to create a summary of the current state.

Side Effects, Front and Center!:
One System’s Side Effect is Another’s Meat and Potatoes.

We think of computation in terms of its consequences. The big MapReduce job returns a large result. Web interactions display information. Enterprise applications update the database and return an answer. These are the reasons we do our work. What we rarely discuss are the side effects of doing the work we intend. Side effects may be unwanted, or they may actually cause desired behavior at different layers of the system. This column points out some fun patterns to keep in mind as we build and use our systems.

The Calculus of Service Availability:
You’re only as available as the sum of your dependencies.

Most services offered by Google aim to offer 99.99 percent (sometimes referred to as the "four 9s") availability to users. Some services contractually commit to a lower figure externally but set a 99.99 percent target internally. This more stringent target accounts for situations in which users become unhappy with service performance well before a contract violation occurs, as the number one aim of an SRE team is to keep users happy. For many services, a 99.99 percent internal target represents the sweet spot that balances cost, complexity, and availability.

Conversations with Technology Leaders: Erik Meijer:
Great engineers are able to maximize their mental power.

Whether you are a leader, a programmer, or just someone aspiring to be better, I am sure there are some smart takeaways from our conversation that will help you grow in your role. Oh, and if you read to the end, you can find out what his favorite job interview question is - and see if you would be able to pass his test.

The IDAR Graph:
An improvement over UML

UML is the de facto standard for representing object-oriented designs. It does a fine job of recording designs, but it has a severe problem: its diagrams don’t convey what humans need to know, making them hard to understand. This is why most software developers use UML only when forced to. People understand an organization, such as a corporation, in terms of a control hierarchy. When faced with an organization of people or objects, the first question usually is, "What’s controlling all this?" Surprisingly, UML has no concept of one object controlling another. Consequently, in every type of UML diagram, no object appears to have greater or lesser control than its neighbors.

The Observer Effect:
Finding the balance between zero and maximum

The problem is a failure to appreciate just what you are asking a system to do when polling it for information. Modern systems contain thousands of values that can be measured and recorded. Blindly retrieving whatever it is that might be exposed by the system is bad enough, but asking for it with a high-frequency poll is much worse.

Too Big NOT to Fail:
Embrace failure so it doesn’t embrace you.

Web-scale infrastructure implies LOTS of servers working together, often tens or hundreds of thousands of servers all working toward the same goal. How can the complexity of these environments be managed? How can commonality and simplicity be introduced?

Research for Practice: Tracing and Debugging Distributed Systems; Programming by Examples:
Expert-curated Guides to the Best of CS Research

This installment of Research for Practice covers two exciting topics in distributed systems and programming methodology. First, Peter Alvaro takes us on a tour of recent techniques for debugging some of the largest and most complex systems in the world: modern distributed systems and service-oriented architectures. The techniques Peter surveys can shed light on order amid the chaos of distributed call graphs. Second, Sumit Gulwani illustrates how to program without explicitly writing programs, instead synthesizing programs from examples! The techniques Sumit presents allow systems to "learn" a program representation from illustrative examples, allowing nonprogrammer users to create increasingly nontrivial functions such as spreadsheet macros.

The Debugging Mindset:
Understanding the psychology of learning strategies leads to effective problem-solving skills.

Software developers spend 35-50 percent of their time validating and debugging software. The cost of debugging, testing, and verification is estimated to account for 50-75 percent of the total budget of software development projects, amounting to more than $100 billion annually. While tools, languages, and environments have reduced the time spent on individual debugging tasks, they have not significantly reduced the total time spent debugging, nor the cost of doing so. Therefore, a hyperfocus on elimination of bugs during development is counterproductive; programmers should instead embrace debugging as an exercise in problem solving.

Forced Exception-Handling:
You can never discount the human element in programming.

Yes, KV also reads "The Morning Paper," although he has to admit that he does not read everything that arrives in his inbox from that list. Of course, the paper you mention piqued my interest, and one of the things you don’t point out is that it’s actually a study of distributed systems failures. Now, how can we make programming harder? I know! Let’s take a problem on a single system and distribute it. Someday I would like to see a paper that tells us if problems in distributed systems increase along with the number of nodes, or the number of interconnections.

MongoDB’s JavaScript Fuzzer:
The fuzzer is for those edge cases that your testing didn’t catch.

As MongoDB becomes more feature-rich and complex with time, the need to develop more sophisticated methods for finding bugs grows as well. Three years ago, MongDB added a home-grown JavaScript fuzzer to its toolkit, and it is now our most prolific bug-finding tool, responsible for detecting almost 200 bugs over the course of two release cycles. These bugs span a range of MongoDB components from sharding to the storage engine, with symptoms ranging from deadlocks to data inconsistency. The fuzzer runs as part of the CI (continuous integration) system, where it frequently catches bugs in newly committed code.

Does Anybody Listen to You?:
How do you step up from mere contributor to real change-maker?

An idea on its own is not worth much. Just because you think you know a better way to do something, even if you’re right, no one is required to care. Making great things happen at work is about more than just being smart. Good ideas succeed or fail depending on your ability to communicate them correctly to the people who have the power to make them happen. When you are navigating an organization, it pays to know whom to talk to and how to reach them. Here is a simple guide to sending your ideas up the chain and actually making them stick.

Making Money Using Math:
Modern applications are increasingly using probabilistic machine-learned models.

A big difference between human-written code and learned models is that the latter are usually not represented by text and hence are not understandable by human developers or manipulable by existing tools. The consequence is that none of the traditional software engineering techniques for conventional programs (such as code reviews, source control, and debugging) are applicable anymore. Since incomprehensibility is not unique to learned code, these aspects are not of concern here.

Pervasive, Dynamic Authentication of Physical Items:
The use of silicon PUF circuits

Authentication of physical items is an age-old problem. Common approaches include the use of bar codes, QR codes, holograms, and RFID (radio-frequency identification) tags. Traditional RFID tags and bar codes use a public identifier as a means of authenticating. A public identifier, however, is static: it is the same each time when queried and can be easily copied by an adversary. Holograms can also be viewed as public identifiers: a knowledgeable verifier knows all the attributes to inspect visually. It is difficult to make hologram-based authentication pervasive; a casual verifier does not know all the attributes to look for.

Research for Practice: Cryptocurrencies, Blockchains, and Smart Contracts; Hardware for Deep Learning:
Expert-curated Guides to the Best of CS Research

First, Arvind Narayanan and Andrew Miller, co-authors of the increasingly popular open-access Princeton Bitcoin textbook, provide an overview of ongoing research in cryptocurrencies. Second, Song Han provides an overview of hardware trends related to another long-studied academic problem that has recently seen an explosion in popularity: deep learning.

Uninitialized Reads:
Understanding the proposed revisions to the C language

Most developers understand that reading uninitialized variables in C is a defect, but some do it anyway. What happens when you read uninitialized objects is unsettled in the current version of the C standard (C11).3 Various proposals have been made to resolve these issues in the planned C2X revision of the standard. Consequently, this is a good time to understand existing behaviors as well as proposed revisions to the standard to influence the evolution of the C language. Given that the behavior of uninitialized reads is unsettled in C11, prudence dictates eliminating uninitialized reads from your code.

Heterogeneous Computing: Here to Stay:
Hardware and Software Perspectives

Mentions of the buzzword heterogeneous computing have been on the rise in the past few years and will continue to be heard for years to come, because heterogeneous computing is here to stay. What is heterogeneous computing, and why is it becoming the norm? How do we deal with it, from both the software side and the hardware side? This article provides answers to some of these questions and presents different points of view on others.

Time, but Faster:
A computing adventure about time through the looking glass

The first premise was summed up perfectly by the late Douglas Adams in The Hitchhiker’s Guide to the Galaxy: "Time is an illusion. Lunchtime doubly so." The concept of time, when colliding with decoupled networks of computers that run at billions of operations per second, is... well, the truth of the matter is that you simply never really know what time it is. That is why Leslie Lamport’s seminal paper on Lamport timestamps was so important to the industry, but this article is actually about wall-clock time, or a reasonably useful estimation of it.

The Chess Player who Couldn’t Pass the Salt:
AI: Soft and hard, weak and strong, narrow and general

The problem inherent in almost all nonspecialist work in AI is that humans actually don’t understand intelligence very well in the first place. Now, computer scientists often think they understand intelligence because they have so often been the "smart" kid, but that’s got very little to do with understanding what intelligence actually is. In the absence of a clear understanding of how the human brain generates and evaluates ideas, which may or may not be a good basis for the concept of intelligence, we have introduced numerous proxies for intelligence, the first of which is game-playing behavior.

Are You Load Balancing Wrong?:
Anyone can use a load balancer. Using them properly is much more difficult.

A reader contacted me recently to ask if it is better to use a load balancer to add capacity or to make a service more resilient to failure. The answer is: both are appropriate uses of a load balancer. The problem, however, is that most people who use load balancers are doing it wrong.

Life Beyond Distributed Transactions:
An apostate’s opinion

This article explores and names some of the practical approaches used in the implementation of large-scale mission-critical applications in a world that rejects distributed transactions. Topics include the management of fine-grained pieces of application data that may be repartitioned over time as the application grows. Design patterns support sending messages between these repartitionable pieces of data.

Research for Practice: Distributed Transactions and Networks as Physical Sensors:
Expert-curated Guides to the Best of CS Research

First, Irene Zhang delivers a whirlwind tour of recent developments in distributed concurrency control. If you thought distributed transactions were prohibitively expensive, Irene’s selections may prompt you to reconsider: the use of atomic clocks, clever replication protocols, and new means of commit ordering all improve performance at scale. Second, Fadel Adib provides a fascinating look at using computer networks as physical sensors. It turns out that the radio waves passing through our environment and bodies are subtly modulated as they do so.

BBR: Congestion-Based Congestion Control:
Measuring bottleneck bandwidth and round-trip propagation time

When bottleneck buffers are large, loss-based congestion control keeps them full, causing bufferbloat. When bottleneck buffers are small, loss-based congestion control misinterprets loss as a signal of congestion, leading to low throughput. Fixing these problems requires an alternative to loss-based congestion control. Finding this alternative requires an understanding of where and how network congestion originates.

Resolving Conflict:
Don’t "win." Resolve.

I am conflicted about conflict. On one hand, I hate it. Hearing people disagree, even about minor things, makes me want to run through the nearest wall and curl up under my bed until it’s over. On the other hand, when it happens, I always want to get into it.

Faucet: Deploying SDN in the Enterprise:
Using OpenFlow and DevOps for rapid development

While SDN as a technology continues to evolve and become even more programmable, Faucet and OpenFlow 1.3 hardware together are sufficient to realize benefits today. This article describes specifically how to take advantage of DevOps practices to develop and deploy features rapidly. It also describes several practical deployment scenarios, including firewalling and network function virtualization.

The Unholy Trinity of Software Development:
Tests, documentation, and code

Questions like this bring to mind the fact that source code, documentation, and testing are the unholy trinity of software development, although many organizations like to see them as separate entities. It is interesting that while many groups pay lip service to "test-driven development," they do not include documentation in TDD.

Industrial Scale Agile - from Craft to Engineering:
Essence is instrumental in moving software development toward a true engineering discipline.

There are many, many ways to illustrate how fragile IT investments can be. You just have to look at the way that, even after huge investments in education and coaching, many organizations are struggling to broaden their agile adoption to the whole of their organization - or at the way other organizations are struggling to maintain the momentum of their agile adoptions as their teams change and their systems mature.

Research for Practice: Web Security and Mobile Web Computing:
Expert-curated Guides to the Best of CS Research

Our third installment of Research for Practice brings readings spanning programming languages, compilers, privacy, and the mobile web.

The Power of Babble:
Expect to be constantly and pleasantly befuddled

Metadata defines the shape, the form, and how to understand our data. It is following the trend taken by natural languages in our increasingly interconnected world. While many concepts can be communicated using shared metadata, no one can keep up with the number of disparate new concepts needed to have a common understanding.

Functional at Scale:
Applying functional programming principles to distributed computing projects

Modern server software is demanding to develop and operate: it must be available at all times and in all locations; it must reply within milliseconds to user requests; it must respond quickly to capacity demands; it must process a lot of data and even more traffic; it must adapt quickly to changing product needs; and in many cases it must accommodate a large engineering organization, its many engineers the proverbial cooks in a big, messy kitchen.

Fresh Starts:
Just because you have been doing it the same way doesn’t mean you are doing it the right way.

I love fresh starts. Growing up, one of my favorite things was starting a new school year. From the fresh school supplies to the promise of a new class of students, teachers, and lessons, I couldn’t wait for summer to be over and to go back to school. The same thing happens with new jobs. They reinvigorate you, excite you, and get you going.

React: Facebook’s Functional Turn on Writing JavaScript:
A discussion with Pete Hunt, Paul O’Shannessy, Dave Smith and Terry Coatta

One of the long-standing ironies of user-friendly JavaScript front ends is that building them typically involved trudging through the DOM (Document Object Model), hardly known for its friendliness to developers. But now developers have a way to avoid directly interacting with the DOM, thanks to Facebook’s decision to open-source its React library for the construction of user interface components.

Cloud Calipers:
Naming the next generation and remembering that the cloud is just other people’s computers

For the time being, we are likely to continue to have programmers who version their functions as a result of the limitations of their languages, but let’s hope we can stop them naming their next generations after the next generation.

Scaling Synchronization in Multicore Programs:
Advanced synchronization methods can boost the performance of multicore software.

Designing software for modern multicore processors poses a dilemma. Traditional software designs, in which threads manipulate shared data, have limited scalability because synchronization of updates to shared data serializes threads and limits parallelism. Alternative distributed software designs, in which threads do not share mutable data, eliminate synchronization and offer better scalability. But distributed designs make it challenging to implement features that shared data structures naturally provide, such as dynamic load balancing and strong consistency guarantees, and are simply not a good fit for every program. Often, however, the performance of shared mutable data structures is limited by the synchronization methods in use today, whether lock-based or lock-free.

10 Optimizations on Linear Search:
The operations side of the story

System administrators (DevOps engineers or SREs or whatever your title) must deal with the operational aspects of computation, not just the theoretical aspects. Operations is where the rubber hits the road. As a result, operations people see things from a different perspective and can realize opportunities outside of the basic O() analysis. Let’s look at the operational aspects of the problem of trying to improve something that is theoretically optimal already.

The Singular Success of SQL:
SQL has a brilliant future as a major figure in the pantheon of data representations.

SQL has a brilliant past and a brilliant future. That future is not as the singular and ubiquitous holder of data but rather as a major figure in the pantheon of data representations. What the heck happens when data is not kept in SQL?

Idle-Time Garbage-Collection Scheduling:
Taking advantage of idleness to reduce dropped frames and memory consumption

Google’s Chrome web browser strives to deliver a smooth user experience. An animation will update the screen at 60 FPS (frames per second), giving Chrome around 16.6 milliseconds to perform the update. Within these 16.6 ms, all input events have to be processed, all animations have to be performed, and finally the frame has to be rendered. A missed deadline will result in dropped frames. These are visible to the user and degrade the user experience. Such sporadic animation artifacts are referred to here as jank. This article describes an approach implemented in the JavaScript engine V8, used by Chrome, to schedule garbage-collection pauses during times when Chrome is idle.

Bad Software Architecture is a People Problem:
When people don’t work well together they make bad decisions.

It all started with a bug. Customers were complaining that their information was out of date on the website. They would make an update and for some reason their changes weren’t being reflected. Caching seemed like the obvious problem, but once we started diving into the details, we realized it was a much bigger issue.

Dynamics of Change: Why Reactivity Matters:
Tame the dynamics of change by centralizing each concern in its own module.

Professional programming is about dealing with software at scale. Everything is trivial when the problem is small and contained: it can be elegantly solved with imperative programming or functional programming or any other paradigm. Real-world challenges arise when programmers have to deal with large amounts of data, network requests, or intertwined entities, as in UI (user interface) programming.

Research for Practice: Distributed Consensus and Implications of NVM on Database Management Systems:
Expert-curated Guides to the Best of CS Research

First, how do large-scale distributed systems mediate access to shared resources, coordinate updates to mutable state, and reliably make decisions in the presence of failures? Second, while consensus concerns distributed shared state, our second selection concerns the impact of hardware trends on single-node shared state.

Cluster-level Logging of Containers with Containers:
Logging Challenges of Container-Based Cloud Deployments

This article shows how cluster-level logging infrastructure can be implemented using open source tools and deployed using the very same abstractions that are used to compose and manage the software systems being logged. Collecting and analyzing log information is an essential aspect of running production systems to ensure their reliability and to provide important auditing information. Many tools have been developed to help with the aggregation and collection of logs for specific software components (e.g., an Apache web server) running on specific servers (e.g., Fluentd and Logstash.)

Chilling the Messenger:
Keeping ego out of software-design review

Trying to correct someone who has just done a lot of work, even if, ultimately, that work is not the right work, is a daunting task. The person in question no doubt believes that he has worked very hard to produce something of value to the rest of the team, and walking in and spitting on it, literally or metaphorically, probably crosses your "offense" line--at least I think it does. I’m a bit surprised that since this is the first sprint and there is already so much code written, shouldn’t the software have shown up after the sprints established what was needed, who the stakeholders were, etc.?

The Hidden Dividends of Microservices:
Microservices aren’t for every company, and the journey isn’t easy.

Microservices are an approach to building distributed systems in which services are exposed only through hardened APIs; the services themselves have a high degree of internal cohesion around a specific and well-bounded context or area of responsibility, and the coupling between them is loose. Such services are typically simple, yet they can be composed into very rich and elaborate applications. The effort required to adopt a microservices-based approach is considerable, particularly in cases that involve migration from more monolithic architectures. The explicit benefits of microservices are well known and numerous, however, and can include increased agility, resilience, scalability, and developer productivity.

Standing on Distributed Shoulders of Giants:
Farsighted Physicists of Yore Were Danged Smart!

If you squint hard enough, many of the challenges of distributed computing appear similar to the work done by the great physicists. Dang, those fellows were smart! Here, we examine some of the most important physics breakthroughs and draw some whimsical parallels to phenomena in the world of computing... just for fun.

Introducing Research for Practice:
Expert-curated guides to the best of CS research

Reading a great research paper is a joy. A team of experts deftly guides you, the reader, through the often complicated research landscape, noting the prior art, the current trends, the pressing issues at hand--and then, sometimes artfully, sometimes through seeming sheer force of will, expands the body of knowledge in a fell swoop of 12 or so pages of prose. A great paper contains a puzzle and a solution; these can be useful, enlightening, or both.

The Small Batches Principle:
Reducing waste, encouraging experimentation, and making everyone happy

The small batches principle is part of the DevOps methodology. It comes from the lean manufacturing movement, which is often called just-in-time manufacturing. It can be applied to just about any kind of process. It also enables the MVP (minimum viable product) methodology, which involves launching a small version of a service to get early feedback that informs the decisions made later in the project.

Debugging Distributed Systems:
Challenges and options for validation and debugging

Distributed systems pose unique challenges for software developers. Reasoning about concurrent activities of system nodes and even understanding the system’s communication topology can be difficult. A standard approach to gaining insight into system activity is to analyze system logs. Unfortunately, this can be a tedious and complex process. This article looks at several key features and debugging challenges that differentiate distributed systems from other kinds of software. The article presents several promising tools and ongoing research to help resolve these challenges.

Nine Things I Didn’t Know I Would Learn Being an Engineer Manager:
Many of the skills aren’t technical at all.

When I moved from being an engineer to being a dev lead, I knew I had a lot to learn. My initial thinking was that I had to be able to do thorough code reviews, design and architect websites, see problems before they happened, and ask insightful technical questions. To me that meant learning the technology and becoming a better engineer. When I actually got into the role (and after doing it almost 15 years), the things I have learned--and that have mattered the most--weren’t those technical details.

Should You Upload or Ship Big Data to the Cloud?:
The accepted wisdom does not always hold true.

It is accepted wisdom that when the data you wish to move into the cloud is at terabyte scale and beyond, you are better off shipping it to the cloud provider, rather than uploading it. This article takes an analytical look at how shipping and uploading strategies compare, the various factors on which they depend, and under what circumstances you are better off shipping rather than uploading data, and vice versa. Such an analytical determination is important to make, given the increasing availability of gigabit-speed Internet connections, along with the explosive growth in data-transfer speeds supported by newer editions of drive interfaces such as SAS and PCI Express.

What Are You Trying to Pull?:
A single cache miss is more expensive than many instructions.

Saving instructions - how very 1990s of him. It’s always nice when people pay attention to details, but sometimes they simply don’t pay attention to the right ones. While KV would never encourage developers to waste instructions, given the state of modern software, it does seem like someone already has. KV would, as you did, come out on the side of legibility over the saving of a few instructions.

The Flame Graph:
This visualization of software execution is a new necessity for performance profiling and debugging.

An everyday problem in our industry is understanding how software is consuming resources, particularly CPUs. What exactly is consuming how much, and how did this change since the last software version? These questions can be answered using software profilers, tools that help direct developers to optimize their code and operators to tune their environment. The output of profilers can be verbose, however, making it laborious to study and comprehend. The flame graph provides a new visualization for profiler output and can make for much faster comprehension, reducing the time for root cause analysis.

Delegation as Art:
Be someone who makes everyone else better.

When I started my career as a junior engineer, I couldn’t wait to be senior. I would regularly review our promotion guidelines and assess my progress and contributions against them. Of course, at the time I didn’t really understand what being senior meant. Being a senior engineer means having strong technical skills, the ability to communicate well and navigate ambiguous situations, and most important of all, the ability to grow and lead other people. Leadership isn’t just for managers anymore.

Why Logical Clocks are Easy:
Sometimes all you need is the right language.

Any computing system can be described as executing sequences of actions, with an action being any relevant change in the state of the system. For example, reading a file to memory, modifying the contents of the file in memory, or writing the new contents to the file are relevant actions for a text editor. In a distributed system, actions execute in multiple locations; in this context, actions are often called events. Examples of events in distributed systems include sending or receiving messages, or changing some state in a node. Not all events are related, but some events can cause and influence how other, later events occur.

Use-Case 2.0:
The Hub of Software Development

Use cases have been around for almost 30 years as a requirements approach and have been part of the inspiration for more-recent techniques such as user stories. Now the inspiration has flown in the other direction. Use-Case 2.0 is the new generation of use-case-driven development-light, agile, and lean-inspired by user stories and the agile methodologies Scrum and Kanban.

GNL is Not Linux:
What’s in a Name?

What, indeed, is in a name? As you’ve already seen, this quasi-technical topic continues to cause a bit of heat in the software community, particularly in the open-source world. You can find the narrative from the GNU side by clicking on the link provided in the postscript to this article, but KV finds that narrative lacking, and so, against my better judgment about pigs and dancing, I will weigh in with a few comments.

More Encryption Means Less Privacy:
Retaining electronic privacy requires more political engagement.

When Edward Snowden made it known to the world that pretty much all traffic on the Internet was collected and searched by the NSA, GCHQ (the UK Government Communications Headquarters) and various other countries’ secret services as well, the IT and networking communities were furious and felt betrayed.

Statistics for Engineers:
Applying statistical techniques to operations data

Modern IT systems collect an increasing wealth of data from network gear, operating systems, applications, and other components. This data needs to be analyzed to derive vital information about the user experience and business performance. For instance, faults need to be detected, service quality needs to be measured and resource usage of the next days and month needs to be forecast.

Borg, Omega, and Kubernetes:
Lessons learned from three container-management systems over a decade

Though widespread interest in software containers is a relatively recent phenomenon, at Google we have been managing Linux containers at scale for more than ten years and built three different container-management systems in that time. Each system was heavily influenced by its predecessors, even though they were developed for different reasons. This article describes the lessons we’ve learned from developing and operating them.

Code Hoarding:
Committing to commits, and the beauty of summarizing graphs

Dear KV, Why are so many useful features of open-source projects hidden under obscure configuration options that mean they’ll get little or no use? Is this just typically poor documentation and promotion, or is there something that makes these developers hide their code? It’s not as if the code seems broken. When I turned these features on in some recent code I came across, the system remained stable under test and in production. I feel that code should either be used or removed from the system. If the code is in a source-code repository, then it’s not really lost, but it’s also not cluttering the rest of the system.

The Paradox of Autonomy and Recognition:
Thoughts on trust and merit in software team culture

Who doesn’t want recognition for their hard work and contributions? Early in my career I wanted to believe that if you worked hard, and added value, you would be rewarded. I wanted to believe in the utopian ideal that hard work, discipline, and contributions were the fuel that propelled you up the corporate ladder. Boy, was I wrong.

How Sysadmins Devalue Themselves:
And how to track on-call coverage

Q: Dear Tom, How can I devalue my work? Lately I’ve felt like everyone appreciates me, and, in fact, I’m overpaid and underutilized. Could you help me devalue myself at work? A: Dear Reader, Absolutely! I know what a pain it is to lug home those big paychecks. It’s so distracting to have people constantly patting you on the back. Ouch! Plus, popularity leads to dates with famous musicians and movie stars. (Just ask someone like Taylor Swift or Leonardo DiCaprio.) Who wants that kind of distraction when there’s a perfectly good video game to be played?

The Verification of a Distributed System:
A practitioner’s guide to increasing confidence in system correctness

Leslie Lamport, known for his seminal work in distributed systems, famously said, "A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable." Given this bleak outlook and the large set of possible failures, how do you even begin to verify and validate that the distributed systems you build are doing the right thing?

Accountability in Algorithmic Decision-making:
A view from computational journalism

Every fiscal quarter automated writing algorithms churn out thousands of corporate earnings articles for the AP (Associated Press) based on little more than structured data. Companies such as Automated Insights, which produces the articles for AP, and Narrative Science can now write straight news articles in almost any domain that has clean and well-structured data: finance, sure, but also sports, weather, and education, among others. The articles aren’t cardboard either; they have variability, tone, and style, and in some cases readers even have difficulty distinguishing the machine-produced articles from human-written ones.

Immutability Changes Everything:
We need it, we can afford it, and the time is now.

There is an inexorable trend toward storing and sending immutable data. We need immutability to coordinate at a distance, and we can afford immutability as storage gets cheaper. This article is an amuse-bouche sampling the repeated patterns of computing that leverage immutability. Climbing up and down the compute stack really does yield a sense of déjà vu all over again.

Time is an Illusion.:
Lunchtime doubly so. - Ford Prefect to Arthur Dent in "The Hitchhiker’s Guide to the Galaxy", by Douglas Adams

One of the more surprising things about digital systems - and, in particular, modern computers - is how poorly they keep time. When most programs ran on a single system this was not a significant issue for the majority of software developers, but once software moved into the distributed-systems realm this inaccuracy became a significant challenge.

Non-volatile Storage:
Implications of the Datacenter’s Shifting Center

For the entire careers of most practicing computer scientists, a fundamental observation has consistently held true: CPUs are significantly more performant and more expensive than I/O devices. The fact that CPUs can process data at extremely high rates, while simultaneously servicing multiple I/O devices, has had a sweeping impact on the design of both hardware and software for systems of all sizes, for pretty much as long as we’ve been building them.

Schema.org: Evolution of Structured Data on the Web:
Big data makes common schemas even more necessary.

Separation between content and presentation has always been one of the important design aspects of the Web. Historically, however, even though most Web sites were driven off structured databases, they published their content purely in HTML. Services such as Web search, price comparison, reservation engines, etc. that operated on this content had access only to HTML. Applications requiring access to the structured data underlying these Web pages had to build custom extractors to convert plain HTML into structured data. These efforts were often laborious and the scrapers were fragile and error-prone, breaking every time a site changed its layout.

A Purpose-built Global Network: Google’s Move to SDN:
A discussion with Amin Vahdat, David Clark, and Jennifer Rexford

Everything about Google is at scale, of course -- a market cap of legendary proportions, an unrivaled talent pool, enough intellectual property to keep armies of attorneys in Guccis for life, and, oh yeah, a private WAN (wide area network) bigger than you can possibly imagine that also happens to be growing substantially faster than the Internet as a whole.

Pickled Patches:
On repositories of patches and tension between security professionals and in-house developers

I recently came upon a software repository that was not a repo of code, but a repo of patches. The project seemed to build itself out of several other components and then had complicated scripts that applied the patches in a particular order. I had to look at this repo because I wanted to fix a bug in the system, but trying to figure out what the code actually looked like at any particular point in time was baffling. Are there tools that would help in working like this?

It Probably Works:
Probabilistic algorithms are all around us--not only are they acceptable, but some programmers actually seek out chances to use them.

Probabilistic algorithms exist to solve problems that are either impossible or unrealistic (too expensive, too time-consuming, etc.) to solve precisely. In an ideal world, you would never actually need to use probabilistic algorithms. To programmers who are not familiar with them, the idea can be positively nervewracking: "How do I know that it will actually work? What if it’s inexplicably wrong? How can I debug it? Maybe we should just punt on this problem, or buy a whole lot more servers..."

Challenges of Memory Management on Modern NUMA System:
Optimizing NUMA systems applications with Carrefour

Modern server-class systems are typically built as several multicore chips put together in a single system. Each chip has a local DRAM (dynamic random-access memory) module; together they are referred to as a node. Nodes are connected via a high-speed interconnect, and the system is fully coherent. This means that, transparently to the programmer, a core can issue requests to its node’s local memory as well as to the memories of other nodes. The key distinction is that remote requests will take longer, because they are subject to longer wire delays and may have to jump several hops as they traverse the interconnect.

Componentizing the Web:
We may be on the cusp of a new revolution in web development.

There is no task in software engineering today quite as herculean as web development. A typical specification for a web application might read: The app must work across a wide variety of browsers. It must run animations at 60 fps. It must be immediately responsive to touch. It must conform to a specific set of design principles and specs. It must work on just about every screen size imaginable, from TVs and 30-inch monitors to mobile phones and watch faces. It must be well-engineered and maintainable in the long term.

Automation Should Be Like Iron Man, Not Ultron:
The "Leftover Principle" Requires Increasingly More Highly-skilled Humans.

A few years ago we automated a major process in our system administration team. Now the system is impossible to debug. Nobody remembers the old manual process and the automation is beyond what any of us can understand. We feel like we’ve painted ourselves into a corner. Is all operations automation doomed to be this way?

Lean Software Development - Building and Shipping Two Versions:
Catering to developers’ strengths while still meeting team objectives

Once I was managing a software team and we were working on several initiatives. Projects were assigned based on who was available, their skillsets, and their development goals. This resulted in two developers, Mary and Melissa, being assigned to the same project.

Fail at Scale:
Reliability in the face of rapid change

Failure is part of engineering any large-scale system. One of Facebook’s cultural values is embracing failure. This can be seen in the posters hung around the walls of our Menlo Park headquarters: "What Would You Do If You Weren’t Afraid?" and "Fortune Favors the Bold."

How to De-identify Your Data:
Balancing statistical accuracy and subject privacy in large social-science data sets

Big data is all the rage; using large data sets promises to give us new insights into questions that have been difficult or impossible to answer in the past. This is especially true in fields such as medicine and the social sciences, where large amounts of data can be gathered and mined to find insightful relationships among variables. Data in such fields involves humans, however, and thus raises issues of privacy that are not faced by fields such as physics or astronomy.

Still Finding the Right Questions:
Branching out and changing with the times at acmqueue

Welcome to the newest incarnation of acmqueue. When we started putting together the first edition of ACM Queue in early 2003, it was a completely new experiment in publishing for ACM. Targeting a practitioner audience meant that much of what we did would differ from academic publishing. We created a new editorial board whose role was not only to vet articles, but also to identify topics and authors that would be of interest to both practitioners and academics. The board created the concept of guest experts who would take on an issue of the magazine and help the board acquire content and sometimes write the overarching piece that tied it all together.

Crash Consistency:
Rethinking the Fundamental Abstractions of the File System

The reading and writing of data, one of the most fundamental aspects of any Von Neumann computer, is surprisingly subtle and full of nuance. For example, consider access to a shared memory in a system with multiple processors. While a simple and intuitive approach known as strong consistency is easiest for programmers to understand, many weaker models are in widespread use (e.g., x86 total store ordering); such approaches improve system performance, but at the cost of making reasoning about system behavior more complex and error-prone.

Testing a Distributed System:
Testing a distributed system can be trying even under the best of circumstances.

Distributed systems can be especially difficult to program, for a variety of reasons. They can be difficult to design, difficult to manage, and, above all, difficult to test. Testing a normal system can be trying even under the best of circumstances, and no matter how diligent the tester is, bugs can still get through. Now take all of the standard issues and multiply them by multiple processes written in multiple languages running on multiple boxes that could potentially all be on different operating systems, and there is potential for a real disaster.

Natural Language Translation at the Intersection of AI and HCI:
Old questions being answered with both AI and HCI

The fields of artificial intelligence (AI) and human-computer interaction (HCI) are influencing each other like never before. Widely used systems such as Google Translate, Facebook Graph Search, and RelateIQ hide the complexity of large-scale AI systems behind intuitive interfaces. But relations were not always so auspicious. The two fields emerged at different points in the history of computer science, with different influences, ambitions, and attendant biases. AI aimed to construct a rival, and perhaps a successor, to the human intellect. Early AI researchers such as McCarthy, Minsky, and Shannon were mathematicians by training, so theorem-proving and formal models were attractive research directions.

Beyond Page Objects: Testing Web Applications with State Objects:
Use states to drive your tests

End-to-end testing of Web applications typically involves tricky interactions with Web pages by means of a framework such as Selenium WebDriver. The recommended method for hiding such Web-page intricacies is to use page objects, but there are questions to answer first: Which page objects should you create when testing Web applications? What actions should you include in a page object? Which test scenarios should you specify, given your page objects?

Hickory Dickory Doc:
On null encryption and automated documentation

Dear KV, While reviewing some encryption code in our product, I came across an option that allowed for null encryption. This means the encryption could be turned on, but the data would never be encrypted or decrypted. It would always be stored "in the clear." I removed the option from our latest source tree because I figured we didn’t want an unsuspecting user to turn on encryption but still have data stored in the clear. One of the other programmers on my team reviewed the potential change and blocked me from committing it, saying that the null code could be used for testing.

Dismantling the Barriers to Entry:
We have to choose to build a web that is accessible to everyone.

A war is being waged in the world of web development. On one side is a vanguard of toolmakers and tool users, who thrive on the destruction of bad old ideas ("old," in this milieu, meaning anything that debuted on Hacker News more than a month ago) and raucous debates about transpilers and suchlike.

Hadoop Superlinear Scalability:
The perpetual motion of parallel performance

We often see more than 100 percent speedup efficiency! came the rejoinder to the innocent reminder that you can’t have more than 100 percent of anything. But this was just the first volley from software engineers during a presentation on how to quantify computer system scalability in terms of the speedup metric. In different venues, on subsequent occasions, that retort seemed to grow into a veritable chorus that not only was superlinear speedup commonly observed, but also the model used to quantify scalability for the past 20 years failed when applied to superlinear speedup data.

Lazarus Code:
No one expects the Spanish Acquisition.

I’ve been asked to look into the possibility of taking a 15-year-old piece of open-source software and updating it to work on a current system used by my company. The code itself doesn’t seem to be too bad, at least no worse than the code I’m used to reading, but I suspect it might be easier to write a new version from scratch than to try to understand code that I didn’t write and which no one has actively maintained for several years.

Evolution and Practice: Low-latency Distributed Applications in Finance:
The finance industry has unique demands for low-latency distributed systems.

Virtually all systems have some requirements for latency, defined here as the time required for a system to respond to input. Latency requirements appear in problem domains as diverse as aircraft flight controls, voice communications, multiplayer gaming, online advertising, and scientific experiments. Distributed systems present special latency considerations. In recent years the automation of financial trading has driven requirements for distributed systems with challenging latency requirements and global geographic distribution. Automated trading provides a window into the engineering challenges of ever-shrinking latency requirements, which may be useful to software engineers in other fields.

The Science of Managing Data Science:
Lessons learned managing a data science research team

What are they doing all day? When I first took over as VP of Engineering at a startup doing data mining and machine learning research, this was what the other executives wanted to know. They knew the team was super smart, and they seemed like they were working really hard, but the executives had lots of questions about the work itself. How did they know that the work they were doing was the "right" work? Were there other projects they could be doing instead? And how could we get this research into the hands of our customers faster?

Using Free and Open Source Tools to Manage Software Quality:
An agile process implementation

The principles of agile software development place more emphasis on individuals and interactions than on processes and tools. They steer us away from heavy documentation requirements and guide us along a path of reacting efficiently to change rather than sticking rigidly to a pre-defined plan. To support this flexible method of operation, it is important to have suitable applications to manage the team’s activities. It is also essential to implement effective frameworks to ensure quality is being built into the product early and at all levels.

From the EDVAC to WEBVACs:
Cloud computing for computer scientists

By now everyone has heard of cloud computing and realized that it is changing how both traditional enterprise IT and emerging startups are building solutions for the future. Is this trend toward the cloud just a shift in the complicated economics of the hardware and software industry, or is it a fundamentally different way of thinking about computing? Having worked in the industry, I can confidently say it is both.

Spicing Up Dart with Side Effects:
A set of extensions to the Dart programming language, designed to support asynchrony and generator functions

The Dart programming language has recently incorporated a set of extensions designed to support asynchrony and generator functions. Because Dart is a language for Web programming, latency is an important concern. To avoid blocking, developers must make methods asynchronous when computing their results requires nontrivial time. Generator functions ease the task of computing iterable sequences.

Reliable Cron across the Planet:
...or How I stopped worrying and learned to love time

This article describes Google’s implementation of a distributed Cron service, serving the vast majority of internal teams that need periodic scheduling of compute jobs. During its existence, we have learned many lessons on how to design and implement what might seem like a basic service. Here, we discuss the problems that distributed Crons face and outline some potential solutions.

There is No Now:
Problems with simultaneity in distributed systems

Now. The time elapsed between when I wrote that word and when you read it was at least a couple of weeks. That kind of delay is one that we take for granted and don’t even think about in written media. "Now." If we were in the same room and instead I spoke aloud, you might have a greater sense of immediacy. You might intuitively feel as if you were hearing the word at exactly the same time that I spoke it. That intuition would be wrong. If, instead of trusting your intuition, you thought about the physics of sound, you would know that time must have elapsed between my speaking and your hearing.

Parallel Processing with Promises:
A simple method of writing a collaborative system

In today’s world, there are many reasons to write concurrent software. The desire to improve performance and increase throughput has led to many different asynchronous techniques. The techniques involved, however, are generally complex and the source of many subtle bugs, especially if they require shared mutable state. If shared state is not required, then these problems can be solved with a better abstraction called promises. These allow programmers to hook asynchronous function calls together, waiting for each to return success or failure before running the next appropriate function in the chain.

Raw Networking:
Relevance and repeatability

Dear KV, The company I work for has decided to use a wireless network link to reduce latency, at least when the weather between the stations is good. It seems to me that for transmission over lossy wireless links we’ll want our own transport protocol that sits directly on top of whatever the radio provides, instead of wasting bits on IP and TCP or UDP headers, which, for a point-to-point network, aren’t really useful.

META II: Digital Vellum in the Digital Scriptorium:
Revisiting Schorre’s 1962 compiler-compiler

Some people do living history -- reviving older skills and material culture by reenacting Waterloo or knapping flint knives. One pleasant rainy weekend in 2012, I set my sights a little more recently and settled in for a little meditative retro-computing, ca. 1962, following the ancient mode of transmission of knowledge: lecture and recitation -- or rather, grace of living in historical times, lecture (here, in the French sense, reading) and transcription (or even more specifically, grace of living post-Post, lecture and reimplementation).

Model-based Testing: Where Does It Stand?:
MBT has positive effects on efficiency and effectiveness, even if it only partially fulfills high expectations.

You have probably heard about MBT (model-based testing), but like many software-engineering professionals who have not used MBT, you might be curious about others’ experience with this test-design method. From mid-June 2014 to early August 2014, we conducted a survey to learn how MBT users view its efficiency and effectiveness. The 2014 MBT User Survey, a follow-up to a similar 2012 survey, was open to all those who have evaluated or used any MBT approach. Its 32 questions included some from a survey distributed at the 2013 User Conference on Advanced Automated Testing. Some questions focused on the efficiency and effectiveness of MBT, providing the figures that managers are most interested in.

Go Static or Go Home:
In the end, dynamic systems are simply less secure.

Most current and historic problems in computer and network security boil down to a single observation: letting other people control our devices is bad for us. At another time, I’ll explain what I mean by "other people" and "bad." For the purpose of this article, I’ll focus entirely on what I mean by control. One way we lose control of our devices is to external distributed denial of service (DDoS) attacks, which fill a network with unwanted traffic, leaving no room for real ("wanted") traffic. Other forms of DDoS are similar: an attack by the Low Orbit Ion Cannon (LOIC), for example, might not totally fill up a network, but it can keep a web server so busy answering useless attack requests that the server can’t answer any useful customer requests.

Securing the Network Time Protocol:
Crackers discover how to use NTP as a weapon for abuse.

In the late 1970s David L. Mills began working on the problem of synchronizing time on networked computers, and NTP (Network Time Protocol) version 1 made its debut in 1980. This was at a time when the net was a much friendlier place - the ARPANET days. NTP version 2 appeared approximately a year later, about the same time as CSNET (Computer Science Network). NSFNET (National Science Foundation Network) launched in 1986. NTP version 3 showed up in 1993.

HTTP/2.0 - The IETF is Phoning It In:
Bad protocol, bad politics

In the long run, the most memorable event of 1989 will probably be that Tim Berners-Lee hacked up the HTTP protocol and named the result the "World Wide Web." Tim’s HTTP protocol ran on 10Mbit/s, Ethernet, and coax cables, and his computer was a NeXT Cube with a 25-MHz clock frequency. Twenty-six years later, my laptop CPU is a hundred times faster and has a thousand times as much RAM as Tim’s machine had, but the HTTP protocol is still the same.

Scalability Techniques for Practical Synchronization Primitives:
Designing locking primitives with performance in mind

In an ideal world, applications are expected to scale automatically when executed on increasingly larger systems. In practice, however, not only does this scaling not occur, but it is common to see performance actually worsen on those larger systems.

Internal Access Controls:
Trust, but Verify

Every day seems to bring news of another dramatic and high-profile security incident, whether it is the discovery of longstanding vulnerabilities in widely used software such as OpenSSL or Bash, or celebrity photographs stolen and publicized. There seems to be an infinite supply of zero-day vulnerabilities and powerful state-sponsored attackers. In the face of such threats, is it even worth trying to protect your systems and data? What can systems security designers and administrators do?

Disambiguating Databases:
Use the database built for your access model.

The topic of data storage is one that doesn’t need to be well understood until something goes wrong (data disappears) or something goes really right (too many customers). Because databases can be treated as black boxes with an API, their inner workings are often overlooked. They’re often treated as magic things that just take data when offered and supply it when asked. Since these two operations are the only understood activities of the technology, they are often the only features presented when comparing different technologies.

Too Big to Fail:
Visibility leads to debuggability.

Our project has been rolling out a well-known, distributed key/value store onto our infrastructure, and we’ve been surprised - more than once - when a simple increase in the number of clients has not only slowed things, but brought them to a complete halt. This then results in rollback while several of us scour the online forums to figure out if anyone else has seen the same problem. The entire reason for using this project’s software is to increase the scale of a large system, so I have been surprised at how many times a small increase in load has led to a complete failure.

A New Software Engineering:
What happened to the promise of rigorous, disciplined, professional practices for software development?

What happened to software engineering? What happened to the promise of rigorous, disciplined, professional practices for software development, like those observed in other engineering disciplines? What has been adopted under the rubric of "software engineering" is a set of practices largely adapted from other engineering disciplines: project management, design and blueprinting, process control, and so forth. The basic analogy was to treat software as a manufactured product, with all the real "engineering" going on upstream of that - in requirements analysis, design, modeling, etc.

There’s No Such Thing as a General-purpose Processor:
And the belief in such a device is harmful

There is an increasing trend in computer architecture to categorize processors and accelerators as "general purpose." Of the papers published at this year’s International Symposium on Computer Architecture (ISCA 2014), nine out of 45 explicitly referred to general-purpose processors; one additionally referred to general-purpose FPGAs (field-programmable gate arrays), and another referred to general-purpose MIMD (multiple instruction, multiple data) supercomputers, stretching the definition to the breaking point. This article presents the argument that there is no such thing as a truly general-purpose processor and that the belief in such a device is harmful.

The Responsive Enterprise: Embracing the Hacker Way:
Soon every company will be a software company.

As of July 2014, Facebook, founded in 2004, is in the top 20 of the most valuable companies in the S&P 500, putting the 10-year-old software company in the same league as IBM, Oracle, and Coca-Cola. Of the top five fastest-growing companies with regard to market capitalization in 2014 (table 1), three are software companies: Apple, Google, and Microsoft (in fact, one could argue that Intel is also driven by software, making it four out of five).

Evolution of the Product Manager:
Better education needed to develop the discipline

Software practitioners know that product management is a key piece of software development. Product managers talk to users to help figure out what to build, define requirements, and write functional specifications. They work closely with engineers throughout the process of building software. They serve as a sounding board for ideas, help balance the schedule when technical challenges occur - and push back to executive teams when technical revisions are needed. Product managers are involved from before the first code is written, until after it goes out the door.

Productivity in Parallel Programming: A Decade of Progress:
Looking at the design and benefits of X10

In 2002 DARPA (Defense Advanced Research Projects Agency) launched a major initiative in HPCS (high-productivity computing systems). The program was motivated by the belief that the utilization of the coming generation of parallel machines was gated by the difficulty of writing, debugging, tuning, and maintaining software at peta scale.

JavaScript and the Netflix User Interface:
Conditional dependency resolution

In the two decades since its introduction, JavaScript has become the de facto official language of the Web. JavaScript trumps every other language when it comes to the number of runtime environments in the wild. Nearly every consumer hardware device on the market today supports the language in some way. While this is done most commonly through the integration of a Web browser application, many devices now also support Web views natively as part of the operating system UI (user interface).

Port Squatting:
Don’t irk your local sysadmin.

Dear KV, A few years ago you upbraided some developers for not following the correct process when requesting a reserved network port from IETF (Internet Engineering Task Force). While I get that squatting a used port is poor practice, I wonder if you, yourself, have ever tried to get IETF to allocate a port. We recently went through this with a new protocol on an open-source project, and it was a nontrivial and frustrating exercise.

Security Collapse in the HTTPS Market:
Assessing legal and technical solutions to secure HTTPS

HTTPS (Hypertext Transfer Protocol Secure) has evolved into the de facto standard for secure Web browsing. Through the certificate-based authentication protocol, Web services and Internet users first authenticate one another ("shake hands") using a TLS/SSL certificate, encrypt Web communications end-to-end, and show a padlock in the browser to signal that a communication is secure. In recent years, HTTPS has become an essential technology to protect social, political, and economic activities online.

Why Is It Taking So Long to Secure Internet Routing?:
Routing security incidents can still slip past deployed security defenses.

BGP (Border Gateway Protocol) is the glue that sticks the Internet together, enabling data communications between large networks operated by different organizations. BGP makes Internet communications global by setting up routes for traffic between organizations - for example, from Boston University’s network, through larger ISPs (Internet service providers) such as Level3, Pakistan Telecom, and China Telecom, then on to residential networks such as Comcast or enterprise networks such as Bank of America.

Certificate Transparency:
Public, verifiable, append-only logs

On August 28, 2011, a mis-issued wildcard HTTPS certificate for google.com was used to conduct a man-in-the-middle attack against multiple users in Iran. The certificate had been issued by a Dutch CA (certificate authority) known as DigiNotar, a subsidiary of VASCO Data Security International. Later analysis showed that DigiNotar had been aware of the breach in its systems for more than a month - since at least July 19. It also showed that at least 531 fraudulent certificates had been issued. The final count may never be known, since DigiNotar did not have records of all the mis-issued certificates.

Securing the Tangled Web:
Preventing script injection vulnerabilities through software design

Script injection vulnerabilities are a bane of Web application development: deceptively simple in cause and remedy, they are nevertheless surprisingly difficult to prevent in large-scale Web development.

Privacy, Anonymity, and Big Data in the Social Sciences:
Quality social science research and the privacy of human subjects requires trust.

Open data has tremendous potential for science, but, in human subjects research, there is a tension between privacy and releasing high-quality open data. Federal law governing student privacy and the release of student records suggests that anonymizing student data protects student privacy. Guided by this standard, we de-identified and released a data set from 16 MOOCs (massive open online courses) from MITx and HarvardX on the edX platform. In this article, we show that these and other de-identification procedures necessitate changes to data sets that threaten replication and extension of baseline analyses. To balance student privacy and the benefits of open data, we suggest focusing on protecting privacy without anonymizing data by instead expanding policies that compel researchers to uphold the privacy of the subjects in open data sets.

The Network is Reliable:
An informal survey of real-world communications failures

The network is reliable tops Peter Deutsch’s classic list, "Eight fallacies of distributed computing", "all [of which] prove to be false in the long run and all [of which] cause big trouble and painful learning experiences." Accounting for and understanding the implications of network behavior is key to designing robust distributed programs; in fact, six of Deutsch’s "fallacies" directly pertain to limitations on networked communications.

Undergraduate Software Engineering: Addressing the Needs of Professional Software Development:
Addressing the Needs of Professional Software Development

In the fall semester of 1996 RIT (Rochester Institute of Technology) launched the first undergraduate software engineering program in the United States. The culmination of five years of planning, development, and review, the program was designed from the outset to prepare graduates for professional positions in commercial and industrial software development.

Bringing Arbitrary Compute to Authoritative Data:
Many disparate use cases can be satisfied with a single storage system.

While the term ’big data’ is vague enough to have lost much of its meaning, today’s storage systems are growing more quickly and managing more data than ever before. Consumer devices generate large numbers of photos, videos, and other large digital assets. Machines are rapidly catching up to humans in data generation through extensive recording of system logs and metrics, as well as applications such as video capture and genome sequencing. Large data sets are now commonplace, and people increasingly want to run sophisticated analyses on the data.

ACM and the Professional Programmer:
How do you, the reader, stay informed about research that influences your work?

In the very early days of computing, professional programming was nearly synonymous with academic research because computers tended to be devices that existed only or largely in academic settings. As computers became commercially available, they began to be found in private-sector, business environments. The 1950s and 1960s brought computing in the form of automation and data processing to the private sector and, along with this came a growing community of professionals whose focus on computing was pragmatic and production oriented. Computing was (and still is) evolving, and the academic community continued to explore new software and hardware concepts and constructs. New languages were invented (and are still being invented) to try new ideas in the formulation of programs.

Outsourcing Responsibility:
What do you do when your debugger fails you?

Dear KV, I’ve been assigned to help with a new project and have been looking over the admittedly skimpy documentation the team has placed on the internal wiki. I spent a day or so staring at what seemed to be a long list of open-source projects that they intend to integrate into the system they have been building, but I couldn’t find where their original work was described.

Quality Software Costs Money - Heartbleed Was Free:
How to generate funding for FOSS

The world runs on free and open-source software, FOSS for short, and to some degree it has predictably infiltrated just about any software-based product anywhere in the world.

Who Must You Trust?:
You must have some trust if you want to get anything done.

In his novel The Diamond Age, author Neal Stephenson describes a constructed society (called a phyle) based on extreme trust in one’s fellow members. Part of the membership requirements is that, from time to time, each member is called upon to undertake certain tasks to reinforce that trust. For example, a phyle member might be told to go to a particular location at the top of a cliff at a specific time, where he will find bungee cords with ankle harnesses attached. The other ends of the cords trail off into the bushes. At the appointed time he is to fasten the harnesses to his ankles and jump off the cliff.

Automated QA Testing at EA: Driven by Events:
A discussion with Michael Donat, Jafar Husain, and Terry Coatta

To millions of game geeks, the position of QA (quality assurance) tester at Electronic Arts must seem like a dream job. But from the company’s perspective, the overhead associated with QA can look downright frightening, particularly in an era of massively multiplayer games.

Design Exploration through Code-generating DSLs:
High-level DSLs for low-level programming

DSLs (domain-specific languages) make programs shorter and easier to write. They can be stand-alone - for example, LaTeX, Makefiles, and SQL - or they can be embedded in a host language. You might think that DSLs embedded in high-level languages would be abstract or mathematically oriented, far from the nitty-gritty of low-level programming. This is not the case. This article demonstrates how high-level EDSLs (embedded DSLs) really can ease low-level programming. There is no contradiction.

Finding More Than One Worm in the Apple:
If you see something, say something.

In February Apple revealed and fixed an SSL (Secure Sockets Layer) vulnerability that had gone undiscovered since the release of iOS 6.0 in September 2012. It left users vulnerable to man-in-the-middle attacks thanks to a short circuit in the SSL/TLS (Transport Layer Security) handshake algorithm introduced by the duplication of a goto statement. Since the discovery of this very serious bug, many people have written about potential causes.

Domain-specific Languages and Code Synthesis Using Haskell:
Looking at embedded DSLs

There are many ways to give instructions to a computer: an electrical engineer might write a MATLAB program; a database administrator might write an SQL script; a hardware engineer might write in Verilog; and an accountant might write a spreadsheet with embedded formulas. Aside from the difference in language used in each of these examples, there is an important difference in form and idiom. Each uses a language customized to the job at hand, and each builds computational requests in a form both familiar and productive for programmers (although accountants may not think of themselves as programmers).

The NSA and Snowden: Securing the All-Seeing Eye:
How good security at the NSA could have stopped him

Edward Snowden, while an NSA (National Security Agency) contractor at Booz Allen Hamilton in Hawaii, copied up to 1.7 million top-secret and above documents, smuggling copies on a thumb drive out of the secure facility in which he worked, and later released many to the press. This has altered the relationship of the U.S. government with the American people, as well as with other countries. This article examines the computer security aspects of how the NSA could have prevented this, perhaps the most damaging breach of secrets in U.S. history.

The Curse of the Excluded Middle:
Mostly functional programming does not work.

There is a trend in the software industry to sell "mostly functional" programming as the silver bullet for solving problems developers face with concurrency, parallelism (manycore), and, of course, Big Data. Contemporary imperative languages could continue the ongoing trend, embrace closures, and try to limit mutation and other side effects. Unfortunately, just as "mostly secure" does not work, "mostly functional" does not work either. Instead, developers should seriously consider a completely fundamentalist option as well: embrace pure lazy functional programming with all effects explicitly surfaced in the type system using monads.

Forked Over:
Shortchanged by open source

How can one make reasonable packages based on open-source software when most open-source projects simply advise you to take the latest bits on GitHub or SourceForge? We could fork the code, as GitHub encourages us to do, and then make our own releases, but that puts the release-engineering work that we would expect from the project onto us.

Don’t Settle for Eventual Consistency:
Stronger properties for low-latency geo-replicated storage

Geo-replicated storage provides copies of the same data at multiple, geographically distinct locations. Facebook, for example, geo-replicates its data (profiles, friends lists, likes, etc.) to data centers on the east and west coasts of the United States, and in Europe. In each data center, a tier of separate Web servers accepts browser requests and then handles those requests by reading and writing data from the storage system.

Please Put OpenSSL Out of Its Misery:
OpenSSL must die, for it will never get any better.

The OpenSSL software package is around 300,000 lines of code, which means there are probably around 299 bugs still there, now that the Heartbleed bug which allowed pretty much anybody to retrieve internal state to which they should normally not have access has been fixed.

A Primer on Provenance:
Better understanding of data requires tracking its history and context.

Assessing the quality or validity of a piece of data is not usually done in isolation. You typically examine the context in which the data appears and try to determine its original sources or review the process through which it was created. This is not so straightforward when dealing with digital data, however: the result of a computation might have been derived from numerous sources and by applying complex successive transformations, possibly over long periods of time.

Multipath TCP:
Decoupled from IP, TCP is at last able to support multihomed hosts.

The Internet relies heavily on two protocols. In the network layer, IP (Internet Protocol) provides an unreliable datagram service and ensures that any host can exchange packets with any other host. Since its creation in the 1970s, IP has seen the addition of several features, including multicast, IPsec (IP security), and QoS (quality of service). The latest revision, IPv6 (IP version 6), supports 16-byte addresses.

Major-league SEMAT: Why Should an Executive Care?:
Becoming better, faster, cheaper, and happier

In today’s ever more competitive world, boards of directors and executives demand that CIOs and their teams deliver "more with less." Studies show, without any real surprise, that there is no one-size-fits-all method to suit all software initiatives, and that a practice-based approach with some light but effective degree of order and governance is the goal of most software-development departments.

The Logic of Logging:
And the illogic of PDF

I work in a pretty open environment, and by open I mean that many people have the ability to become the root user on our servers so that they can fix things as they break.

Eventually Consistent: Not What You Were Expecting?:
Methods of quantifying consistency (or lack thereof) in eventually consistent storage systems

Storage systems continue to lay the foundation for modern Internet services such as Web search, e-commerce, and social networking. Pressures caused by rapidly growing user bases and data sets have driven system designs away from conventional centralized databases and toward more scalable distributed solutions, including simple NoSQL key-value storage systems, as well as more elaborate NewSQL databases that support transactions at scale.

Scaling Existing Lock-based Applications with Lock Elision:
Lock elision enables existing lock-based programs to achieve the performance benefits of nonblocking synchronization and fine-grain locking with minor software engineering effort.

Multithreaded applications take advantage of increasing core counts to achieve high performance. Such programs, however, typically require programmers to reason about data shared among multiple threads. Programmers use synchronization mechanisms such as mutual-exclusion locks to ensure correct updates to shared data in the presence of accesses from multiple threads. Unfortunately, these mechanisms serialize thread accesses to the data and limit scalability.

Rate-limiting State:
The edge of the Internet is an unruly place

By design, the Internet core is dumb, and the edge is smart. This design decision has enabled the Internet’s wildcat growth, since without complexity the core can grow at the speed of demand. On the downside, the decision to put all smartness at the edge means we’re at the mercy of scale when it comes to the quality of the Internet’s aggregate traffic load. Not all device and software builders have the skills and the quality assurance budgets that something the size of the Internet deserves.

The API Performance Contract:
How can the expected interactions between caller and implementation be guaranteed?

When you call functions in an API, you expect them to work correctly; sometimes this expectation is called a contract between the caller and the implementation. Callers also have performance expectations about these functions, and often the success of a software system depends on the API meeting these expectations. So there’s a performance contract as well as a correctness contract. The performance contract is usually implicit, often vague, and sometimes breached (by caller or implementation). How can this aspect of API design and documentation be improved?

Provenance in Sensor Data Management:
A cohesive, independent solution for bringing provenance to scientific research

In today’s information-driven workplaces, data is constantly being moved around and undergoing transformation. The typical business-as-usual approach is to use e-mail attachments, shared network locations, databases, and more recently, the cloud. More often than not, there are multiple versions of the data sitting in different locations, and users of this data are confounded by the lack of metadata describing its provenance or in other words, its lineage. The ProvDMS project at the Oak Ridge National Laboratory (ORNL) described in this article aims to solve this issue in the context of sensor data.

Node at LinkedIn: The Pursuit of Thinner, Lighter, Faster:
A discussion with Kiran Prasad, Kelly Norton, and Terry Coatta

Node.js, the server-side JavaScript-based software platform used to build scalable network applications, has been all the rage among many developers for the past couple of years, although its popularity has also managed to enrage some others, who have unleashed a barrage of negative blog posts to point out its perceived shortcomings. Still, while new and untested, Node continues to win more converts.

This is the Foo Field:
The meaning of bits and avoiding upgrade bog downs

When will someone write documentation that tells you what the bits mean rather than what they set? I’ve been working to integrate a library into our system, and every time I try to figure out what it wants from my code, all it tells me is what a part of it is: "This is the foo field." The problem is that it doesn’t tell me what happens when I set foo. It’s as if I’m supposed to know that already.

Unikernels: Rise of the Virtual Library Operating System:
What if all the software layers in a virtual appliance were compiled within the same safe, high-level language framework?

Cloud computing has been pioneering the business of renting computing resources in large data centers to multiple (and possibly competing) tenants. The basic enabling technology for the cloud is operating-system virtualization such as Xen1 or VMWare, which allows customers to multiplex VMs (virtual machines) on a shared cluster of physical machines. Each VM presents as a self-contained computer, booting a standard operating-system kernel and running unmodified applications just as if it were executing on a physical machine.

Toward Software-defined SLAs:
Enterprise computing in the public cloud

The public cloud has introduced new technology and architectures that could reshape enterprise computing. In particular, the public cloud is a new design center for enterprise applications, platform software, and services. API-driven orchestration of large-scale, on-demand resources is an important new design attribute, which differentiates public-cloud from conventional enterprise data-center infrastructure. Enterprise applications must adapt to the new public-cloud design center, but at the same time new software and system design patterns can add enterprise attributes and service levels to public-cloud services.

The Road to SDN:
An intellectual history of programmable networks

Designing and managing networks has become more innovative over the past few years with the aid of SDN (software-defined networking). This technology seems to have appeared suddenly, but it is actually part of a long history of trying to make computer networks more programmable.

Center Wheel for Success:
Not invented here syndrome is not unique to the IT world.

When I first read the claim that HealthCare.gov, the Web site initiated by the Affordable Care Act, had cost $500 million to create, I didn’t believe the number. There is no way to make a Web site cost that much. But the actual number seems not to be an order-of-magnitude lower, and as I understand the reports, the Web site doesn’t have much to show for the high cost in term of performance, features, or quality in general.

The Software Inferno:
Dante’s tale, as experienced by a software architect

The Software Inferno is a tale that parallels The Inferno, Part One of The Divine Comedy written by Dante Alighieri in the early 1300s. That literary masterpiece describes the condemnation and punishment faced by a variety of sinners in their hell-spent afterlives as recompense for atrocities committed during their earthly existences. The Software Inferno is a similar account, describing a journey where "sinners against software" are encountered amidst their torment, within their assigned areas of eternal condemnation, and paying their penance.

Making the Web Faster with HTTP 2.0:
HTTP continues to evolve

HTTP (Hypertext Transfer Protocol) is one of the most widely used application protocols on the Internet. Since its publication, RFC 2616 (HTTP 1.1) has served as a foundation for the unprecedented growth of the Internet: billions of devices of all shapes and sizes, from desktop computers to the tiny Web devices in our pockets, speak HTTP every day to deliver news, video, and millions of other Web applications we have all come to depend on in our everyday lives.

Intermediate Representation:
The increasing significance of intermediate representations in compilers

Program compilation is a complicated process. A compiler is a software program that translates a high-level source language program into a form ready to execute on a computer. Early in the evolution of compilers, designers introduced IRs (intermediate representations, also commonly called intermediate languages) to manage the complexity of the compilation process. The use of an IR as the compiler’s internal representation of the program enables the compiler to be broken up into multiple phases and components, thus benefiting from modularity.

The Challenge of Cross-language Interoperability:
Interfacing between languages is increasingly important.

Interoperability between languages has been a problem since the second programming language was invented. Solutions have ranged from language-independent object models such as COM (Component Object Model) and CORBA (Common Object Request Broker Architecture) to VMs (virtual machines) designed to integrate languages, such as JVM (Java Virtual Machine) and CLR (Common Language Runtime). With software becoming ever more complex and hardware less homogeneous, the likelihood of a single language being the correct tool for an entire program is lower than ever. As modern compilers become more modular, there is potential for a new generation of interesting solutions.

Bugs and Bragging Rights:
It’s not always size that matters.

Dear KV, I’ve been dealing with a large program written in Java that seems to spend most of its time asking me to restart it because it has run out of memory.

Agile and SEMAT - Perfect Partners:
Combining agile and SEMAT yields more advantages than either one alone

Today, as always, many different initiatives are under way to improve the ways in which software is developed. The most popular and prevalent of these is the agile movement. One of the newer kids on the block is the SEMAT (Software Engineering Method and Theory) initiative. As with any new initiative, people are struggling to see how it fits into the world and relates to all the other things going on. For example, does it improve or replace their current ways of working?

Adopting DevOps Practices in Quality Assurance:
Merging the art and science of software development

Software life-cycle management was, for a very long time, a controlled exercise. The duration of product design, development, and support was predictable enough that companies and their employees scheduled their finances, vacations, surgeries, and mergers around product releases. When developers were busy, QA (quality assurance) had it easy. As the coding portion of a release cycle came to a close, QA took over while support ramped up. Then when the product released, the development staff exhaled, rested, and started the loop again while the support staff transitioned to busily supporting the new product.

Passively Measuring TCP Round-trip Times:
A close look at RTT measurements with TCP

Measuring and monitoring network RTT (round-trip time) is important for multiple reasons: it allows network operators and end users to understand their network performance and help optimize their environment, and it helps businesses understand the responsiveness of their services to sections of their user base.

Leaking Space:
Eliminating memory hogs

A space leak occurs when a computer program uses more memory than necessary. In contrast to memory leaks, where the leaked memory is never released, the memory consumed by a space leak is released, but later than expected. This article presents example space leaks and how to spot and eliminate them.

Barbarians at the Gateways:
High-frequency Trading and Exchange Technology

I am a former high-frequency trader. For a few wonderful years I led a group of brilliant engineers and mathematicians, and together we traded in the electronic marketplaces and pushed systems to the edge of their capability.

Online Algorithms in High-frequency Trading:
The challenges faced by competing HFT algorithms

HFT (high-frequency trading) has emerged as a powerful force in modern financial markets. Only 20 years ago, most of the trading volume occurred in exchanges such as the New York Stock Exchange, where humans dressed in brightly colored outfits would gesticulate and scream their trading intentions. Nowadays, trading occurs mostly in electronic servers in data centers, where computers communicate their trading intentions through network messages. This transition from physical exchanges to electronic platforms has been particularly profitable for HFT firms, which invested heavily in the infrastructure of this new environment.

A Lesson in Resource Management:
Waste not memory, want not memory—unless it doesn’t matter

Dear KV, I’ve been reworking a device driver for a high-end, high-performance networking card and I have a resource allocation problem. The devices I’m working with have several network ports, but these are not always in use; in fact, many of our customers use only one of the four available ports. It would greatly simplify the logic in my driver if I could allocate the resources for all the ports -- no matter how many there are -- when the device driver is first loaded into the system, instead of dealing with allocation whenever an administrator brings up an interface.

The Balancing Act of Choosing Nonblocking Features:
Design requirements of nonblocking systems

What is nonblocking progress? Consider the simple example of incrementing a counter C shared among multiple threads. One way to do so is by protecting the steps of incrementing C by a mutual exclusion lock L (i.e., acquire(L); old := C ; C := old+1; release(L);). If a thread P is holding L, then a different thread Q must wait for P to release L before Q can proceed to operate on C. That is, Q is blocked by P.

NUMA (Non-Uniform Memory Access): An Overview:
NUMA becomes more common because memory controllers get close to execution units on microprocessors.

NUMA (non-uniform memory access) is the phenomenon that memory at various points in the address space of a processor have different performance characteristics. At current processor speeds, the signal path length from the processor to memory plays a significant role. Increased signal path length not only increases latency to memory but also quickly becomes a throughput bottleneck if the signal path is shared by multiple processors. The performance differences to memory were noticeable first on large-scale systems where data paths were spanning motherboards or chassis. These systems required modified operating-system kernels with NUMA support that explicitly understood the topological properties of the system’s memory (such as the chassis in which a region of memory was located) in order to avoid excessively long signal path lengths.

20 Obstacles to Scalability:
Watch out for these pitfalls that can prevent Web application scaling.

Web applications can grow in fits and starts. Customer numbers can increase rapidly, and application usage patterns can vary seasonally. This unpredictability necessitates an application that is scalable. What is the best way of achieving scalability?

Rules for Mobile Performance Optimization:
An overview of techniques to speed page loading

Performance has always been crucial to the success of Web sites. A growing body of research has proven that even small improvements in page-load times lead to more sales, more ad revenue, more stickiness, and more customer satisfaction for enterprises ranging from small e-commerce shops to megachains such as Walmart.

More Encryption Is Not the Solution:
Cryptography as privacy works only if both ends work at it in good faith.

The recent exposure of the dragnet-style surveillance of Internet traffic has provoked a number of responses that are variations of the general formula, "More encryption is the solution." This is not the case. In fact, more encryption will probably only make the privacy crisis worse than it already is.

Best Practices on the Move: Building Web Apps for Mobile Devices:
Which practices should be modified or avoided altogether by developers for the mobile Web?

If it wasn’t your priority last year or the year before, it’s sure to be your priority now: bring your Web site or service to mobile devices in 2013 or suffer the consequences. Early adopters have been talking about mobile taking over since 1999 - anticipating the trend by only a decade or so. Today, mobile Web traffic is dramatically on the rise, and creating a slick mobile experience is at the top of everyone’s mind. Total mobile data traffic is expected to exceed 10 exabytes per month by 2017.

The Antifragile Organization:
Embracing Failure to Improve Resilience and Maximize Availability

Failure is inevitable. Disks fail. Software bugs lie dormant waiting for just the right conditions to bite. People make mistakes. Data centers are built on farms of unreliable commodity hardware. If you’re running in a cloud environment, then many of these factors are outside of your control. To compound the problem, failure is not predictable and doesn’t occur with uniform probability and frequency. The lack of a uniform frequency increases uncertainty and risk in the system.

The Naming of Hosts is a Difficult Matter:
Also, the perils of premature rebooting

The naming of hosts is a difficult matter that ranks with coding style, editor choice, and language preference in the pantheon of things computer people fight about that don’t matter to anyone else in the whole world.

Nonblocking Algorithms and Scalable Multicore Programming:
Exploring some alternatives to lock-based synchronization

Real-world systems with complicated quality-of-service guarantees may require a delicate balance between throughput and latency to meet operating requirements in a cost-efficient manner. The increasing availability and decreasing cost of commodity multicore and many-core systems make concurrency and parallelism increasingly necessary for meeting demanding performance requirements. Unfortunately, the design and implementation of correct, efficient, and scalable concurrent software is often a daunting task.

Proving the Correctness of Nonblocking Data Structures:
So you’ve decided to use a nonblocking data structure, and now you need to be certain of its correctness. How can this be achieved?

Nonblocking synchronization can yield astonishing results in terms of scalability and realtime response, but at the expense of verification state space.

Structured Deferral: Synchronization via Procrastination:
We simply do not have a synchronization mechanism that can enforce mutual exclusion.

Developers often take a proactive approach to software design, especially those from cultures valuing industriousness over procrastination. Lazy approaches, however, have proven their value, with examples including reference counting, garbage collection, and lazy evaluation. This structured deferral takes the form of synchronization via procrastination, specifically reference counting, hazard pointers, and RCU (read-copy-update).

Realtime GPU Audio:
Finite difference-based sound synthesis using graphics processors

Today’s CPUs are capable of supporting realtime audio for many popular applications, but some compute-intensive audio applications require hardware acceleration. This article looks at some realtime sound-synthesis applications and shares the authors’ experiences implementing them on GPUs (graphics processing units).

There’s Just No Getting around It: You’re Building a Distributed System:
Building a distributed system requires a methodical approach to requirements.

Distributed systems are difficult to understand, design, build, and operate. They introduce exponentially more variables into a design than a single machine does, making the root cause of an application problem much harder to discover. It should be said that if an application does not have meaningful SLAs (service-level agreements) and can tolerate extended downtime and/or performance degradation, then the barrier to entry is greatly reduced. Most modern applications, however, have an expectation of resiliency from their users, and SLAs are typically measured by "the number of nines" (e.g., 99.9 or 99.99 percent availability per month).

Resolved: the Internet Is No Place for Critical Infrastructure:
Risk is a necessary consequence of dependence

What is critical? To what degree is critical defined as a matter of principle, and to what degree is it defined operationally? I am distinguishing what we say from what we do.

Cherry-picking and the Scientific Method:
Software is supposed be a part of computer science, and science demands proof.

So while haggling with the cherry seller, it became obvious that buying a whole flat of cherries would be a better deal than buying a single basket, even though that was all we really wanted. Not wanting to pass up a deal, however, my friend bought the entire flat and off we went, eating and talking. It took another 45 minutes to get home, and during that time we had eaten more than half the flat of cherries.

A File System All Its Own:
Flash memory has come a long way. Now it’s time for software to catch up.

In the past five years, flash memory has progressed from a promising accelerator, whose place in the data center was still uncertain, to an established enterprise component for storing performance-critical data. It’s rise to prominence followed its proliferation in the consumer world and the volume economics that followed (see figure 1). With SSDs (solid-state devices), flash arrived in a form optimized for compatibility - just replace a hard drive with an SSD for radically better performance. But the properties of the NAND flash memory used by SSDs differ significantly from those of the magnetic media in the hard drives they often displace.

Eventual Consistency Today: Limitations, Extensions, and Beyond:
How can applications be built on eventually consistent infrastructure given no guarantee of safety?

In a July 2000 conference keynote, Eric Brewer, now VP of engineering at Google and a professor at the University of California, Berkeley, publicly postulated the CAP (consistency, availability, and partition tolerance) theorem, which would change the landscape of how distributed storage systems were architected. Brewer’s conjecture--based on his experiences building infrastructure for some of the first Internet search engines at Inktomi--states that distributed systems requiring always-on, highly available operation cannot guarantee the illusion of coherent, consistent single-system operation in the presence of network partitions, which cut communication between active servers.

Discrimination in Online Ad Delivery:
Google ads, black names and white names, racial discrimination, and click advertising

Do online ads suggestive of arrest records appear more often with searches of black-sounding names than white-sounding names? What is a black-sounding name or white-sounding name, anyway? How many more times would an ad have to appear adversely affecting one racial group for it to be considered discrimination? Is online activity so ubiquitous that computer scientists have to think about societal consequences such as structural racism in technology design? If so, how is this technology to be built? Let’s take a scientific dive into online ad delivery to find answers.

How Fast is Your Web Site?:
Web site performance data has never been more readily available.

The overwhelming evidence indicates that a Web site’s performance (speed) correlates directly to its success, across industries and business metrics. With such a clear correlation (and even proven causation), it is important to monitor how your Web site performs. So, how fast is your Web site?

FPGA Programming for the Masses:
The programmability of FPGAs must improve if they are to be part of mainstream computing.

When looking at how hardware influences computing performance, we have GPPs (general-purpose processors) on one end of the spectrum and ASICs (application-specific integrated circuits) on the other. Processors are highly programmable but often inefficient in terms of power and performance. ASICs implement a dedicated and fixed function and provide the best power and performance characteristics, but any functional change requires a complete (and extremely expensive) re-spinning of the circuits.

The Evolution of Web Development for Mobile Devices:
Building Web sites that perform well on mobile devices remains a challenge.

The biggest change in Web development over the past few years has been the remarkable rise of mobile computing. Mobile phones used to be extremely limited devices that were best used for making phone calls and sending short text messages. Today’s mobile phones are more powerful than the computers that took Apollo 11 to the moon with the ability to send data to and from nearly anywhere.

Swamped by Automation:
Whenever someone asks you to trust them, don’t.

So your group fell for the "just install this software and things will be great" ploy. It’s an old trick that continues to snag sysadmins and others who have supporting roles around developers. Whenever someone asks you to trust them, don’t. Cynical as that might be, it’s better than being suckered.

The Story of the Teapot in DHTML:
It’s easy to do amazing things, such as rendering the classic teapot in HTML and CSS.

Before there was SVG (Scalable Vector Graphics), WebGL (Web Graphics Library), Canvas, or much of anything for graphics in the browser, it was possible to do quite a lot more than was initially obvious. To demonstrate, we created a JavaScript program that renders polygonal 3D graphics using nothing more than HTML and CSS. Our proof-of-concept is fast enough to support physics-based small-game content, but we started with the iconic 3D "Utah teapot" because it tells the whole story in one picture. It’s feasible to render this classic object using just regular DIV elements, CSS styles, and a little bit of JavaScript code.

Making the Mobile Web Faster:
Mobile performance issues? Fix the back end, not just the client.

Mobile clients have been on the rise and will only continue to grow. This means that if you are serving clients over the Internet, you cannot ignore the customer experience on a mobile device. There are many informative articles on mobile performance, and just as many on general API design, but you’ll find few discussing the design considerations needed to optimize the back-end systems for mobile clients. Whether you have an app, mobile Web site, or both, it is likely that these clients are consuming APIs from your back-end systems.

Hazy: Making it Easier to Build and Maintain Big-data Analytics:
Racing to unleash the full potential of big data with the latest statistical and machine-learning techniques.

The rise of big data presents both big opportunities and big challenges in domains ranging from enterprises to sciences. The opportunities include better-informed business decisions, more efficient supply-chain management and resource allocation, more effective targeting of products and advertisements, better ways to "organize the world’s information," faster turnaround of scientific discoveries, etc.

A Decade of OS Access-control Extensibility:
Open source security foundations for mobile and embedded devices

To discuss operating system security is to marvel at the diversity of deployed access-control models: Unix and Windows NT multiuser security; Type Enforcement in SELinux; anti-malware products; app sandboxing in Apple OS X, Apple iOS, and Google Android; and application-facing systems such as Capsicum in FreeBSD. This diversity is the result of a stunning transition from the narrow 1990s Unix and NT status quo to ’security localization’ - the adaptation of operating-system security models to site-local or product-specific requirements.

Divided by Division:
Is there a best used-by date for software?

Do you know of any rule of thumb for how often a piece of software should need maintenance? I’m not thinking about bug fixes, since bugs are there from the moment the code is written, but about the constant refactoring that seems to go on in code. Sometimes I feel as if programmers use refactoring as a way of keeping their jobs, rather than offering any real improvement.

Rethinking Passwords:
Our authentication system is lacking. Is improvement possible?

There is an authentication plague upon the land. We have to claim and assert our identity repeatedly to a host of authentication trolls, each jealously guarding an Internet service of some sort. Each troll has specific rules for passwords, and the rules vary widely and incomprehensibly.

Thinking Methodically about Performance:
The USE method addresses shortcomings in other commonly used methodologies.

Performance issues can be complex and mysterious, providing little or no clue to their origin. In the absence of a starting point, performance issues are often analyzed randomly: guessing where the problem may be and then changing things until it goes away. While this can deliver results it can also be time-consuming, disruptive, and may ultimately overlook certain issues. This article describes system-performance issues and the methodologies in use today for analyzing them, and it proposes a new methodology for approaching and solving a class of issues.

Code Abuse:
One programmer’s extension is another programmer’s abuse.

During some recent downtime at work, I’ve been cleaning up a set of libraries, removing dead code, updating documentation blocks, and fixing minor bugs that have been annoying but not critical. This bit of code spelunking has revealed how some of the libraries have been not only used, but also abused. The fact that everyone and their sister use the timing library for just about any event they can think of isn’t so bad, as it is a library that’s meant to call out to code periodically (although some of the events seem as if they don’t need to be events at all).

Splinternet Behind the Great Firewall of China:
Once China opened its door to the world, it could not close it again.

What if you could not access YouTube, Facebook, Twitter, and Wikipedia? How would you feel if Google informed you that your connection had been reset during a search? What if Gmail was only periodically available, and Google Docs, which was used to compose this article, was completely unreachable? What a mess!

Browser Security Case Study: Appearances Can Be Deceiving:
A discussion with Jeremiah Grossman, Ben Livshits, Rebecca Bace, and George Neville-Neil

It seems every day we learn of some new security breach. It’s all there for the taking on the Internet: more and more sensitive data every second. As for privacy, we Facebook, we Google, we bank online, we shop online, we invest online& we put it all out there. And just how well protected is all that personally identifiable information? Not very.

Condos and Clouds:
Constraints in an environment empower the services.

Living in a condominium has its constraints and its services. By defining the lifestyle and limits on usage patterns, it is possible to pack many homes close together and to provide the residents with many conveniences. Condo living can offer a great value to those interested and willing to live within its constraints and enjoy the sharing of common services.

The Web Won’t Be Safe or Secure until We Break It:
Unless you’ve taken very particular precautions, assume every Web site you visit knows exactly who you are.

The Internet was designed to deliver information, but few people envisioned the vast amounts of information that would be involved or the personal nature of that information. Similarly, few could have foreseen the potential flaws in the design of the Internet that would expose this personal information, compromising the data of individuals and companies.

The Essence of Software Engineering: The SEMAT Kernel:
A thinking framework in the form of an actionable kernel

Everyone who develops software knows that it is a complex and risky business, and its participants are always on the lookout for new ideas that will lead to better software. Fortunately, software engineering is still a young and growing profession that sees innovations and improvements in best practices every year. Just look, for example, at the improvements and benefits that lean and agile thinking have brought to software-development teams.

Anatomy of a Solid-state Drive:
While the ubiquitous SSD shares many features with the hard-disk drive, under the surface they are completely different.

Over the past several years, a new type of storage device has entered laptops and data centers, fundamentally changing expectations regarding the power, size, and performance dynamics of storage. The SSD (solid-state drive) is a technology that has been around for more than 30 years but remained too expensive for broad adoption.

Queue Portrait: Robert Watson:
Queue’s Kode Vicious interviews Robert Watson, a security researcher and open source developer at the University of Cambridge about a project studying the boundaries of the hardware/software interface.

Robert Watson is a security researcher and open source developer at the University of Cambridge looking at the hardware-software interface. He talks to us about spanning industry and academia, the importance of open source in software research, and challenges facing research that spans traditional boundaries in computer science. We also learn a bit about CPU security, and why applications, rather than operating systems, are increasingly the focus of security research. What are the challenges in the evolving hardware-software interface? Could open source hardware provide a platform for hardware-software research? And why is current hardware part of the problem?

Sender-side Buffers and the Case for Multimedia Adaptation:
A proposal to improve the performance and availability of streaming video and other time-sensitive media

The Internet/Web architecture has developed to the point where it is common for the most popular sites to operate at a virtually unlimited scale, and many sites now cater to hundreds of millions of unique users. Performance and availability are generally essential to attract and sustain such user bases. As such, the network and server infrastructure plays a critical role in the fierce competition for users. Web pages should load in tens to a few hundred milliseconds at most.

Weathering the Unexpected:
Failures happen, and resilience drills help organizations prepare for them.

Whether it is a hurricane blowing down power lines, a volcanic-ash cloud grounding all flights for a continent, or a humble rodent gnawing through underground fibers -- the unexpected happens. We cannot do much to prevent it, but there is a lot we can do to be prepared for it. To this end, Google runs an annual, company-wide, multi-day Disaster Recovery Testing event -- DiRT -- the objective of which is to ensure that Google’s services and internal business operations continue to run following a disaster.

Resilience Engineering: Learning to Embrace Failure:
A discussion with Jesse Robbins, Kripa Krishnan, John Allspaw, and Tom Limoncelli

In the early 2000s, Amazon created GameDay, a program designed to increase resilience by purposely injecting major failures into critical systems semi-regularly to discover flaws and subtle dependencies. Basically, a GameDay exercise tests a company’s systems, software, and people in the course of preparing for a response to a disastrous event. Widespread acceptance of the GameDay concept has taken a few years, but many companies now see its value and have started to adopt their own versions. This discussion considers some of those experiences.

Disks from the Perspective of a File System:
Disks lie. And the controllers that run them are partners in crime.

Most applications do not deal with disks directly, instead storing their data in files in a file system, which protects us from those scoundrel disks. After all, a key task of the file system is to ensure that the file system can always be recovered to a consistent state after an unplanned system crash (for example, a power failure). While a good file system will be able to beat the disks into submission, the required effort can be great and the reduced performance annoying.

Toward Higher Precision:
An introduction to PTP and its significance to NTP practitioners

It is difficult to overstate the importance of synchronized time to modern computer systems. Our lives today depend on the financial transactions, telecommunications, power generation and delivery, high-speed manufacturing, and discoveries in "big physics," among many other things, that are driven by fast, powerful computing devices coordinated in time with each other.

Fault Injection in Production:
Making the case for resilience testing

When we build Web infrastructures at Etsy, we aim to make them resilient. This means designing them carefully so that they can sustain their (increasingly critical) operations in the face of failure. Thankfully, there have been a couple of decades and reams of paper spent on researching how fault tolerance and graceful degradation can be brought to computer systems. That helps the cause.

A Generation Lost in the Bazaar:
Quality happens only when someone is responsible for it.

Thirteen years ago, Eric Raymond’s book "The Cathedral and the Bazaar" (O’Reilly Media, 2001) redefined our vocabulary and all but promised an end to the waterfall model and big software companies, thanks to the new grass-roots open source software development movement. I found the book thought provoking, but it did not convince me. On the other hand, being deeply involved in open source, I couldn’t help but think that it would be nice if he was right.

Can More Code Mean Fewer Bugs?:
The bytes you save today may bite you tomorrow

Dear One, You almost had me with your appeal to simplicity, that having a single line with system() on it reduces the potential for bugs. Almost, but not quite.

All Your Database Are Belong to Us:
In the big open world of the cloud, highly available distributed objects will rule.

In the database world, the raw physical data model is at the center of the universe, and queries freely assume intimate details of the data representation (indexes, statistics, metadata). This closed-world assumption and the resulting lack of abstraction have the pleasant effect of allowing the data to outlive the application. On the other hand, this makes it hard to evolve the underlying model independently from the queries over the model.

Software Needs Seatbelts and Airbags:
Finding and fixing bugs in deployed software is difficult and time-consuming. Here are some alternatives.

Like death and taxes, buggy code is an unfortunate fact of life. Nearly every program ships with known bugs, and probably all of them end up with bugs that are discovered only post-deployment. There are many reasons for this sad state of affairs.

A New Objective-C Runtime: from Research to Production:
Backward compatibility always trumps new features.

The path from the research prototype (Étoilé runtime) to the shipping version (GNUstep runtime) involved a complete rewrite and redesign. This isn’t necessarily a bad thing: part of the point of building a prototype is to learn what makes sense and what doesn’t, and to investigate what is feasible in a world where you control the entire system, but not necessarily in production.

Multitier Programming in Hop:
A first step toward programming 21st-century applications

The Web is becoming the richest platform on which to create computer applications. Its power comes from three elements: (1) modern Web browsers enable highly sophisticated GUIs with 3D, multimedia, fancy typesetting, etc.; (2) calling existing services through Web APIs makes it possible to develop sophisticated applications from independently available components; and (3) open data availability allows applications to access a wide set of information that was unreachable or that simply did not exist before. The combination of these three elements has already given birth to revolutionary applications such as Google Maps, radio podcasts, and social networks.

OpenFlow: A Radical New Idea in Networking:
An open standard that enables software-defined networking

Computer networks have historically evolved box by box, with individual network elements occupying specific ecological niches as routers, switches, load balancers, NATs (network address translations), or firewalls. Software-defined networking proposes to overturn that ecology, turning the network as a whole into a platform and the individual network elements into programmable entities. The apps running on the network platform can optimize traffic flows to take the shortest path, just as the current distributed protocols do, but they can also optimize the network to maximize link utilization, create different reachability domains for different users, or make device mobility seamless.

Extending the Semantics of Scheduling Priorities:
Increasing parallelism demands new paradigms.

Application performance is directly affected by the hardware resources that the application requires, the degree to which such resources are available, and how the operating system addresses its requirements with regard to the other processes in the system. Ideally, an application would have access to all the resources it could use and be allowed to complete its work without competing with any other activity in the system. In a world of highly shared hardware resources and generalpurpose, time-share-based operating systems, however, no guarantees can be made as to how well resourced an application will be.

LinkedIn Password Leak: Salt Their Hide:
If it does not take a full second to calculate the password hash, it is too weak.

6.5 million unsalted SHA1 hashed LinkedIn passwords have appeared in the criminal underground. There are two words in that sentence that should cause LinkedIn no end of concern: "unsalted" and "SHA1."

A Nice Piece of Code:
Colorful metaphors and properly reusing functions

In the last installment of Kode Vicious (A System is not a Product, ACM Queue 10 (4), April 2012), I mentioned that I had recently read two pieces of code that had actually lowered, rather than raised, my blood pressure. As promised, this edition’s KV covers that second piece of code.

Getting What You Measure:
Four common pitfalls in using software metrics for project management

Software metrics - helpful tools or a waste of time? For every developer who treasures these mathematical abstractions of software systems there is a developer who thinks software metrics are invented just to keep project managers busy. Software metrics can be very powerful tools that help achieve your goals but it is important to use them correctly, as they also have the power to demotivate project teams and steer development in the wrong direction.

My Compiler Does Not Understand Me:
Until our programming languages catch up, code will be full of horrors.

Only lately have a lot of smart people found audiences for making sound points about what and how we code. Various colleagues have been beating drums and heads together for ages trying to make certain that wise insights about programming stick to neurons. Articles on coding style in this and other publications have provided further examples of such advocacy.

Modeling People and Places with Internet Photo Collections:
Understanding the world from the sea of online photos

This article describes our work in using online photo collections to reconstruct information about the world and its inhabitants at both global and local scales. This work has been driven by the dramatic growth of social content-sharing Web sites, which have created immense online collections of user-generated visual data. Flickr.com alone currently hosts more than 6 billion images taken by more than 40 million unique users, while Facebook.com has said it grows by nearly 250 million photos every day.

Controlling Queue Delay:
A modern AQM is just one piece of the solution to bufferbloat.

Nearly three decades after it was first diagnosed, the "persistently full buffer problem" recently exposed as part of "bufferbloat", is still with us and made increasingly critical by two trends. First, cheap memory and a "more is better" mentality have led to the inflation and proliferation of buffers. Second, dynamically varying path characteristics are much more common today and are the norm at the consumer Internet edge. Reasonably sized buffers become extremely oversized when link rates and path delays fall below nominal values.

A Guided Tour through Data-center Networking:
A good user experience depends on predictable performance within the data-center network.

The magic of the cloud is that it is always on and always available from anywhere. Users have come to expect that services are there when they need them. A data center (or warehouse-scale computer) is the nexus from which all the services flow. It is often housed in a nondescript warehouse-sized building bearing no indication of what lies inside. Amidst the whirring fans and refrigerator-sized computer racks is a tapestry of electrical cables and fiber optics weaving everything together -- the data-center network.

Realtime Computer Vision with OpenCV:
Mobile computer-vision technology will soon become as ubiquitous as touch interfaces.

Computer vision is a rapidly growing field devoted to analyzing, modifying, and high-level understanding of images. Its objective is to determine what is happening in front of a camera and use that understanding to control a computer or robotic system, or to provide people with new images that are more informative or aesthetically pleasing than the original camera images. Application areas for computer-vision technology include video surveillance, biometrics, automotive, photography, movie production, Web search, medicine, augmented reality gaming, new user interfaces, and many more.

Idempotence Is Not a Medical Condition:
An essential property for reliable systems