Check out Pat's
Scattered Thoughts on Distributed Systems
pathelland.substack.com
Escaping the Singularity
PDF
Escaping the Singularity
It's Not Your Grandmother's Database Anymore
The Best Place to Build a Subway
Building projects despite (and because of) existing complex systems
Pat Helland
Many engineering projects are big and complex. They require integrating into the existing environment to tie into stuff that precedes the new, big, complex thing. It is common to bemoan the challenges of dealing with the preexisting stuff. Many times, engineers don't realize that their projects (and their paychecks) exist only because of the preexisting and complex systems that impose constraints on the new work.
This column looks at some sophisticated urban redevelopment projects that are very much part of daily life in San Francisco and compares them with the challenges inherent in building software.
The Challenge of Infrastructure
The best place to build a subway is in the open cornfields of Nebraska.
It's pretty darned flat. You can space the subway stations at regular intervals in a consistent fashion. There are no pesky historic monuments that get in the way. No cathedrals and no civic center.
Unfortunately, there are no passengers in the corn fields and no economic reason to have a subway there. This leads to a chicken-and-egg problem. You can't easily build infrastructure where it's crowded, and you can't afford to build infrastructure where it's not crowded.
I'm fond of saying there are two kinds of software companies: those that start out building scalable infrastructure and those that are in business.
This means successful companies will invest in evolving their infrastructure for enhanced scalability. That's hard.
Infrastructure and Users Growing Together
Urban planning is fascinating. There are questions of financing the changes to the city and the churn of land use and infrastructure. In general, a decision is made that the old occupants must leave, and they are bought out at what is supposed to be fair market value. Significant amenities such as parks, transit centers, and more are planned for the area. With the now-vacant land and promised fancy amenities, developers can purchase parcels to develop skyscrapers for offices, hotels, and housing. The price of the property under the skyscrapers pays for the original land purchase and the amenities.
I recently spent five years living in Golden Gateway, right by the Ferry Building in the Financial District in San Francisco. The area was redeveloped in the 1970s. It involved tearing down an old, dilapidated produce market, which was moved about five miles south. Five skyscrapers of office building, a retail mall, hundreds of apartments, multiple parks, and a Hyatt Regency now occupy the space.
My job is smack in the middle of the Transbay redevelopment, also in the Financial District. The one- to two-story rundown light manufacturing in this area south of Market was condemned, and the land was transferred to city ownership. Transbay has been an ongoing project since around 2000 and seems to be more than half done. As I go in and out of work, I see brand-new towers, an amazing transit center and park, as well as cranes and construction.
Other redevelopments include the Yerba Buena district (now including the Moscone Convention Center) and the Mission Bay area (now including the new basketball arena, Chase Center).
Many of these massive projects have incurred various costs and challenges. In the 1950s and 1960s, the Fillmore redevelopment targeted a largely black part of town. A 36-block area was torn down including housing, a distinct lifestyle, and a world-famous jazz community. Most of the previous occupants could not afford to return.
In many cases, these huge, multi-decade redevelopment projects bring new life to part of a city, but sometimes we can't foresee what we're going to lose.
Jasper O'Farrell's Risky Plan
In 1847, San Francisco was a tiny town of about 600 people, formerly known as Yerba Buena, that just the previous year had joined up with the United States. Gold was not discovered in California until the next year, which would transform San Francisco into a major city.
A few years earlier, in 1835, William Richardson had settled in Yerba Buena, and laid out the streets for an expanded settlement on a north-south grid. This area, called Portsmouth Square, is now part of Chinatown. By 1847, numerous buildings were aligned on this north-south grid.
In the southern part of the town was Mission Dolores. Mission Street ran 4½ miles from the mission to San Francisco Bay in a northeasterly direction. I work in a building on Mission Street.
In 1847, the new American military mayor of San Francisco commissioned Jasper O'Farrell to perform a land survey of San Francisco. He corrected a number of the property boundaries and street alignments in the northern part of the city. He decided to cut the city into two grids: the northern grid running north-south and east-west, and the southern grid running northeast-southwest and southeast-northwest. The existing northern part of the city had a few hundred wooden structures. The street widths varied from 45 feet to 69 feet, some a bit larger. In the southern part of the city, where there was little or no settlement, O'Farrell decided to make the streets wider. Mission Street was laid out to be 82½ feet wide.
To separate these two parts of the grid, O'Farrell created a massive 120-foot-wide Market Street paralleling Mission Street in the southern part. This was a ridiculously large waste of land for a street, especially in a town of 600 people. Remember, the transportation at the time consisted of horses and wagons. What would be the reason for such a wide street?
The locals of the town were furious. That land was valuable and was being taken from them for no value at all. Quickly, a mob formed and decided Jasper O'Farrell should hang for wasting their land. The townsfolk set out to get him. Fortunately, a friend tipped O'Farrell off and he rode a horse to North Beach, caught a boat to the North Bay, and settled in Sonoma County, where he died in 1875.1
Serendipity Where You Least Expect It
In the 1950s, plans were laid for BART, Bay Area Rapid Transit. This commuter train would run at high speed, connecting San Francisco, through tubes under San Francisco Bay, to the East Bay including Oakland, Berkeley, and more. To everyone's surprise, the vote for increased taxes squeaked by in 1962, and BART was funded.
The Transbay Tube connected to San Francisco at Market Street, and plans included a tunnel under Market Street for about two miles. The Market Street Subway is shared with the light-rail Muni Metro lines that run trains throughout San Francisco. Muni runs only within the city of San Francisco.
Construction on Market Street started in 1967 and continued for almost 10 years. Using the cut-and-cover approach dramatically reduced the construction cost. Cut-and-cover involves digging a huge trench and building the stations and the tracks in the open air. When done, the underground project is covered with dirt. It's unlikely that the Market Street Subway could have been funded with a less intrusive approach. Unfortunately, for 10 years Market Street was a mess, and lots of businesses went bankrupt.
Still, the 120 foot width of Market Street opened up the possibility that construction with cut-and-cover could be attempted. Without that generous width, we likely wouldn't have BART today.
Why Does That Overpass Have a Bend in It?
When I have the occasion to sit in a window seat on a plane, during takeoff and landing I always look at the overpasses on the freeways. Frequently, I see a gentle bend in the road as it crosses the freeway. This causes me to smile as I recognize the remnant of a freeway's widening.
Many times, I've lived near or commuted through a freeway-widening construction project. Each time, the process seems nonsensical. Buildings on one side of the crossover road are torn down. Then, new foundations and a bridge are built side by side with the old bridge. In general, there's no interruption of the traffic on the overpass and, even more important, no interruption on the freeway. Sometimes, with excruciating pain, a freeway will have to be shut down over a long holiday weekend.
When the construction is done, the beautiful new overpass will have been constructed right next to the old one. The new overpass connects to the side street just where the old one did. It just swerves to the side as it crosses the freeway to allow for two overpasses before tearing down the old one.
One especially challenging part of evolving a complex system is keeping it going while it's being changed. Years back, any new version of software had to be sent in a box, and after installing, it would run better on the data stored on disk. By the early 1980s, I was worried about wide area network distributed transactions and how I could compatibly evolve the protocol. It was not unusual for this to take three releases of planned messaging changes, with each release being six months apart.
Now, everyone supports cloud-based solutions. Everything runs 24/7. That is a huge value for customers and puts additional constraints on the engineers supporting the system and the application. Just like the folks widening the freeway need to keep it running 24/7, we need to plan for the evolution of the system and the detailed steps required to get from here to there.
Conclusion
I've spent almost 35 years of my 40-plus-year career working at companies with thousands of engineers. Having so many engineers means it is both easier and harder to get projects completed. With lots of resources, you have the ability to assemble a substantial team. While you have the benefit of lots of resources, however, there's a lot of interdependency and engineering nuance to consider. Even more, there's the legacy of a large code base. Legacy typically offers much more good than bad.
Cities are usually designed around the prevailing transportation. I've been privileged to visit Old Jerusalem, which was built with donkeys in mind for transportation. Most streets in the old part of the city are perhaps 20-25 feet wide. Cars cannot drive on these streets. Of course, widening them would be impossible without destroying the buildings next to them.
Applications start with communication and data or database expectations, as well as application structure expectations. Just like cities and transportation evolve, the compute infrastructure evolves.
Starting with a clean slate may seem to be more desirable. There are fewer constraints. There's also an increased chance that your software project will not take root and will not matter to anyone. The best hope is to build something that has an appropriate investment in infrastructure based on the economics. While doing that, try to have the insight to leave especially wide roads, perhaps 120 feet wide, even if they don't matter much now. Just make sure the townsfolk don't become a mob looking for vengeance!
References
1. Prendergast, T. (1942). Forgotten pioneers. San Francisco: Books for Libraries Press, p.71.
Related articles
Eventual Consistency Today: Limitations, Extensions, and Beyond
How can applications be built on eventually consistent infrastructure given no guarantee of safety?
Peter Bailis and Ali Ghodsi
https://queue.acm.org/detail.cfm?id=2462076
Condos and Clouds
Constraints in an environment empower the services.
Pat Helland
https://queue.acm.org/detail.cfm?id=2398392
Sizing your System
Kode Vicious
https://queue.acm.org/detail.cfm?id=1413256
Pat Helland has been implementing transaction systems, databases, application platforms, distributed systems, fault-tolerant systems, and messaging systems since 1978. For recreation, he occasionally writes technical papers. He currently works at Salesforce.
Copyright © 2020 held by owner/author. Publication rights licensed to ACM.
Originally published in Queue vol. 18, no. 1—
Comment on this article in the ACM Digital Library
More related articles:
Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, John Wilkes - Borg, Omega, and Kubernetes
Though widespread interest in software containers is a relatively recent phenomenon, at Google we have been managing Linux containers at scale for more than ten years and built three different container-management systems in that time. Each system was heavily influenced by its predecessors, even though they were developed for different reasons. This article describes the lessons we’ve learned from developing and operating them.
Rishiyur S. Nikhil - Abstraction in Hardware System Design
The history of software engineering is one of continuing development of abstraction mechanisms designed to tackle ever-increasing complexity. Hardware design, however, is not as current. For example, the two most commonly used HDLs date back to the 1980s. Updates to the standards lag behind modern programming languages in structural abstractions such as types, encapsulation, and parameterization. Their behavioral semantics lag even further. They are specified in terms of event-driven simulators running on uniprocessor von Neumann machines.
John R. Mashey - The Long Road to 64 Bits
Shakespeare’s words often cover circumstances beyond his wildest dreams. Toil and trouble accompany major computing transitions, even when people plan ahead. To calibrate “tomorrow’s legacy today,” we should study “tomorrow’s legacy yesterday.” Much of tomorrow’s software will still be driven by decades-old decisions. Past decisions have unanticipated side effects that last decades and can be difficult to undo.
© ACM, Inc. All Rights Reserved.