Performance

Vol. 4 No. 1 – February 2006

Performance

Interviews

A Conversation with Jarod Jenson

One of the industry's go-to guys in performance improvement for business systems is Jarod Jenson, the chief systems architect for a consulting company he founded called Aeysis. He received a B.S. degree in computer science from Texas A&M University in 1995, then went to work for Baylor College of Medicine as a system administrator. From there he moved to Enron, where he played a major role in developing EnronOnline. After the collapse of Enron, Jenson worked briefly for UBS Warburg Energy before setting up his own consulting company. His focus since then has been on performance and scalability with applications at numerous companies where he has earned a reputation for quickly delivering substantial performance gains. Recently, Jarod made a splash at JavaOne: he ran a booth to which attendees could bring their Java applications, and he guaranteed application performance improvements in under an hour.

A Conversation with Jarod Jenson

Pinpointing performance problems

One of the industry’s go-to guys in performance improvement for business systems is Jarod Jenson, the chief systems architect for a consulting company he founded called Aeysis. He received a B.S. degree in computer science from Texas A&M University in 1995, then went to work for Baylor College of Medicine as a system administrator. From there he moved to Enron, where he played a major role in developing EnronOnline. After the collapse of Enron, Jenson worked briefly for UBS Warburg Energy before setting up his own consulting company. His focus since then has been on performance and scalability with applications at numerous companies where he has earned a reputation for quickly delivering substantial performance gains. Recently, Jarod made a splash at JavaOne: he ran a booth to which attendees could bring their Java applications, and he guaranteed application performance improvements in under an hour.

Leading the questioning of Jenson is Kirk McKusick, well known for his extensive work in the open source software community, primarily with FreeBSD, although he points out that he has worked with other systems, including Solaris. McKusick has a consultancy in Berkeley, where he also teaches and writes. He is a past president of the Usenix Association and is a member of the editorial board of ACM Queue.

Kode Vicious

Gettin' Your Kode On

Dear KV, Simple question: When is the right time to call the c_str() method on a string to get the actual pointer?

Gettin’ Your Kode On

A koder with attitude, KV answers your questions. Miss Manners he ain’t.

Another year is upon us and we are happy to have Kode Vicious still ranting against the ills of insecure programming, insufficient commenting, and numerous other forms of koding malpractice. Yet despite his best efforts, the bittersweet truth is that these problems are not going away anytime soon, and therefore should continue to provide ample fodder for future KV columns. Oh, to live in a world that doesn’t need KV’s advice—or doctors, for that matter.

Dear KV,
Simple question: When is the right time to call the c_str() method on a string to get the actual pointer?
Hanging by a String

by George Neville-Neil

Articles

Hidden in Plain Sight

In December 1997, Sun Microsystems had just announced its new flagship machine: a 64-processor symmetric multiprocessor supporting up to 64 gigabytes of memory and thousands of I/O devices. As with any new machine launch, Sun was working feverishly on benchmarks to prove the machine’s performance. While the benchmarks were generally impressive, there was one in particular—an especially complicated benchmark involving several machines—that was exhibiting unexpectedly low performance. The benchmark machine—a fully racked-out behemoth with the maximum configuration of 64 processors—would occasionally become mysteriously distracted: Benchmark activity would practically cease, but the operating system kernel remained furiously busy. After some number of minutes spent on unknown work, the operating system would suddenly right itself: Benchmark activity would resume at full throttle and run to completion. Those running the benchmark could see that the machine was on course to break the world record, but these minutes-long periods of unknown kernel activity were enough to be the difference between first and worst.

Hidden in Plain Sight

Improvements in the observability of software can help you diagnose your most crippling performance problems.

BRYAN CANTRILL, SUN MICROSYSTEMS

In December 1997, Sun Microsystems had just announced its new flagship machine: a 64-processor symmetric multiprocessor supporting up to 64 gigabytes of memory and thousands of I/O devices. As with any new machine launch, Sun was working feverishly on benchmarks to prove the machine’s performance. While the benchmarks were generally impressive, there was one in particular—an especially complicated benchmark involving several machines—that was exhibiting unexpectedly low performance. The benchmark machine—a fully racked-out behemoth with the maximum configuration of 64 processors—would occasionally become mysteriously distracted: Benchmark activity would practically cease, but the operating system kernel remained furiously busy. After some number of minutes spent on unknown work, the operating system would suddenly right itself: Benchmark activity would resume at full throttle and run to completion. Those running the benchmark could see that the machine was on course to break the world record, but these minutes-long periods of unknown kernel activity were enough to be the difference between first and worst.

Given the importance of both the benchmark and the machine, Sun’s top development and support engineers were called in, and work proceeded on the problem around the clock and around the globe. Beginning with the initial symptom of the problem—high lock contention in the networking stack—progress on the problem was very slow and painful. Based on available data, a hypothesis would be formed, but because there was no way to explore the hypothesis without gathering additional data, custom data-gathering kernel modules had to be created and loaded. Creating these modules was a delicate undertaking. The entire cycle of hypothesis creation through instrumentation and analysis was taking hours—and that was assuming that no mistakes were made. Mistakes were painful, as any error in creating the custom module would result in a reboot—and reboots of this particular machine and configuration took on the order of 90 minutes.

by Bryan Cantrill

A High-Performance Team

You work in the product development group of a software company, where the product is often compared with the competition on performance grounds. Performance is an important part of your business; but so is adding new functionality, fixing bugs, and working on new projects. So how do you lead your team to develop high-performance software, as well as doing everything else? And how do you keep that performance high throughout cycles of maintenance and enhancement?

High-Performance Team

From design to production, performance should be part of the process.

PHILIP BEEVERS, ROYALBLUE

You work in the product development group of a software company, where the product is often compared with the competition on performance grounds. Performance is an important part of your business; but so is adding new functionality, fixing bugs, and working on new projects. So how do you lead your team to develop high-performance software, as well as doing everything else? And how do you keep that performance high throughout cycles of maintenance and enhancement?

TELL YOUR PEOPLE

If performance is important to your business, your employees need to know. Throughout your development team, you’re looking for a balanced performance culture—not the kind where developers spend hours profiling and optimizing deep into the night, missing deadlines as a result, but the kind where people feel within their rights to comment that a particular feature is a bit slow. Developers need to believe that, fundamentally, their product is high-performance, and that something is wrong if it’s not.

by Philip Beevers

Curmudgeon

It Isn't Your Father's Realtime Anymore

Isn’t it a shame the way the term realtime has become so misused? I’ve noticed a slow devolution since 1982, when realtime systems became the main focus of my research, teaching, and consulting. Over these past 20-plus years, I have watched my beloved realtime become one of the most overloaded, overused, and overrated terms in the lexicon of computing. Worse, it has been purloined by users outside of the computing community and has been shamelessly exploited by marketing opportunists. Adding to the humiliation, there is not even consistency of spelling: it variously appears as realtime (ACM Queue’s style preference), real-time (the author’s style preference), or real time.

It Isn’t Your Father’s Realtime Anymore

The misuse and abuse of a noble term

Phillip Laplante, Penn State University

Isn’t it a shame the way the term realtime has become so misused? I’ve noticed a slow devolution since 1982, when realtime systems became the main focus of my research, teaching, and consulting. Over these past 20-plus years, I have watched my beloved realtime become one of the most overloaded, overused, and overrated terms in the lexicon of computing. Worse, it has been purloined by users outside of the computing community and has been shamelessly exploited by marketing opportunists. Adding to the humiliation, there is not even consistency of spelling: it variously appears as realtime (ACM Queue’s style preference), real-time (the author’s style preference), or real time.

WHAT IS REALTIME?

To understand my frustration, consider the history of this noble term. Realtime appears to have first been used in conjunction with Project Whirlwind, a Navy flight simulator begun in 1947. During the 1950s and ’60s realtime systems were primarily those that were distinguished from batch-processing systems or online (nonbatch) systems for use in controlling embedded systems. Indeed, in one of the first textbooks on the subject, Design of Real-Time Computer Systems (Prentice Hall, 1967), James Martin described realtime as “[a system] which controls an environment by receiving data, processing them, and taking action or returning results sufficiently quickly to affect the functioning of the environment at that time.” For the next 30 years or so, the term realtime was applied only to industrial control, weapons systems, and other exotic applications, all of which were essentially characterized as those where inability to meet deadlines led to failure—usually a spectacularly catastrophic one.

by Phillip Laplante

Articles

Modern Performance Monitoring

The modern Unix server floor can be a diverse universe of hardware from several vendors and software from several sources. Often, the personnel needed to resolve server floor performance issues are not available or, for security reasons, not allowed to be present at the very moment of occurrence. Even when, as luck might have it, the right personnel are actually present to witness a performance “event,” the tools to measure and analyze the performance of the hardware and software have traditionally been sparse and vendor-specific. Because few real Unix cross-platform toolkits exist, there is no standard method for administrators to accurately view an event, record it for later analysis, or share it with additional qualified personnel for an efficient resolution.

Modern Performance Monitoring

Today’s diverse and decentralized computer world demands new thinking about performance monitoring and analysis.

MARK PURDY, PURSOFT

The modern Unix server floor can be a diverse universe of hardware from several vendors and software from several sources. Often, the personnel needed to resolve server floor performance issues are not available or, for security reasons, not allowed to be present at the very moment of occurrence. Even when, as luck might have it, the right personnel are actually present to witness a performance “event,” the tools to measure and analyze the performance of the hardware and software have traditionally been sparse and vendor-specific. Because few real Unix cross-platform toolkits exist, there is no standard method for administrators to accurately view an event, record it for later analysis, or share it with additional qualified personnel for an efficient resolution.

TRACKING DOWN LATENCY

The first step in studying the performance of a Unix server environment is to track down any application’s transaction latency from the user’s request event to the ultimate screen paint. The next step is to diagram the server topology. To do this, many elements come into question: the network topology, database update (and locks), disk arrays, process scheduling, CPU performance, memory affinity, and driver/interrupt service times. You can then use a set of performance analysis tools to study the hardware and software environment along the complete path of the measured transaction latency. This step-by-step breakdown of each part of a user transaction should reveal the elements where latencies are experienced. A performance analysis toolkit should measure these elements and the performance degradation associated with them.

by Mark Purdy

Performance Anti-Patterns

Performance pathologies can be found in almost any software, from user to kernel, applications, drivers, etc. At Sun we’ve spent the last several years applying state-of-the-art tools to a Unix kernel, system libraries, and user applications, and have found that many apparently disparate performance problems in fact have the same underlying causes. Since software patterns are considered abstractions of positive experience, we can talk about the various approaches that led to these performance problems as anti-patterns—something to be avoided rather than emulated.

Performance Anti-Patterns

Want your apps to run faster? Here’s what not to do.

BART SMAALDERS, SUN MICROSYSTEMS

Performance pathologies can be found in almost any software, from user to kernel, applications, drivers, etc. At Sun we’ve spent the last several years applying state-of-the-art tools to a Unix kernel, system libraries, and user applications, and have found that many apparently disparate performance problems in fact have the same underlying causes. Since software patterns are considered abstractions of positive experience, we can talk about the various approaches that led to these performance problems as anti-patterns—something to be avoided rather than emulated.

Some of these anti-patterns have their roots in hardware issues, some are the result of poor development or management practices, and some are just common mistakes—but we’ve seen them all repeatedly. In this article we discuss these mistakes: what causes them, how to find them, and how best to avoid them.

by Bart Smaalders