In December 1997, Sun Microsystems had just announced its new flagship machine: a 64-processor symmetric multiprocessor supporting up to 64 gigabytes of memory and thousands of I/O devices. As with any new machine launch, Sun was working feverishly on benchmarks to prove the machine’s performance. While the benchmarks were generally impressive, there was one in particular—an especially complicated benchmark involving several machines—that was exhibiting unexpectedly low performance. The benchmark machine—a fully racked-out behemoth with the maximum configuration of 64 processors—would occasionally become mysteriously distracted: Benchmark activity would practically cease, but the operating system kernel remained furiously busy. After some number of minutes spent on unknown work, the operating system would suddenly right itself: Benchmark activity would resume at full throttle and run to completion. Those running the benchmark could see that the machine was on course to break the world record, but these minutes-long periods of unknown kernel activity were enough to be the difference between first and worst.
You work in the product development group of a software company, where the product is often compared with the competition on performance grounds. Performance is an important part of your business; but so is adding new functionality, fixing bugs, and working on new projects. So how do you lead your team to develop high-performance software, as well as doing everything else? And how do you keep that performance high throughout cycles of maintenance and enhancement?
Performance pathologies can be found in almost any software, from user to kernel, applications, drivers, etc. At Sun we’ve spent the last several years applying state-of-the-art tools to a Unix kernel, system libraries, and user applications, and have found that many apparently disparate performance problems in fact have the same underlying causes. Since software patterns are considered abstractions of positive experience, we can talk about the various approaches that led to these performance problems as anti-patterns—something to be avoided rather than emulated.
The modern Unix server floor can be a diverse universe of hardware from several vendors and software from several sources. Often, the personnel needed to resolve server floor performance issues are not available or, for security reasons, not allowed to be present at the very moment of occurrence. Even when, as luck might have it, the right personnel are actually present to witness a performance “event,” the tools to measure and analyze the performance of the hardware and software have traditionally been sparse and vendor-specific. Because few real Unix cross-platform toolkits exist, there is no standard method for administrators to accurately view an event, record it for later analysis, or share it with additional qualified personnel for an efficient resolution.
One of the industry's go-to guys in performance improvement for business systems is Jarod Jenson, the chief systems architect for a consulting company he founded called Aeysis. He received a B.S. degree in computer science from Texas A&M University in 1995, then went to work for Baylor College of Medicine as a system administrator. From there he moved to Enron, where he played a major role in developing EnronOnline. After the collapse of Enron, Jenson worked briefly for UBS Warburg Energy before setting up his own consulting company. His focus since then has been on performance and scalability with applications at numerous companies where he has earned a reputation for quickly delivering substantial performance gains. Recently, Jarod made a splash at JavaOne: he ran a booth to which attendees could bring their Java applications, and he guaranteed application performance improvements in under an hour.
Isn’t it a shame the way the term realtime has become so misused? I’ve noticed a slow devolution since 1982, when realtime systems became the main focus of my research, teaching, and consulting. Over these past 20-plus years, I have watched my beloved realtime become one of the most overloaded, overused, and overrated terms in the lexicon of computing. Worse, it has been purloined by users outside of the computing community and has been shamelessly exploited by marketing opportunists. Adding to the humiliation, there is not even consistency of spelling: it variously appears as realtime (ACM Queue’s style preference), real-time (the author’s style preference), or real time.
Dear KV, Simple question: When is the right time to call the c_str() method on a string to get the actual pointer?