In the July/August 2008 issue of ACM Queue we published part one of a two-part discussion about the practice of software engineering. The goal was to gain some perspective on the tools, techniques, and methodologies that software engineers use in their daily lives. Three members of Queue’s editorial advisory board participated: Steve Bourne, Eric Allman, and Bryan Cantrill, each of whom has made significant and lasting real-world contributions to the field (for more information on each of the participants, see part one). In part two we rejoin their conversation as they discuss XP (Extreme Programming) and Agile.
ERIC ALLMAN We’ve talked a fair amount about tools, design methodologies, and a lot about debuggers, but we haven’t really talked much about programming practices and styles. These days you probably have to be dead not to have heard about XP and Agile. I’m interested in XP, and have been for a while. One of the things that interests me, though, is that it doesn’t really look like there’s much new in it. A lot of the principles have been around awhile—for example, unit testing? Gosh, when did I first hear about unit testing? I might have been out of diapers, but not by much.
Pair programming seems to be one of the most controversial principles. It’s also one of the most variable. For example, I’ve worked with people where we were together in a room and we’d go off and each be coding something and then come back and compare things—design reviews, that sort of thing. That’s not the same as having us both at one keyboard. I’m skeptical about having half as many keyboards as you do programmers. That seems pretty crazy and pretty expensive to me.
BRYAN CANTRILL I think Agile goes too far—for our team, anyway. Co-developing, where you’re sitting together all the time, is just too much. Does anyone make an office chair that has two seats in it? Actually, someone should. I think I just stumbled upon “the Agile chair.”
EA The next start-up.
BC Exactly. The Agile swivel chair—with two chairs mated, so you can both turn around at the same time.
EA That’s a great visual.
BC I think it’s the power of peer pressure, and Agile captures a little bit of that by having the peer seated right next to you—“should we be spacing out on the Internet right now or should we be working on this problem?” I always wonder how much of Agile’s productivity gains come from the fact that it’s really hard to space out on the Internet when you have someone right next to you.
EA XP also has certain ideas about coding standards, though they do not seem very controversial, and people have been doing this for a long time.
BC The coding standards are interesting. I’m a bit of a style fascist.
EA I am, too—and I’m right and you’re wrong.
BC You’re a bit more of a bellicose style fascist than I am. We might have the same style, who knows? Do you automatically enforce style?
EA We don’t have tools to do it. I actually got involved in coding styles back when I was working on the INGRES database project. What we found was that every time someone new joined the project, they spent the first couple of weeks reformatting the code to be in whatever format they liked, which has all kinds of implications. A waste of time is one of them. Another one is diffs don’t work anymore, so you can’t really see what they’re doing. So, we decided we had to have one style that would be used throughout the system. I worked with another guy on the project there and came up with the coding style that sometimes is called ANF (Allman Normal Form), which I have adapted a little bit over the years as new syntaxes have come along.
STEVE BOURNE Let me just take a slightly opposing point of view for a second. I’ve managed engineering shops in which we’ve enforced coding styles through tools so that people can read other people’s code, but that seems to be about it. It doesn’t actually make the code work, and it doesn’t actually make it structured well, either, so what’s the point?
BC The point is much larger than readability. First of all, readability has an impact on correctness. If I can read code, I can see things that are wrong with it.
We believe that on the eighth day, God created 80 columns, and ye shall never exceed 80 columns. How does being fascist about 80 columns affect anything? Well, it actually affects the way you write code. When you need to fit within 80 columns, having a 75-character variable name is probably not such a good idea. It’s probably a waste of columns, and we have eight character tab stops. When you automatically enforce that, you are forcing a refactoring to occur. You identify the stuff that should probably be offloaded to a separate function or method. One of the things that I really bridle at is exceeding 80 columns—bridle being a euphemism for either self-destruct or, I don’t know, go on a berserker rage.
EA Remember your blood pressure meds.
BC Thank you. Fortunately I’m medicated today. When you allow people to go into 133 columns, or longer, you allow a kind of laziness where someone can load a single line up with so much information or get too deeply nested. Eighty columns and eight-column tab stops doesn’t really allow more than four levels of indentation as a practical matter, and that is a good thing from a code structure perspective.
SB I like the principles that you just described because they actually produce, on average, reasonable results, as opposed to the crude tools we have for forcing people down these channels.
BC Right, and I think to a certain degree, it doesn’t matter which style you pick.
EA Up to a point.
BC Eric, don’t you guys do an interesting thing with bracing IF statements?
EA That’s the most controversial part of it. The argument for putting the open brace on the line with the IF statement used to be because that way you can fit more on a screen. Today monitors are big, so that’s just not an issue. But I’ve never felt that that was a good argument. If you can drop your eye and see the curly braces line up, it actually makes it easier to see the nesting.
BC I agree with you. We don’t actually enforce it—in fact, it would be a violation of our style to brace the IF on a separate line—but my personal style is to use braces if I have in the body of an IF statement a conditional that is more than one line. You can split a single line of C over two lines, obviously, and if I do that, I always brace it because it does allow your eye to see the code-flow.
SB My eye is not very good at that, either. The number of bugs I’ve had to remove from code because the braces weren’t in the right place is more than I care to mention. Of course, the answer to that is to have tools. If you look at something like Python, it won’t even compile a program if it’s not laid out properly.
BC That’s interesting. Python does go to an extreme.
SB I wrote some Python code without the tools to support your intent, and it’s actually quite difficult. A lot of other people find it quite easy, so I’m probably in the minority.
EA I have to admit I find it jarring. All my life I’ve been using this one style.
SB I don’t mean to be too critical—there are many things about Python that I like. It has some language features that somebody, presumably, actually thought about. So I quite like a lot of it, but the idea that they went back to Fortran layout was surprising.
BC Although it’s perhaps heresy to say it these days, I’m not enamored with Python for a couple of reasons. One is I think there’s a difference between having a style that you enforce and having white space be relevant to the language.
EA One of the big mistakes I made with sendmail was using tabs as active characters in the config file. The reason I did it was because make files were exactly the same way, so I just kind of emulated that idea. Then I talked to [make creator] Stu Feldman and he said, “The biggest mistake I made in make was using tabs as active characters.”
SB The point I was making—and I don’t disagree with you—is that there are some guidelines that improve the code, but my question is, how much does this actually help you structure the code so you can see the relationships between the functions that you’re writing and deduce what happens if you make a change? Those seem to be the important questions about code, not so much “Can I read it? Can I write it?”
BC “Can I read it? Can I write it?” are important questions, especially if you’re in an environment where you may need to modify someone else’s code and vice versa.
SB I agree, but there are higher-level structural issues in code that I don’t see that helping with, and they are the ones that are actually, in my opinion, more important than what I’m calling these lower-level issues.
BC An interesting metric would be how many lines of code you read for every line that you write. In my case, I read a lot of code. Even when I’m writing new code, I’m reading a lot of code to figure out how to write that code and how to structure it. So for me, the style gives you the parameters in which you develop new code, and I think it does actually help.
EA I believe Fred Brooks said something about no silver bullet [“No Silver Bullet—Essence and Accidents of Software Engineering,” 1986], and coding standards are not silver bullets, but I think they are a reasonable part of a program to produce clean code. It doesn’t matter, as Bryan points out, if it’s only one person who’s ever going to be reading the code. Who cares? But when was the last time there was only one person involved in any kind of code that made it out of the room?
BC It still matters with one person because it’s really not just one person. It’s multiple people—namely, it is you at different times of your life. I’ve written lots of code where I’m the only one who will ever edit it, and if I die, that source will no longer be edited at all. The problem is, I’m editing it at different times of my life, and the state that I may have once had on the problem falls out of my head; and when I go back in, it has been written by a different person—it has been written by the “past-me.” So, I think even for one person, style is important.
SB I’ll amplify that. When I’m writing code, I’m making deals with myself, but I’m two different people in different parts of the code. That’s what you do when you write code. When you abstract something out, you’re making a deal with yourself: here’s what it’s going to do and here’s the interface. Then you put your other hat on, and you’re now the client of that code that you just wrote for yourself. I think that’s an important distinction to make when you’re writing code.
EA Another element of XP that is perhaps a little controversial—and it’s not really about coding and to me it’s one of those mythical beasts—is the 40-hour workweek. I certainly do believe that programmers should not be working 80-hour weeks all the time, but management, which usually means the CFO, loves working programmers for 80-hour weeks, week in and week out. You can’t be productive that way in the long term.
BC No, not at all. But software development is a series of sprints, and the marathon is really only a composition of sprints.
EA Yes, it’s a series of sprints, but why is there never a rest break between the sprints? How do you convince management that they have to give you some time to chill out?
BC By writing great software. We’ve been able to structure the environment here because enough of us have a track record of writing great software that management knows to leave us alone.
You’ve also got to be willing to schedule yourself. If you want to get to the point where you have some flexibility to pursue things, you need to be as transparent and as honest about your schedules as possible, because—just like Wall Street—more than actually wanting results of a certain level, management really wants predictability.
They want to know if it’s going to deliver six months later than you thought it was going to deliver. In the abstract, it’s not like that six months, in terms of the time to market, is going to cost us the business. In some cases maybe it would, but in most cases it’s not going to. If I went to a customer and promised this is going to be finished in June, and now it’s not going to be finished until December, then you’ve impacted the relationship with that customer. But as long as you’re predictable, I think you can get that time for yourself.
In terms of how you convince management, you don’t treat scheduling as a kind of tedious exercise, and you don’t tell management what they want to hear. A lot of engineers think that the best thing to do is commit to the most aggressive date possible. No, don’t do that. You should commit to a date that makes sense and try as hard as you can to meet that date. As soon as you know you’re not going to meet that date, you want to make that clear and slip the schedule accordingly. If you do that, then I think you can get the time.
SB I don’t know if it’s in the Extreme Programming dictionary or not, but the simple version of this is don’t commit to the date until you’ve written the prototype, because you don’t really know what you’re doing until you’ve got something you know you can build. Even then, even if you do commit to the date, it’s important to manage expectations after that to make sure that if the date is going to change, you say the date is going to change.
BC Steve, you hit on a very important point, which is that the transparency of schedule actually interrelates with your development methodology. If you use a waterfall methodology, you’re much more likely to slip because there’s so much that’s unforeseen. When you’re using rapid prototyping, and you’ve got an idea of what the problem space is, you’re much more likely to come up with a schedule that you’re going to hit.
EA As I understand it, XP uses rapid prototyping, but it’s called “spike solutions.” I’m also reminded of my past where we always said, “Prototypes are great...and they always get shipped to customers.”
BC Which is the other half of the coin. When you do rapid prototyping, management sees a prototype and gets very excited.
EA They say, “Good, it’s working. Sell it.”
BC Or they say, “You showed me a prototype eight months ago. What have you been doing for the last eight months?” You say, “Well, I’ve been doing the 90 percent of the project that you don’t necessarily care about because it doesn’t impact the prototype that you see, but it does impact the quality of the product you deliver.”
Management needs to realize that when they see a prototype they still need to trust the schedule that they’re getting out of the engineer. They should view the prototype as a way of knowing that they’re on the right track and as something that they can talk about with customers.
EA I think you’re assuming a pretty enlightened management. Unfortunately, that has not been my experience in most of the real world.
There’s another XP tenet that I think ties into this, which is to make short software iterations. Two weeks is the one that you usually hear. That sounds great to me for small projects, but I’m not sure how you can, for example, put SMP (symmetric multiprocessing) into the kernel in two weeks.
BC We absolutely do it. In fact, we’ve got as big a software project as you can find out there, and we operate on a two-week model. Every two weeks, you cut a build. You don’t do SMP in the kernel in two weeks, but you have a bringover/modify/merge model for your source base. DTrace was integrated into a Solaris build that way. We didn’t do DTrace in two weeks, but what we did do is have a DTrace gate that was tracking the Solaris gate.
Every two weeks, we would pull down the changes into our DTrace gate, so we were tracking it for the two-and-a-half years that DTrace was in development. Then as you get closer and closer, you’re obviously tracking that more and more tightly, and then you’re finally integrating.
But every two weeks, you’re cutting a build. Obviously, lots of software takes much longer than two weeks in development, but the key is that you are constantly cutting builds because those builds do force you to get to a certain level of quality.
EA I tend to agree with that, but those builds are necessarily going to have half-implemented features in them and pieces that aren’t completely tested.
BC Our mantra is production quality all the time. There would need to be a business case for a half-implemented feature to be in your source base. Why does the half-implemented feature need to be in my source base and not in a gate that’s tracking my source base? Sometimes there’s a good reason for that because I’m maintaining all these hooks and these subsystems that are changing and it’s killing me and I’ve got to integrate my hooks, but in most cases, you don’t actually need to have a half-implemented feature.
EA When you say a gate, are you talking about a branch?
BC Yes, I’m talking about, effectively, a branch that is kept in sync with your main trunk.
EA I would argue, though, that putting things out on a branch is cheating on some level because ultimately you have to merge it back in. One of the arguments for the two-week model is that there’s always code review and the changes will always be small enough that you can do a pretty quick code review.
Now you suddenly have this branch with a year’s work on it. Do you now say, “OK, next Tuesday, I’m going to merge that into the trunk”?
BC But in that branch, you code review as you go. Should you ever go more than two weeks between put-backs, to either the main repository or a branch? My answer to that would probably be only in rare circumstances. There have been times where I needed to go more than two weeks between put-backs.
I’m not putting back to the main trunk necessarily, I’m putting back to all my branches, which I will then put back to the main trunk in, say, two years. But if I’m going more than two weeks, I’ve got to have a whole lot of balls in the air, which is sometimes the case.
EA One of the things that I’ve felt for some time is the trunk should always build. Branches can be in partial states, but if you’ve got something half implemented, the worst case, of course, is when the trunk won’t even compile.
BC I think that branches should always build, the trunks should always build. Now the branches of a branch—that is to say, your own private work space—well, that isn’t built when you go home at night. Fine, because no one is using it.
SB I think one of the things we might want to touch on here is dependency management. You said something earlier that is really important: you need to be able to pull features out of a release if you find that for some reason they don’t work down the road. That gets really difficult, however, depending on what the dependency trees look like, so my question is, have you seen improvements in the tool base and in the IDEs for managing dependencies between the different components of the software that you’re building?
EA I’ve definitely seen great improvements—particularly considering that when I started programming, there was no support for it whatsoever. It was just human intelligence, or sometimes not so intelligent. There is support now. I still haven’t seen anything that actually solves the problem. It always gets something wrong or it depends on you to fill in something, but it’s a lot better.
SB Can you find listed dependencies, and do they have names, and do you know who owns them and when they’re going to change? Those seem to be all the important questions.
BC I think as a practical matter, you don’t need to know all those things. Our dependencies are always within the same source base, so there are very few dependencies. We have a luxury because we are the operating system, so for us there aren’t a whole lot of dependencies that are outside the source space. Most projects are going to have dependencies outside the source base—namely, the operating system, certain system libraries, and so on.
Within our group you need to be aware of your dependencies, you need to be aware of what else is going on, but if you have a model where you are constantly in sync with the principal trunk of whatever work everyone is engaged in, you will catch that breakage earlier rather than later.
So, for example, all of a sudden my code doesn’t work and I’ve got two weeks of changes that I need to review. I know it worked with the last build. It doesn’t work with this build. I’ve got no idea why. The failure mode is crazy. I can whittle the problem from one side and actually debug it. I can begin to whittle the problem from the other side, actually going through two weeks of changes and say, “Well, did the XML library change over the course of the last two weeks? Was there a change in the subsystem? Oh, wow, there was a change in the subsystem, let me look at the diffs for this; maybe I can zoom in on this really quickly.”
In some cases—we have had to do this from time to time—you start backing stuff out of your branch. You say, “I’m going to take the stuff that I synced up and rip out half of it and see if I solved the problem.” Those are extreme cases, but by having that two-week granularity it’s still tenable.
SB This has been a good conversation. It’s clear that the software we’ve been talking about in this discussion is mostly system software and really not that far up the food chain in terms of applications. There are doubtless a bunch of things in more layered software that we haven’t addressed here.
EA You’re absolutely correct. In some sense, what we’ve talked about may not be completely applicable to the usual business app or Web site. But I think, actually, a lot of it is transferable. The basic concepts of good hygiene on getting code written are not really any different for server code versus user-interface code.
The other closing remark I have is this: I think the interesting commonality that we have is that all three of us at some point in our careers wrote a tool that we needed for ourselves. I would say that was incredibly empowering. We solved interesting problems, and we came up with lasting solutions. So, when you have a problem for which you think, “Boy, it would be great if a tool did this”—especially given the plug-in architecture of things like Eclipse and NetBeans—go innovate it! Invent a tool that you need. You will do your best work and you will quickly prove to yourself that the time invested in those tools is time well spent. You’ll prove it to yourself and to your management, and you’re going to make the world a better place.
EA To that, though, I would add one thing, which is to avoid the “not-invented-here” mind-set. Don’t implement a tool that somebody has already implemented. A lot of people don’t do the research to see if there’s a tool out there.
BC Do the research; find the tool. A lot of that stuff is open source. Contribute to a tool that is halfway down a trajectory that you envision. But view your tooling as part of your responsibility as a software engineer.
Originally published in Queue vol. 6, no. 5—
see this item in the ACM Digital Library