The Yin and Yang of Software Development

July 14, 2008
Volume 5, issue 4

The Yin and Yang of Software Development

How infrastructure elements allow development teams to increase productivity without restricting creativity

MIKE VIZARD: Hello and welcome to another premium edition of the ACM Queuecast with your host Mike Vizard. This edition is sponsored by Parasoft, the leading provider of tools for automating the management of the software development process.

Joining me today is Sergei Sokolov, Solutions Manager, for C++ Plus at Parasoft, and we're going to talk about the yin and yang of application development and how to bring some structure to what's inherently a process driven by innovation.

Sergei, welcome to the show.

SERGEI SOKOLOV: Glad to be here Michael.

MV: So one of the things that comes to mind here is this whole concept of yin and yang when it's applying application development as everybody thinks it's all yin and there's no yang. There's no really structured approach to innovation because it's all left-brain activity in people's minds. So can you really apply some sense of structure to the application development process that really boosts productivity?

SS: It's a rather difficult question because there is a bit of history obviously in software development in that there are many people who think that software development is art and we actually have some folks at Parasoft who put it to me quite in these words.

At the same time, I think any creativtiy which software development inherently involves, can benefit from structure and the root of that is, software development is actually not all art. The thought process behind it is in fact art, but there is a lot of mechanical stuff that goes after knowledge has been acquired and then needs to be put into the software and the mechanics are writing code, building code, testing code, making sure somebody else understands the code you have written, and so forth.

And so while it's probably very difficult to get a structure into the thought process even though that's a separate topic, there's certainly quite a bit of structure that can be put into the implementation process once the thought has matured, and I think this is where the place of structure is in software development.

MV: Is there any way to come up with some simple steps that outline some things that people should be checking off on a regular basis as to here are some five or six best practices that you should be doing on a ... project to boost productivity?

SS: Opinions differ, but obviously if you are attempting to put structure in place, then it's better that you have a few simple steps so people could grasp that and reduce that to a plan as opposed to going on a wild goose chase.

Parasoft as an organization has been in the software business for 20 years. So we certainly have accumulated quite a bit of knowledge of the things that work and the things that don't and in fact, we have come up with a few ingredients if you will that do contribute to productivity and that do so through imposing some structure on those very mechanical aspects that frequently tend to bite people.

What we consider as an essential platform for productive development are five steps, or maybe you should say five components. They're based on our own experience -- as I said this was quite extensive -- as well as refer to some of the research that has gone on in the industry in the last, 20 years give or take, seeing that if people start doing something, and they try to get money for it, which is kind of the case with software, they try to find the ways to do it more economically, more efficiently, and more productively.

So understandably there's quite a bit of research.

So the things that we consider essential going back to that question are first of all the build or the basic ability to build your software on a daily basis. As I talk to customers, prospects, and just visitors at tech shows, it's a little bit disturbing to realize how many of the engineering groups do not have a regular build.

They think that as long as they can produce the software on demand, they're going to be fine, but that's not a controlled process.

So the basic build step, to build infrastructure, is the element number one. And we can talk in a bit more detail as far as what we're actually looking for in terms of regular builds.

Consistent coding helps a bit because this is the basic process of reducing thought into code. And consistent coding has two benefits, one is other people can read what you wrote because the software code is nothing more than captured knowledge, and if you want to communicate that knowledge to others, it's like writing and essays, and you'd better write in a way that other people can understand. And on the other hand, some basic idioms and best practices in the language, whether it's Java, C++, C Sharp, or whatever else that people use, if they can consistently use certain patterns than they would probably be better equipped to reduce their thought into code much faster because they would be using something that they're used to.

Once the code is written, the best form to check that it is correct or probably complete is code review. Engaging somebody else on the team to look over your shoulder in sort of figurative or a literal way and say, hey, have I done the right thing here?

So code review -- regular code review, consistent code review -- is our Step Number Three.

Once the code was kind of bliffed and that, okay, it seems like it's done fine, testing, as early testing as possible, gets you to the next level of productivity. What we have been finding, and it's not only our experience, it's been confirmed by numerous studies, that the practice of writing code composing the application, and then giving it to somebody else to test, is usually not that effective because what people find in that stage are relatively complex use case related bugs and tracking that to a couple of lines of code that can be changed to fix the effect is pretty involved and timeconsuming.

There are various graphical representations of the escalating cost the longer you wait with testing, and some say it's a factor of five, between fixing a problem when you code, versus fixing a problem, let's say at the late stages of QA, and some say it's a factor of 100 and of course, it may depend on the nature of the project and so forth, but nobody disputes that there is a difference.

So the earlier you test, the more economical you can make detecting the problems and actually the faster you can complete your code. I just came back from training at one of our large customers, and there was a fellow who was discussing unit testing, and I asked the audience who had done unit testing in the past, and there were a few folks who had actually been quite proficient at that, and they said that with unit testing, which is the form of early testing, they could clearly see that if they do it properly, they don't have any overtime when they complete the software. The software works and they can predict very well when it's going to come together.

To me that's a symptom of great productivity because you can estimate more correctly when you're going to be done, and you are sure that you're going to be done without working 12 or 16 hour days.

And finally, once we get over this topic of early testing, we get to automation of that through regressions. Again one somewhat misplaced notion of regression testing, that as of late, points to testing by the QA team late in the cycle when the software is ready and they do it for a given release. So Release 1 comes out, they do this testing cycle. They verify that the release works according to specs and then six months later, there is a Release 1.1, and they go through the same cycle towards the end of those six months, and they call it regression testing.

They make sure that Release 1.1 one needs to work like Release 1.0, and indeed does so.

And there's basically a six-month gap between these test sessions, and of course, a lot has gone on between these two sessions, and of course, if the functionality changed backtracking on why the functionality changed is quite difficult.

Our notion of regression testing is continuous regression testing, or nightly regression testing. So along with your build that we like to see done nightly, we also encourage people to have regression tests run nightly as well because that allows them to check the consistency of their code on a daily basis.

You can't come up with anything more productive than that because if somebody makes a mistake and some functionality is destroyed, then you know the next day that this has happened and not those two, three or four months down the road.

So the turnaround for addressing these effects obviously turns out to be a lot shorter than if such effects are discovered at the Eleventh Hour.

MV: So, that all sounds pretty fundamental. What prevents organizations from actually imposing that kind of structure? Is it just simple inertia? Or is there something about the nature of software development that maybe it's the way the teams work together that kind of hangs everybody up?

SS: This is probably the most important question about software development. What prevents people from doing things better? I think there is a fundamental stigma of change. People obviously have different backgrounds and unfortunately more often than not, given the history of software development, developers come from teams that haven't had structure.

I was very lucky and the first company that I started working for was a very small company, was just five people, after getting my Master's from University of Massachusetts, happened to have many of these best practices implemented. The nightly build, the regression tests, some of the early testing and code review.

So I was lucky to have been exposed to it from the early days, and I haven't really seen anything much different in other organizations that I worked for. Maybe that's how I selected the organizations that I worked for, maybe that's how they selected me, but not everybody's path resembles mine, and when there is a call to do something differently, people would always come up with why do I need to do something differently?

Oh, it actually looks like it's going to be more work. Which, let's say, if you do unit testing, it has the appearance of doing more work.

You haven't been doing testing before. Now you do need to do testing. You have to write some more code. You have to check the results. Oh, we're going to be late with the project. There's, of course, nothing further from the truth because the investment of time that you put into, in this example, early testing, you get back with a huge margin at the end when you don't have to work 15 hours a day and as I said, there are numerous examples in the industry when it indeed has happened, but I think just the basic resistance of change and inertia is one of the fundamental obstacles to consistently moving in that direction.

MV: And yet, there's a real cost here because the vast majority of application development projects come in late, and when they're late, there's usually added cost on top of that. So, is there some way of measuring that or thinking about whether your ROIs are out, having a little more structure around the application development process?

SS: Truthfully, I haven't seen a single reliable ROI formula that can be applied. It doesn't mean that ROI studies aren't available. Right? Let's say if we talk to prospects and they request an ROI, the first approach that we used to take was, well, let's construct a complicated spreadsheet and try to plug in some parameters with cost per defect and the expected number of defects and come up with some number that is in, I don't know, $80,000 per project or $200,000 per project, and then you work backwards from there.

I think there is a better approach and the better approach is case studies.

Because case studies are not models. They are real things. And when we have a case study like our recent one from NEC in the Philippines, they are a group that has implemented C++ tests and have reported roughly a third in productivity improvement over the course of the project. That's the real-life example of a ROI, and while for a given organization there may be some variation -- maybe they will get 25 percent, maybe they'll get 40, maybe somewhere around that -- the fact that somebody did something right and they got 30 percent improvement in productivity is indisputable.

So I think people need to change their approach to ROI. They need to take the case studies as proof that good things happen, and then based on that faith try it in their own environment and just take careful metrics and come up with their own ROI assessment.

MV: Given all that, how does the Parasoft software actually work in practice? What are the components? What are the elements?

SS: Parasoft Software provides automation to most of these practices that we talk about -- the code analysis, the code review, the early testing and regression testing and the reporting on all of the above.

And of course depending on which of these practices are more important to a given organization, we can structure the software differently, but at a basic level, each of our tools is a system, that comprises components, addressing these specific steps.

So let's say for language tools, of which we have a C++ test, 4C and C++ J tests for Java, JSP, and dot-tests for the dot-net environment, all these tools are very similarly structured. They have a code analysis component that consists of a large number of coding rules that are derived from best practices. And these help establish consistent coding patterns with the team.

There is a more sophisticated form of code analysis which relies on path stimulation and tries to identify possible errors based on different data that the program can see in real life. So this is all at the level of code analysis.

We have a component for automation of code review. Obviously it doesn't automatically review your code and give you the results. That would be a pretty difficult problem, but again, what we try to do is enable the creative aspect by automating the mundane and that's pretty much the corner theme of all of our tools. Right? There's the repetitive element that if you speed up, you basically contribute to productivity.

And so, for code review, we have the software that tracks the code changes, displays the differences to developers, allows them to annotate their comments and then tracks the status for the code review issues until they're completed.

All of these things have been tenuous in code reviews because they don't tend to be done very consistently, and if something is not done consistently, productivity typically suffers.

We support early testing through our tools as unit testing and component testing. Whatever the people choose to use depending on the state of their development. We have auto-generators for tests that can generate tests based on code structure and put all the test infrastructure in place -- the test drivers, the individual tests, in a structured manner.

We have coverage analysis that basically measures how well the tests cover the code and direct people to improve the tests where necessary.

So again that basically takes care of the mundane. About half the time, I would imagine for implementing unit testing, goes into running the test drivers and making sure it's all structure that can be re-run from different machines and so forth, and the tools take care of it, leaving more time for the folks to actually do the tests themselves which is analyze use cases and do the appropriate calls in the software, corresponding to their architecture and algorithms.

And last but not least as far as the regression capability goes, that's automated with all of our tools. So whatever tests you have, you can run through, can handle an interface on a nightly basis and get the reports through HTML or into our reporting system that lets you keep track of how well you're marching along on a daily basis.

So we indeed try to cover all of the steps that we think are fundamental.

MV: How difficult is it to set something like that up? Because would that be something that people would struggle with or is it relatively straightforward?

SS: No, it is relatively straightforward. In fact, our language tools at this point are, full plug-ins to the two most significant IDs out there, the Visual Studio and Eclipse, and from that point of view there's basically no threshold for people. If you're working in Visual Studio, we will work on the project or a solution once it's loaded. If you are writing Java in Eclipse then basically we're right there as a plug-in, and right after you've written a new class, you can do analysis, you can do unit testing of that class, you can save the test and put the whole thing in source control.

Eclipse is becoming actually a driving ID in embedded development, so there is a very noticeable consolidation of solutions and vendors towards that, and we very much welcome that because we can now provide one solution for multiple environments, and the embedded industry is known for the multiplicity of environments. That will work for most people and bring them the value in the same sense that if they have something in their ID, they get the plug-in, in the form of let's say C++ test, and they can get to use this plug-in right away.

MV: And what are the costs associated with building something like that? Is it brutally expensive or is it something that just about any organization can afford to do?

SS: Very interesting question, and there are actually many components to the price of implementing such a solution in an organization. I previously worked in the electronic design automation industry, which produces software for design of electronic chips, and it was wide knowledge that it's not the cost of the tools that determine the price of the whole package but rather the cost of stitching all the tools together.

So the cost of implementing all of the tools from different vendors frequently exceeded the cost of these tools by a factor of two or three.

Well, with software tools and the IDs and the standardization of the desktop, that fortunately is not the case with regular software development. So in terms of cost to organizations, beyond just acquiring Parasoft Software or somebody else's software, if that helps the costs. There are costs with maintenance of the infrastructure and maintaining the flow and keeping an eye on the process that can't be ignored. But this is relatively minor compared to the cost of software.

At least based on the response we get, our software is affordable. It's not cheap because the good things aren't cheap, but at the same time, we haven't been told that we're extraordinarily expensive either.

MV: So what would be your best advice to people who want to go and bring a little structure to the application development processes? Just to get things off the ground and rolling and how to break that first piece of inertia?

SS: I think the best advice I can give is don't try to make everything work on Day 1 because nothing happens overnight. Even within Parasoft where obviously we've had years of experience of integrating these practices and the development flow before we market them, we see a very gradual adoption curve and that has to do with -- the development life is multidimensional, and so you have to determine the steps that will bring the biggest bang for the buck at the current stage, and for different people, it may be different.

Depends on the state of software, whether it's the software that's being written from scratch, whether it's the software that's being incremented, whether it's maintenance of legacy code, and so the paths may be different.

My standard recommendation for everybody is if you don't have a nightly build, get it in place. Seeing that unless you can build your software on a regular basis, nothing else matters.

Past that point, basically, the additional four steps that we advocate are laid out more or less in the order that we recommend to use them, but there certainly can be variations.

For example, if it's a lean organization, they don't have a lot of funds to play with, then just basically doing a regular build and establishing a code review process will get them more than half the way to producing the defect-free software because code review as such if done correctly on average was analyzed to find 60 to 70 percent of all defects.

If that practice is in place and there's desire to bring on more automation, then certainly code analysis is the next step because it's typically very nonintrusive, it can be done on a nightly basis without much of user intervention and it helps the code review. So you're putting in a component that makes the best bug-finding exercise, the code review, more effective by focusing it more, getting rid of concerns such as are you compliant with coding guidelines?

Are you writing your C++ in the appropriate way and concentrating on the algorithms? And then move towards testing.

MV: This has been another premium edition of the ACM Queuecast sponsored by Parasoft, the leading provider of tools for automating the management of the software development process. For more information, please check out www.Parasoft.com, and I'd like to thank Sergei for sharing his knowledge and his advice and Sergei, we wish you the best of luck.

SS: Thank you very much. It was my pleasure to be on this program.

Originally published in Queue vol. 5, no. 4—
Comment on this article in the ACM Digital Library