A Chance Gardener:
Harvesting open-source products and planting the next crop
It is a very natural progression for a company to go from being a pure consumer of open source, to interacting with the project via patch submission, and then becoming a direct contributor. No one would expect a company to be a direct contributor to all the open-source projects it consumes, as most companies consume far more software than they would ever produce, which is the bounty of the open-source garden. It ought to be the goal of every company consuming open source to contribute something back, however, so that its garden continues to bear fruit, instead of rotting vegetables.
A Lesson in Resource Management:
Waste not memory, want not memory—unless it doesn’t matter
Dear KV, I’ve been reworking a device driver for a high-end, high-performance networking card and I have a resource allocation problem. The devices I’m working with have several network ports, but these are not always in use; in fact, many of our customers use only one of the four available ports. It would greatly simplify the logic in my driver if I could allocate the resources for all the ports -- no matter how many there are -- when the device driver is first loaded into the system, instead of dealing with allocation whenever an administrator brings up an interface.
A License to Kode:
Code-scanning software is expensive and I’m not sure it’s worth it. What do you think?
While it’s sometimes tempting to blame the coders, the seeds of many problems are sown well before any lines of code (dodgy as they may be) have been written. Everything from the choice of tools to the choice of a software license can affect the quality, usability, and commercial potential of a product. This month Kode Vicious takes a step away from coding technique and addresses some of these tough decisions with which developers must grapple.
A Nice Piece of Code:
Colorful metaphors and properly reusing functions
In the last installment of Kode Vicious (A System is not a Product, ACM Queue 10 (4), April 2012), I mentioned that I had recently read two pieces of code that had actually lowered, rather than raised, my blood pressure. As promised, this edition’s KV covers that second piece of code.
A Paucity of Ports:
Debugging an ephemeral problem
I’ve been debugging a network problem in what should be a simple piece of network code. We have a small server process that listens for commands from all the other systems in our data center and then farms the commands out to other servers to be run. For each command issued, the client sets up a new TCP connection, sends the command, and then closes the connection after our server acknowledges the command.
A System is not a Product:
Stopping to smell the code before wasting time reentering configuration data
Every once in a while, I come across a piece of good code and like to take a moment to recognize this fact, if only to keep my blood pressure low before my yearly medical checkup. The first such piece of code to catch my eye was clocksource.h in Linux. Linux interfaces with hardware clocks, such as the crystal on a motherboard, through a set of structures that are put together like a set of Russian dolls.
APIs with an Appetite:
Time for everyone’s favorite subject again: API design. This one just doesn’t get old, does it? Well, OK, maybe it does, but leave it to Kode Vicious to inject some fresh insight into this age-old programming challenge. This month KV turns the spotlight on the delicate art of API sizing.
Dear KV, This may sound funny to you, but one of my co-workers recently called one of my designs fat. My project is to define a set of database APIs that will be used by all kinds of different front-end Web services to store and retrieve data. The problem is that a one-size-fits-all approach can’t work because each customer of the system has different needs. Some are storing images, some are storing text, sound, video, and just about anything else you can imagine. In the current design each type of data has its own specific set of APIs to store, search, retrieve, and manipulate its own type of data.
Advice to a Newbie:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I am new to programming and just started reading some books about programming, particularly C++ and Visual Basic. I truly enjoy programming a lot, to the extent that for the past couple of months I have never missed a day without writing some code. My main concern now is what the world holds for programmers. If someone is called a programmer (i.e., professionally), what will he or she really be programming? As in, will you always be inventing new software or what, really? This is mainly in the case of someone who will not be working for someone else.
Avoiding Obsolescence:
Overspecialization can be the kiss of death for sysadmins.
Dear KV, What is the biggest threat to systems administrators? Not the technical threat (security, outages, etc.), but the biggest threat to systems administrators as a profession?
Beautiful Code Exists, if You Know Where to Look:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
I’ve been reading your rants for a while now and I can’t help asking, is there any code you do like? You always seem so negative; I really wonder if you actually believe the world of programming is such an ugly place or if there is, somewhere, some happy place that you go to but never tell your readers about.
Bound by the Speed of Light:
There’s only so much you can do to optimize NFS over a WAN.
I’ve been asked to optimize our NFS (network file system) set up for a global network, but NFS doesn’t work the same over a long link as it does over a LAN. Management keeps yelling that we have a multigigabit link between our remote sites but what our users experience when they try to access their files over the WAN link is truly frustrating. Is this just an impossible task?
Broken Builds:
Frequent broken builds could be symptomatic of deeper problems within a development project.
Is there anything more aggravating to programmers than fellow team members checking in code that breaks a build? I find myself constantly tracking down minor mistakes in other people’s code simply because they didn’t check that their changes didn’t break the build. The worst part is when someone has broken the build and they get indignant about my pointing it out. Are there any better ways to protect against these types of problems?
Broken Hearts and Coffee Mugs:
The ordeal of security reviews
Overall, there are two broad types of security review: white box and black box. A white-box review is one in which the attackers have nearly full access to information such as code, design documents, and other information that will make it easier for them to design and carry out a successful attack. A black-box review, or test, is one in which the attackers can see the system only in the same way that a normal user or consumer would.
Bugs and Bragging Rights:
It’s not always size that matters.
Dear KV, I’ve been dealing with a large program written in Java that seems to spend most of its time asking me to restart it because it has run out of memory.
Can More Code Mean Fewer Bugs?:
The bytes you save today may bite you tomorrow
Dear One, You almost had me with your appeal to simplicity, that having a single line with system() on it reduces the potential for bugs. Almost, but not quite.
Cherry-picking and the Scientific Method:
Software is supposed be a part of computer science, and science demands proof.
So while haggling with the cherry seller, it became obvious that buying a whole flat of cherries would be a better deal than buying a single basket, even though that was all we really wanted. Not wanting to pass up a deal, however, my friend bought the entire flat and off we went, eating and talking. It took another 45 minutes to get home, and during that time we had eaten more than half the flat of cherries.
Chilling the Messenger:
Keeping ego out of software-design review
Trying to correct someone who has just done a lot of work, even if, ultimately, that work is not the right work, is a daunting task. The person in question no doubt believes that he has worked very hard to produce something of value to the rest of the team, and walking in and spitting on it, literally or metaphorically, probably crosses your "offense" line--at least I think it does. I’m a bit surprised that since this is the first sprint and there is already so much code written, shouldn’t the software have shown up after the sprints established what was needed, who the stakeholders were, etc.?
Cloud Calipers:
Naming the next generation and remembering that the cloud is just other people’s computers
For the time being, we are likely to continue to have programmers who version their functions as a result of the limitations of their languages, but let’s hope we can stop them naming their next generations after the next generation.
Code Abuse:
One programmer’s extension is another programmer’s abuse.
During some recent downtime at work, I’ve been cleaning up a set of libraries, removing dead code, updating documentation blocks, and fixing minor bugs that have been annoying but not critical. This bit of code spelunking has revealed how some of the libraries have been not only used, but also abused. The fact that everyone and their sister use the timing library for just about any event they can think of isn’t so bad, as it is a library that’s meant to call out to code periodically (although some of the events seem as if they don’t need to be events at all).
Code Hoarding:
Committing to commits, and the beauty of summarizing graphs
Dear KV, Why are so many useful features of open-source projects hidden under obscure configuration options that mean they’ll get little or no use? Is this just typically poor documentation and promotion, or is there something that makes these developers hide their code? It’s not as if the code seems broken. When I turned these features on in some recent code I came across, the system remained stable under test and in production. I feel that code should either be used or removed from the system. If the code is in a source-code repository, then it’s not really lost, but it’s also not cluttering the rest of the system.
Code Rototilling:
KV hates unnecessary work.
Dear KV, Whenever a certain programmer I work with needs to add a variable to a function and the name collides with a previously used name, he changes all of the previous instances to a new different name so that he can reuse the name himself. This causes his diffs to be far larger than they need to be and annoys the hell out of me. Whenever I challenge him on this, he says that the old usage was wrong, anyway, but I think that’s just him making an excuse.
Cold, Hard Cache:
On the implementation and maintenance of caches
Dear KV, Our latest project at work requires a large number of slightly different software stacks to deploy within our cloud infrastructure. With modern hardware, I can test this deployment on a laptop. The problem I keep running up against is that our deployment system seems to secretly cache some of my files and settings and not clear them, even when I repeatedly issue the command to do so. I’ve resorted to repeatedly using the find command so that I can blow away the offending files. What I’ve found is that the system caches data in many places so I’ve started a list.
Collecting Counters:
Gathering statistics is important, but so is making them available to others.
Over the past month I’ve been trying to figure out a problem that occurs on our systems when the network is under heavy load. After about two weeks I was able to narrow down the problem from "the network is broken" (a phrase that my coworkers use mostly to annoy me), to being something that is going wrong on the network interfaces in our systems.
Commitment Issues:
When is the right time to commit changes?
One of the other people on my project insists on checking in unrelated changes in large batches. When I say unrelated, what I mean is he will fix several unrelated bugs and then make a few minor changes to spacing and indentation across the entire source tree. He will then commit all of these changes at once, usually with a short commit message that lists only the bugs he claims to have fixed. Do you think I’m being too picky in wanting each checkin to address only one issue or problem?
Debugging Devices:
What is the proper way to debug malfunctioning hardware?
I suggest taking a very sharp knife and cutting the board traces at random until the thing either works, or smells funny! I gather you’re not asking the same question that led me to use the word changeineer in another column. I figure you have an actually malfunctioning piece of hardware and that you’ve already sent three previous versions back to the manufacturer, complete with nasty letters containing veiled references to legal action should they continue to send you broken products.
Debugging on Live Systems:
It’s more of a social than a technical problem.
I’ve been trying to debug a problem on a system at work, but the control freaks who run our production systems don’t want to give me access to the systems on which the bug always occurs. I haven’t been able to reproduce the problem in the test environment on my desktop, but every day the bug happens on several production systems.
Divided by Division:
Is there a best used-by date for software?
Do you know of any rule of thumb for how often a piece of software should need maintenance? I’m not thinking about bug fixes, since bugs are there from the moment the code is written, but about the constant refactoring that seems to go on in code. Sometimes I feel as if programmers use refactoring as a way of keeping their jobs, rather than offering any real improvement.
Don’t be Typecast as a Software Developer:
Kode Vicious’s temper obviously suffers from having to clean up after the mistakes of his peers. What would he have them learn now so that he can look forward to a graceful and mellow old age?
I would like to think that learning more will help me in my everyday job of writing glue and customization code at a systems integrator. But the obvious applicable knowledge is specific to tools and packages that may become obsolete or discontinued even within the lifetime of the project, and in some cases have already reached this destination.
Every Silver Lining Has a Cloud:
Cache is king. And if your cache is cut, you’re going to feel it.
Clearly, your management has never heard the phrase, "You get what you pay for." Or perhaps they heard it and didn’t realize it applied to them. The savings in cloud computing comes at the expense of a loss of control over your systems, which is summed up best in the popular nerd sticker that says, "The Cloud is Just Other People’s Computers." Some providers now have something called Metal-as-a-Service, which I really think ought to mean that an ’80s metal band shows up at your office, plays a gig, smashes the furniture, and urinates on the carpet, but alas, it’s just the cloud providers’ way of finally admitting that cloud computing isn’t really the right answer for all applications.
Facing the Strain:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I’ve been working on a software team that produces an end-user application on several different operating system platforms. I started out as the build engineer, setting up the build system, then the nightly test scripts, and now I work on several of the components themselves, as well as maintaining the build system. The biggest problem Ive seen in building software is the lack of API stability. It’s OK when new APIs are added--you can ignore those if you like--and when APIs are removed I know, because the build breaks. The biggest problem is when someone changes an API, as this isn’t discovered until some test script--or worse, a user--executes the code and it blows up.
File-system Litter:
Cleaning up your storage space quickly and efficiently
Dear KV, We recently ran out of storage space on a very large file server and upon closer inspection we found that it was just one employee who had used it all up. The space was taken up almost exclusively by small files that were the result of running some data-analysis scripts. These files were completely unnecessary after they had been read once. The code that generated the files had no good way of cleaning them up once they had been created; it just went on believing that storage was infinite. Now we’ve had to put quotas on our file servers and, of course, deal with weekly cries for more disk space.
Forced Exception-Handling:
You can never discount the human element in programming.
Yes, KV also reads "The Morning Paper," although he has to admit that he does not read everything that arrives in his inbox from that list. Of course, the paper you mention piqued my interest, and one of the things you don’t point out is that it’s actually a study of distributed systems failures. Now, how can we make programming harder? I know! Let’s take a problem on a single system and distribute it. Someday I would like to see a paper that tells us if problems in distributed systems increase along with the number of nodes, or the number of interconnections.
Forked Over:
Shortchanged by open source
How can one make reasonable packages based on open-source software when most open-source projects simply advise you to take the latest bits on GitHub or SourceForge? We could fork the code, as GitHub encourages us to do, and then make our own releases, but that puts the release-engineering work that we would expect from the project onto us.
GNL is Not Linux:
What’s in a Name?
What, indeed, is in a name? As you’ve already seen, this quasi-technical topic continues to cause a bit of heat in the software community, particularly in the open-source world. You can find the narrative from the GNU side by clicking on the link provided in the postscript to this article, but KV finds that narrative lacking, and so, against my better judgment about pigs and dancing, I will weigh in with a few comments.
Gardening Tips:
A good library is like a garden.
I’ve been maintaining a set of libraries for my company for the past year. The libraries are used to interface to some special hardware that we sell, and all of the code we sell to our end users runs on top of the libraries, which talk, pretty much directly, to our hardware. The one problem I keep having is that the application programmers continually reach around the library to talk directly to the hardware, and this causes bugs in our systems because the library code maintains state about the hardware. If I make the library stateless, then every library call will have to talk to the hardware, which will slow down the library and all of the code that uses it.
Get Real about Realtime:
Dear KV, I’m working on a networked system that has become very sensitive to timing issues.
I’m working on a networked system that has become very sensitive to timing issues. When the system was first developed the bandwidth requirements were well within the tolerance of off-the-shelf hardware and software, but in the past three years things have changed. The data stream has remained the same but now the system is being called on to react more quickly to events as they arrive. The system is written in C++ and runs on top of Linux. In a recent project meeting I suggested that the quickest route to decreasing latency was to move to a realtime version of Linux, since realtime operating systems are designed to provide the lowest-latency services to applications.
Gettin’ Your Head Straight:
Kode Vicious is hungry. He sustains himself on your questions from the software development trenches (and lots of beer). Without your monthly missives, KV is like a fish out of water, or a scientist without a problem to solve. So please, do you part to keep him sane (or at least free from psychotic episodes), occupied, and useful.
Dear KV, One of the biggest problems I have is memory. Not the RAM in my computer, but the wet squishy stuff in my head. It seems that no matter how many signs I put up around my cube, nor how often I turn off all the annoying instant messaging clients I need to use for work, I can’t get through more than 15 minutes of work without someone interrupting me, and then I lose my train of thought. If this happens when I’m reading e-mail, that’s not a problem, but when working on code, in particular when debugging a difficult problem in code, this makes my life very difficult.
Gettin’ Your Kode On:
Dear KV, Simple question: When is the right time to call the c_str() method on a string to get the actual pointer?
Another year is upon us and we are happy to have Kode Vicious still ranting against the ills of insecure programming, insufficient commenting, and numerous other forms of koding malpractice. Yet despite his best efforts, the bittersweet truth is that these problems are not going away anytime soon, and therefore should continue to provide ample fodder for future KV columns. Oh, to live in a world that doesn’t need KV’s advice or doctors, for that matter.
Hickory Dickory Doc:
On null encryption and automated documentation
Dear KV, While reviewing some encryption code in our product, I came across an option that allowed for null encryption. This means the encryption could be turned on, but the data would never be encrypted or decrypted. It would always be stored "in the clear." I removed the option from our latest source tree because I figured we didn’t want an unsuspecting user to turn on encryption but still have data stored in the clear. One of the other programmers on my team reviewed the potential change and blocked me from committing it, saying that the null code could be used for testing.
How to Improve Security?:
It takes more than flossing once a year.
We recently had a security compromise at work, and now the whole IT department is scrambling to improve security. One problem this whole episode has brought to light is that so much security advice is generic. It’s like being told to lock your door when you go out at night, without saying what kind of lock you ought to own or how many are enough to protect your house. I think by now most people know they need to lock their doors, so why aren’t there more specific guidelines for securing systems?
Human-KV Interaction:
We can’t guarantee you’ll agree with his advice, but it’ll probably be more effective than anything you’ve tried thus far.
Welcome to another installment of Kode Vicious, the monthly forum for Queue’s resident kode maven and occasional rabble-rouser. KV likes to hear your tales from the coding trenches, preferably those ending with a focused question about programming. KV also has a tried-and-true arsenal of techniques for dealing with interpersonal relationships so feel free to enlist his help with those characters as well.
Interviewing Techniques:
Separating the good programmers from the bad
My work group has just been given approval to hire four new programmers, and now all of us have to interview people, both on the phone and in person. I hate interviewing people. I never know what to ask. I’ve also noticed that people tend to be careless with the truth when writing their resumes. We’re considering a programming test for our next round of interviewees, because we realized that some previous candidates clearly couldn’t program their way out of a paper bag. There have to be tricks to speeding up hiring without compromising whom we hire.
IoT: The Internet of Terror:
If it seems like the sky is falling, that’s because it is.
It is true that many security-focused engineers can sound like Chicken Little, running around announcing that the sky is falling, but, unless you’ve been living under a rock, you will notice that, indeed, the sky IS falling. Not a day goes by without a significant attack against networked systems making the news, and the Internet of Terror is leading the charge in taking distributed systems down the road to hell - a road that you wish to pave with your good intentions.
KV the Konqueror:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Suppose I’m a customer of Sincere-and-Authentic’s (“Kode Vicious Battles On,” April 2005:15-17), and suppose the sysadmin at my ISP is an unscrupulous, albeit music-loving, geek. He figured out that I have an account with Sincere-and-Authentic. He put in a filter in the access router to log all packets belonging to a session between me and S&A. He would later mine the logs and retrieve the music—without paying for it.
KV the Loudmouth:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
What requirement is being satisfied by having Unclear build a P2P file-sharing system? Based upon the answer, it may be more effective, and perhaps even more secure, to use an existing open source project or purchase commercial software to address the business need.
Kabin Fever:
KV's guidelines for KFH (koding from home)
Let me invite you to my next Zoom meeting on how to host Zoom meetings! As a devotee of mobile computing and remote work from my earliest days at university, I have, over time, developed a number of useful habits for maintaining a good and productive working rhythm, and I've found that many of these apply well to those of you who are newly working from home.
Know Your Algorithms:
Stop using hardware to solve software problems.
Knowing that your CPU is in use 100 percent of the time doesn’t tell you much about the overall system other than it’s busy, but busy with what? Maybe it’s sitting in a tight loop, or some clown added a bunch of delay loops during testing that are no longer necessary. Until you profile your system, you have no idea why the CPU is busy. All systems provide some form of profiling so that you can track down where the bottlenecks are, and it’s your responsibility to apply these tools before you spend money on brand new hardware.
Kode Vicious:
The Doctor is In
KV is back on duty and ready to treat another koding illness: bad APIs. This is one of the most widespread pathologies affecting, and sometimes infecting, us all. But whether we write APIs or simply use APIs (or both), we would all do well to read on and heed the vicious one’s advice.
Kode Vicious Battles On:
Kode Vicious is at it again, dragging you out of your koding quagmires and kombating the enemies of kommon sense.
Dear KV, I’m maintaining some C code at work that is driving me right out of my mind. It seems I cannot go more than three lines in any file without coming across a chunk of code that is conditionally compiled.
Kode Vicious Bugs Out:
What do you do when tools fail?
This month Kode Vicious serves up a mixed bag, including tackling the uncertainties of heisenbugs -- a nasty type of bug that’s been known to drive coders certifiably insane. He also gives us his list of must-reads. Are any of your favorites on the list? Read on to find out!
Kode Vicious Cycles On:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Not only does California give you plenty of sun, it also apparently has employers that give you plenty of time to play around with the smaller problems that you like, in a programming language that’s irrelevant to the later implementation.
Kode Vicious Gets Dirty:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear Kode Vicious, I am a new Webmaster of a (rather new) Web site in my company’s intranet. Recently I noticed that although I have implemented some user authentication (a start *.asp page linked to an SQL server, having usernames and passwords), some of the users found out that it is also possible to enter a rather longer URL to a specific page within that Web site (instead of entering the homepage), and they go directly to that page without being authenticated (and without their login being recorded in the SQL database).
Kode Vicious Plays in Traffic:
With increasing complexity comes increasing risk.
There is no single answer to the question of how to apply software to systems that can, literally, kill us, but there are models to follow that may help ameliorate the risk. The risks involved in these systems come from three major areas: marketing, accounting, and management. It is not that it is impossible to engineer such systems safely, but the history of automated systems shows us that it is difficult to do so cheaply and quickly.
Kode Vicious Reloaded:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
The program should be a small project, but every time I start specifying the objects and methods it seems to grow to a huge size, both in the number of lines and the size of the final program.
Kode Vicious Strikes Again:
Kall us krazy, but we’re making Kode Vicious a regular.
Dear Kode Vicious, I have this problem. I can never seem to find bits of code I know I wrote. This isn’t so much work code--that’s on our source server--but you know, those bits of test code I wrote last month, I can never find them. How do you deal with this?
Kode Vicious Unleashed:
Koding konundrums driving you nuts? Ko-workers making you krazy? Not to worry, Kode Vicious has you covered.
Dear KV, My officemate writes methods that are 1,000 lines long and claims they are easier to understand than if they were broken down into a smaller set of methods. How can we convince him his code is a maintenance nightmare?
Kode Vicious Unscripted:
The problem? Computers make it too easy to copy data.
Some months, when he’s feeling ambitious, Kode Vicious reads through all of your letters carefully, agonizing for days over which to respond to. Most of the time, though, he takes a less measured approach. This usually involves printing the letters out, throwing them up in the air, and seeing which land face up, repeating the process until only two remain. And occasionally, KV dispenses with reader feedback altogether, as is the case this month.
Kode Vicious to the Rescue:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear Kode Vicious, Where I work we use a mixture of C++ code, Python, and shell scripts in our product. I always have a hard time trying to figure out when it’s appropriate to use which for a certain job. Do you code in only assembler and C, or is this a problem for you as well?
Kode Vicious vs. Mothra:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, My co-workers keep doing really bad things in the code, such as writing C++ code with macros that have gotos that jump out of them, and using assert in lower-level functions as an error-handling facility. I keep trying to get them to stop doing these things, but the standard response I get is, “Yeah, it’s not pretty, but it works.” How can I get them to start asking, “Is there a better way to do this?” They listen to my arguments but don’t seem convinced. In some cases they even insist they are following good practices.
Kode Vicious: The Return:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, Whenever my team reviews my code, they always complain that I don’t check for return values from system calls. I can see having to check a regular function call, because I don’t trust my co-workers, but system calls are written by people who know what they’re doing--and, besides, if a system call fails, there isn’t much I can do to recover. Why bother?
Koding Academies:
A low-risk path to becoming a front-end plumber
Encourage your friend to pick a course that will introduce concepts that can be used into the future, rather than just a specific set of buzzword technologies that are hot this year. Most courses are based around Python. Encourage your friend to study that as a first computer language, as the concepts learned in Python can be applied in other languages and other fields.
Latency and Livelocks:
Sometimes data just doesn’t travel as fast as it should.
Dear KV: My company has a very large database with all of our customer information. The database is replicated to several locations around the world to improve performance locally, so that when customers in Asia want to look at their data, they don’t have to wait for it to come from the United States, where my company is based...
Lazarus Code:
No one expects the Spanish Acquisition.
I’ve been asked to look into the possibility of taking a 15-year-old piece of open-source software and updating it to work on a current system used by my company. The code itself doesn’t seem to be too bad, at least no worse than the code I’m used to reading, but I suspect it might be easier to write a new version from scratch than to try to understand code that I didn’t write and which no one has actively maintained for several years.
Logging on with KV:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I’ve been stuck with writing the logging system for a new payment processing system at work. As you might imagine, this requires logging a lot of data because we have to be able to reconcile the data in our logs with our customers and other users, such as credit card companies, at the end of each billing cycle, and we have to be prepared if there is any argument over the bill itself. I’ve been given the job for two reasons: because I’m the newest person in the group and because no one thinks writing yet another logging system is very interesting.
MUST and MUST NOT:
On writing documentation
Pronouncements without background or explanatory material are useless to those who are not also deeply steeped in the art and science of computer security or security in general. It takes a particular bend of mind to think like an attacker and a defender all at once, and most people are incapable of doing this; so, if you want the people reading the document to follow your guidance, then you must take them on a journey from ignorance to knowledge.
Master of Tickets:
Valuing the quality, not the quantity, of work
Many silly metrics have been created to measure work, including the rate at which tickets are closed, the number of lines of code a programmer writes in a day, and the number of words an author can compose in an hour. All of these measures have one thing in common: They fail to take into account the quality of the output. If Alice writes 1,000 lines of impossible-to-read, buggy code in a day and Carol writes 100 lines of well-crafted, easy-to-use code in the same time, then who should be rewarded?
Merge Early, Merge Often:
Integrating changes in branched development
When doing merged development, how often should you merge? It’s obvious that if I wait too long, then I spend days in merge hell, where nothing seems to work and where I wind up using the revert command more often than commit; but the whole point of branched development is to be able to protect the main branch of development from unstable changes. Is there a happy middle ground?
Numbers Are for Computers, Strings Are for Humans:
How and where software should translate data into a human-readable form
Unless what you are processing, storing, or transmitting are, quite literally, strings that come from and are meant to be shown to humans, you should avoid processing, storing, or transmitting that data as strings. Remember, numbers are for computers, strings are for humans. Let the computer do the work of presenting your data to the humans in a form they might find palatable. That’s where those extra bytes and instructions should be spent, not doing the inverse.
Outsourcing Responsibility:
What do you do when your debugger fails you?
Dear KV, I’ve been assigned to help with a new project and have been looking over the admittedly skimpy documentation the team has placed on the internal wiki. I spent a day or so staring at what seemed to be a long list of open-source projects that they intend to integrate into the system they have been building, but I couldn’t find where their original work was described.
Painting the Bike Shed:
A sure-fire technique for ending pointless coding debates
Last week one of our newer engineers checked in a short program to help in debugging problems in the code that we’re developing. Even though this was a test program, several people read the code and then commented on the changes they wanted to see. The code didn’t have any major problems, but it seemed to generate a lot of e-mail for what was being checked in. Eventually the comments in the thread were longer than the program itself. At some point in the thread the programmer who submitted the code said, "Look, I’ve checked in the code; you can paint the bike shed any color you want now," and then refused to make any more changes to the code.
Peerless P2P:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I’ve just started on a project working with P2P software, and I have a few questions. Now, I know what you’re thinking, and no this isn’t some copyright-violating piece of kowboy kode. It’s a respectable corporate application for people to use to exchange data such as documents, presentations, and work-related information. My biggest issue with this project is security, for example, accidentally exposing our users data or leaving them open to viruses. There must be more things to worry about, but those are the top two. So, I want to ask "What would KV do?"
Phishing for Solutions:
Re: phishing, doesn’t the URL already give away enough information?
Dear KV, I noticed you covered cross-site scripting a few issues back, and I’m wondering if you have any advice on another Web problem, phishing. I work at a large financial institution and every time we roll out a new service, the security team comes down on us because either the login page looks different or they claim that it’s easy to phish information from our users using one of our forms. It’s not like we want our users to be phished, but I don’t think it’s a technical problem. Our users are just stupid and give away their information to anyone who seems willing to put up a reasonable fake of one of our pages.
Pickled Patches:
On repositories of patches and tension between security professionals and in-house developers
I recently came upon a software repository that was not a repo of code, but a repo of patches. The project seemed to build itself out of several other components and then had complicated scripts that applied the patches in a particular order. I had to look at this repo because I wanted to fix a bug in the system, but trying to figure out what the code actually looked like at any particular point in time was baffling. Are there tools that would help in working like this?
Pointless PKI:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
We’ve had problems in the past with internal compromises, and management has decided that the only way to protect the information is to encrypt it during transmission.
Poisonous Programmers:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I hope you don’t mind if I ask you about a non-work-related problem, though I guess if you do mind you just won’t answer. I work on an open source project when I have the time, and we have some annoying nontechnical problems. The problems are really people, and I think you know the ones I mean: people who constantly fight with other members of the project over what seem to be the most trivial points, or who contribute very little to the project but seem to require a huge amount of help for their particular needs. I find myself thinking it would be nice if such people just went away, but I don’t think starting a flame war on our mailing lists over these things would really help.
Popping Kernels:
Choosing between programming in the kernel or in user space
In a world in which high-performance code continues to be written in a fancy assembler, a.k.a. C, with no memory safety and plenty of other risks, the only recourse is to stick to software engineering basics. Reduce the amount of code in harm’s way, keep coupling between subsystems efficient and explicit, and work to provide better tools for the job, such as static code checkers and large suites of runtime tests.
Port Squatting:
Don’t irk your local sysadmin.
Dear KV, A few years ago you upbraided some developers for not following the correct process when requesting a reserved network port from IETF (Internet Engineering Task Force). While I get that squatting a used port is poor practice, I wonder if you, yourself, have ever tried to get IETF to allocate a port. We recently went through this with a new protocol on an open-source project, and it was a nontrivial and frustrating exercise.
Porting with Autotools:
Using tools such as Automake and Autoconf with preexisting code bases can be a major hassle.
A piece of C code I’ve been working on recently needs to be ported to another platform, and at work we’re looking at Autotools, including Automake and Autoconf, to achieve this. The problem is that every time I attempt to get the code building with these tools I feel like a rat in a maze. I can almost get things to build but not quite.
Pride and Prejudice: (The Vasa):
What can software engineers learn from shipbuilders?
I teach computer science to undergraduate students at a school in California, and one of my friends in the English department, of all places, made an interesting comment to me the other day. He wanted to know if my students had ever read Frankenstein and if I felt it would make them better engineers. I asked him why he thought I should assign this book, and he said he felt that a book could change the way in which people think about their relationship to the world, and in particular to technology. He wasn’t being condescending; he was dead serious. Given the number of Frankenstein-like projects that seem to get built with information technology, perhaps it’s not a bad idea to teach these lessons to computer science undergraduates, to give them some notion that they have a social responsibility.
Raw Networking:
Relevance and repeatability
Dear KV, The company I work for has decided to use a wireless network link to reduce latency, at least when the weather between the stations is good. It seems to me that for transmission over lossy wireless links we’ll want our own transport protocol that sits directly on top of whatever the radio provides, instead of wasting bits on IP and TCP or UDP headers, which, for a point-to-point network, aren’t really useful.
Reducing the Attack Surface:
Sometimes you can give the monkey a less dangerous club.
The best way to reduce the attack surface of a piece of software is to remove any unnecessary code. Since you now have two teams demanding that you leave in the code, it’s probably time to think about making two different versions of your binary. The application sounds like it’s an embedded system, so I’ll guess that it’s written in C and take it from there.
Removing Kode:
Dead functions and dead features
Removing dead code from systems is one of KV's favorite koding pastimes because there is nothing quite like that feeling you get when you get rid of something you know wasn't being used. Code removal is like cleaning house, only sometimes you clean house with a flame thrower, which, honestly, is very satisfying. Since you're using a version-control system (you had better be using a VCS!), it's very easy to remove code without worry. If you ever need the code you removed, you can retrieve it from the VCS at will.
Saddle Up, Aspiring Code Jockeys:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I am an IT consultant/contractor. I work mainly on networks and Microsoft operating systems. I have been doing this work for more than eight years. Unfortunately, it is starting to bore me. My question is: How would I go about getting back into programming? I say getting back into because I have some experience. In high school I took two classes of programming in Applesoft BASIC. I loved it, aced everything, and was the best programming student the teacher ever saw. This boosted my interest in computer science, which I pursued in college. In college, I took classes in C++, Java, and Web development.
Sanity vs. Invisible Markings:
Tabs vs. spaces
Invisible and near-invisible markings bring us to the human part of the problem?not that code editor authors aren’t human, but most of us will not write new editors, though all of us will use editors. As we all know, once upon a time computers had small memories and the difference between a tab, which is a single byte, and a corresponding number of spaces (8) could be a significant difference between the size of source code stored on a precious disk, and also transferred, over whatever primitive and slow bus, from storage into memory.
Scale Failure:
Using a tool for the wrong job is OK until the day when it isn’t.
Dear KV, I have been digging into a network-based logging system at work because, from time to time, the system jams up, even when there seems to be no good reason for it to do so. What I found would be funny, if only it weren’t my job to fix it: the central dispatcher for the entire logging system is a simple for loop around a pair of read and write calls; the for loop takes input from one of a set of file descriptors and sends output to one of another set of file descriptors. The system works fine so long as none of the remote readers or writers ever blocks, and normally that’s not a problem.
Sizing Your System:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I’m working on a network server that gets into the situation you called livelock in a previous response to a letter (Queue May/June 2008). Our problem is that our system has only a fixed amount of memory to receive network data, but the system is frequently overwhelmed and can’t make progress. When I ask our application engineers about how much data they expect, the only answer I get is "a lot," which isn’t much help. How can I figure out how to size our systems appropriately?
Standards Advice:
Easing the pain of implementing standards
My mother took language, both written and spoken, very seriously. The last thing I wanted to hear upon showing her an essay I was writing for school was, "Bring me the red pen." In those days I did not have a computer; all my assignments were written longhand or on a typewriter, so the red pen meant a total rewrite. She was a tough editor, but it was impossible to question the quality of her work or the passion that she brought to the writing process.
Storage Strife:
Beware keeping data in binary format
Where I work we are very serious about storing all of our data, not just our source code, in our source-code control system. When we started the company we made the decision to store as much as possible in one place. The problem is that over time we have moved from a pure programming environment to one where there are other people - the kind of people who send e-mails using Outlook and who keep their data in binary and proprietary formats.
Swamped by Automation:
Whenever someone asks you to trust them, don’t.
So your group fell for the "just install this software and things will be great" ploy. It’s an old trick that continues to snag sysadmins and others who have supporting roles around developers. Whenever someone asks you to trust them, don’t. Cynical as that might be, it’s better than being suckered.
Take a Freaking Measurement!:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Have you ever worked with someone who is a complete jerk about measuring everything?
The Chess Player who Couldn’t Pass the Salt:
AI: Soft and hard, weak and strong, narrow and general
The problem inherent in almost all nonspecialist work in AI is that humans actually don’t understand intelligence very well in the first place. Now, computer scientists often think they understand intelligence because they have so often been the "smart" kid, but that’s got very little to do with understanding what intelligence actually is. In the absence of a clear understanding of how the human brain generates and evaluates ideas, which may or may not be a good basis for the concept of intelligence, we have introduced numerous proxies for intelligence, the first of which is game-playing behavior.
The Logic of Logging:
And the illogic of PDF
I work in a pretty open environment, and by open I mean that many people have the ability to become the root user on our servers so that they can fix things as they break.
The Meaning of Maintenance:
Software maintenance is more than just bug fixes.
Isn’t software maintenance a misnomer? I’ve never heard of anyone reviewing a piece of code every year, just to make sure it was still in good shape. It seems like software maintenance is really just a cover for bug fixing. When I think of maintenance I think of taking my car in for an oil change, not fixing a piece of code. Are there any people who actually review code after it has been running in a production environment?
The Naming of Hosts is a Difficult Matter:
Also, the perils of premature rebooting
The naming of hosts is a difficult matter that ranks with coding style, editor choice, and language preference in the pantheon of things computer people fight about that don’t matter to anyone else in the whole world.
The Network Protocol Battle:
A tale of hubris and zealotry
Dear KV, I’ve been working on a personal project that involves creating a new network protocol. Out of curiosity, I tried to find out what would be involved in getting an official protocol number assigned for my project and discovered that it could take a year and could mean a lot of back and forth with the powers that be at the IETF. I knew this wouldn’t be as simple as clicking something on a Web page, but a year seems excessive, and really it’s not a major part of the work, so it seems like this would mainly be a distraction.
The Next Big Thing:
The future of functional programming and KV’s top five protocol-design tips
Dear KV, I know you did a previous article where you listed some books to read. I would also consider adding How to Design Programs, available free on the Web. This book is great for explaining the process of writing a program. It uses the Scheme language and introduces FP. I think FP could be the future of programming. John Backus of the IBM Research Laboratory suggested this in 1977. Even Microsoft has yielded to FP by introducing FP concepts in C# with LINQ.
The Non-psychopath's Guide to Managing an Open-source Project:
Respect your staff, learn from others, and know when to let go.
Transitioning from one of the technical faithful to one of the hated PHBs (pointy-haired bosses), whether in the corporate or the open-source world, is truly a difficult transition. Unless you are a type who has always been meant for the C-suite?, it's going to take a lot of work and a lot of patience, mostly with yourself, to make this transition.
The Obscene Coupling Known as Spaghetti Code:
Teach your junior programmers how to read code
Since you both are working on the same code base, you also have ample opportunity for leadership by showing this person how you code. You must do this carefully or the junior programmer will think you’re pulling rank, but, with a bit of gentle show and tell, you can get your Padawan to see what you’re driving at. This human interaction is often difficult for those of us who prefer to spend our days with seemingly logical machines.
The Observer Effect:
Finding the balance between zero and maximum
The problem is a failure to appreciate just what you are asking a system to do when polling it for information. Modern systems contain thousands of values that can be measured and recorded. Blindly retrieving whatever it is that might be exposed by the system is bad enough, but asking for it with a high-frequency poll is much worse.
The Unholy Trinity of Software Development:
Tests, documentation, and code
Questions like this bring to mind the fact that source code, documentation, and testing are the unholy trinity of software development, although many organizations like to see them as separate entities. It is interesting that while many groups pay lip service to "test-driven development," they do not include documentation in TDD.
The Virtue of Paranoia:
A koder with attitude, KV answers your questions. Miss Manners he ain’t.
Dear KV, I just joined a company that massages large amounts of data into an internal format for its own applications to work on. Although the data is backed up regularly, I have noticed that access to this data, which has accumulated to be several petabytes in size, is not particularly well secured. There is no encryption, and although the data is not easily reachable from the Internet, everyone at the company has direct access to the volumes, both physically and electronically, all the time.
The Worst Idea of All Time:
Revelations at 100!
In February 2004, with the other members of the Queue editorial board, I was at our monthly in-person dinner meeting, where we gather to come up with interesting discussion topics that will result in practitioner-oriented articles (and the best authors to write them) for publication in Queue. It was only our second year in business, and although we had published some successful and widely read articles, Queue still had no regular columnists. I was initially invited to board meetings by another editorial board member, Eric Allman, and had written a couple of articles for the publication. I was also co-authoring my first book but had never been a columnist.
This is the Foo Field:
The meaning of bits and avoiding upgrade bog downs
When will someone write documentation that tells you what the bits mean rather than what they set? I’ve been working to integrate a library into our system, and every time I try to figure out what it wants from my code, all it tells me is what a part of it is: "This is the foo field." The problem is that it doesn’t tell me what happens when I set foo. It’s as if I’m supposed to know that already.
Too Big to Fail:
Visibility leads to debuggability.
Our project has been rolling out a well-known, distributed key/value store onto our infrastructure, and we’ve been surprised - more than once - when a simple increase in the number of clients has not only slowed things, but brought them to a complete halt. This then results in rollback while several of us scour the online forums to figure out if anyone else has seen the same problem. The entire reason for using this project’s software is to increase the scale of a large system, so I have been surprised at how many times a small increase in load has led to a complete failure.
Understanding the Problem:
Is there any data showing that Java projects are any more or less successful than those using older languages?
I’ve done a one-day intro class and read a book on Java but never had to write any serious code in it. As an admin, however, I’ve been up close and personal with a number of Java server projects, which seem to share a number of problems.
Vicious XSS:
For readers who doubt the relevance of KV’s advice, witness the XSS attack that befell MySpace in October.
For readers who doubt the relevance of KV’s advice, witness the XSS attack that befell MySpace in October. This month Kode Vicious addresses just this sort of XSS attack. It’s a good thing cross-site scripting is not abbreviated CSS, as the MySpace hacker used CSS to perpetrate his XSS attack. That would have made for one confusing story, eh?
Wanton Acts of Debuggery:
Keep your debug messages clear, useful, and not annoying.
Dear KV, Why is it that people who add logging to their programs lack the creativity to differentiate their log messages? If they all say the same thing—for example, DEBUG—it’s hard to tell what is going on, or even why the previous programmer added these statements in the first place.
Watchdogs vs. Snowflakes:
Taking wild-ass guesses
That a system can randomly jam doesn’t just indicate a serious bug in the system; it is also a major source of risk. You don’t say what your distributed job-control system controls, but let’s just say I hope it’s not something with significant, real-world side effects, like a power station, jet aircraft, or financial trading system. The risk, of course, is that the system will jam, not when it’s convenient for someone to add a dummy job to clear the jam, but during some operation that could cause data loss or return incorrect results.
What Are You Trying to Pull?:
A single cache miss is more expensive than many instructions.
Saving instructions - how very 1990s of him. It’s always nice when people pay attention to details, but sometimes they simply don’t pay attention to the right ones. While KV would never encourage developers to waste instructions, given the state of modern software, it does seem like someone already has. KV would, as you did, come out on the side of legibility over the saving of a few instructions.
What is a CSO Good For?:
Security requires more than an off-the-shelf solution.
The CSO is not a security engineer, so let’s contrast the two jobs to create a picture of what we should and should not see.
Writing a Test Plan:
Establish your hypotheses, methodologies, and expected results.
If you can think of each of your tests as an experiment with a hypothesis, a test methodology, and a test result, it should all fall into place rather than falling through the cracks.