A Conversation with Leo Chang of ClickShift

Interviews

June 30, 2006
Volume 4, issue 5

Download PDF version of this article PDF

A Conversation with Leo Chang of Clickshift

To explore this month’s theme of component technologies, we brought together two engineers with lots of experience in the field to discuss some of the current trends and future direction in the world of software components. Queue Editorial board member Terry Coatta is the director of software development at GPS Industries. His expertise is in distributed component systems such as CORBA, EJB, and COM. He joins in the discussion with Leo Chang, the cofounder and CTO of ClickShift, an online campaign optimization and management company. Chang also cofounded a company called flyswat, which was purchased by NBCi, and served as the vice president of software development at Savi Technology.

TERRY COATTA I want to start off by asking how you would define component.

LEO CHANG I think of it as a piece of software that has a published interface, a published functionality, whose users are independent of the developer of the component. Usually I think of it as distributable and often sellable.

TC In a sense, then, components could go all the way back to DLLs and shared libraries.

LC When I think software component, I often think of a library that’s distributable, that’s published, that’s versionable. I think of Web services today as sort of a software component. I think it fits very well with that definition.

TC Let’s go back in time to DLLs or shared libraries that you can give to people, and then move on to COM and CORBA, and now on to Web services. Do you see any trends in this progression over time?

LC I think you see the same goals—and the same problems—repeated over and over again. It would be nice to write something once and have everybody reuse it, or to write something that’s maybe not sellable to an end user but could clearly be used by a bunch of developers. It would be nice if you could distribute that and make it useful and/or make money from everyone distributing it.

You need a way to publish it and advertise it and standardize all those things. Those goals are pretty common across technologies. The challenges are also the same. You have versioning challenges, and you have standards that aren’t always standard, which impairs the adoption of these technologies.

TC You mention standards, and it seems to me most of them have failed to catch on. Why do you think that is?

LC I feel there’s sort of a cultural thing—the not-invented-here attitude that some organizations have—that prevents some of these standards from catching on.

The obvious draw for using a third-party component is this: Why reinvent the wheel? But from my experience, there is a problem with using a third-party anything, whether you call it a component, a library, or whatever. One problem lies in trying to debug your own stuff, but you’re not sure if it’s your stuff or if it’s that third-party weird bug that’s unfixable.

The other thing that impairs adoption is the inherent conflict between designing a standard that is totally flexible and can be all things to all people, and making it simple enough so that people will want to use it. Maybe associated with that is the existence of more than one standard. You’re seeing that a little now with Web services, too.

TC So how do you deal with debugging problems when you’re dealing with third-party components that are basically just black boxes?

LC It’s fundamentally unsolvable, but that doesn’t mean it’s not worth doing. Just like when you buy any product—a car, for example—you buy it on reputation and because other people are using it and finding that it’s reliable. Company A makes good components, and does a really good job of testing them and is responsive to support calls, so that’s why you buy from Company A. Company B has a terrible reputation for making really buggy components that are overpriced, so you wouldn’t buy components from Company B. It doesn’t mean that the whole realm becomes useless because of that. Those are real-world problems that are solved by the market, in a capitalistic way.

TC The open source community is distributing components not as binary entities but as the source code itself. Do you think of open source as an alternative way of approaching components where one isn’t trying to distribute binary functionality but instead is willing to distribute it in a way that’s more amenable to debugging and fixing?

LC I would choose an open source component because of the way it has been developed. If the open source community developed it, then it’s probably very robust. If there is a problem, it will get debugged by many of the talented 400 people who are working on it—or 400,000 people. If worst comes to worst, I can debug it myself by going through the source code, but that doesn’t give me a whole lot of comfort.

TC It also provides another kind of visibility for figuring out whether the performance problem is in your code or in their code. It provides a little bit more visibility into the kinds of interactions that are happening in the system. You can see when you’re calling into the components and maybe when it’s calling back into the system.

LC That’s true. But if I’m using MySQL and something is wrong, I don’t think, “Oh my god, let’s go through the source code and figure out what’s going on.” It’s a big complicated program and I don’t want to have to do that.

If the component is smaller, then I guess that does become an option. You can look through the source code, or at least let me find a forum where someone else has the same problem and has pointed out that in the 4,000 source code files, it’s these two that matter and here is the problem.

TC One of the things I find interesting about the difference between open source components and the traditional kind of component framework is that, of course, there isn’t that notion of a well-defined standard in the open source world. There isn’t one way of specifying interfaces and one way of instantiating components. Open source is much more flexible and fluid than that.

Having standards is nice in that they force everything down the same path, so you need to learn only one way of doing things. But, as you say, it can sometimes be complex. What do you think about the value of the rigidity of the standards in the interface specifications that are typically associated with components?

LC It can be very frustrating, because you’re trying to do something that seems really quite simple, and you wonder why it is turning out to be so difficult.

TC Do you think there is an issue of scale here? As projects become larger and larger, I find that having those kinds of rigid frameworks helps serve as an organizing principle. You start to appreciate the rigidness, even though it may introduce some additional complexity. Do you have a sense of that as well?

LC As a project gets bigger and bigger, you appreciate that there is a standard way of doing things. If you’re buying components from a hundred different companies, it would certainly be nice if there were one published way to use them.

From an open source standpoint, you may have 4,000 developers each trying to contribute code. Of those, you have a group of as few as one or maybe 10 people who are the leaders and really organizing the project in their own way. In that sense, they’re publishing their way of doing it through their community. It isn’t 4,000 people randomly coding. It is 10 people.

But for our projects, we have several places where we need an outside component. We may have something that’s speaking COM to component A and we may have something that’s speaking SOAP to Component B.

I don’t mind learning a new standard as long as it’s simple and does what it’s supposed to do. Deciding instead that I’m going to speak only SOAP or only COM hasn’t worked well for me.

TC Let’s talk about Web services-based components. Do you realistically foresee a universe where I can pull together a bunch of different services that are on the Web into a larger-scale application?

LC I believe that can happen. I guess it was XML over HTTP that just a few years back was going to change the world, and Web services was going to hook up everything. That was over-hyped.

But years later I think it’s still headed in that direction. There are going to be more and more situations where people will provide services and probably charge for them. Users of these services can combine them in all sorts of different ways to provide functionality to their end users.

TC Can you describe what kinds of Web services you’re integrating and what issues you have encountered in doing that?

LC We’re an online optimization company. You can imagine that Google and Yahoo and many others have Web services that let our system talk to their systems. We have lots of those. That’s very important to what we’re building.

TC So if you are integrating a service from Google and from another service provider, what types of issues come up? Do you run into any interesting development issues?

LC Yes we do. You have company A that’s providing a Web service via SOAP, and company B that’s using its dialect of REST, and company C with its own flavor of XML over HTTP, sort of an XML-RPC thing. It would be nice if they were all speaking the exact same dialect. But as I said before, I have pretty much given up on the success of the standard idea or standard language that everybody is going to speak. I would just like for each one to be relatively simple to use.

For the most part, that’s happened. They are simple enough so we can build our own clients to talk to each one. Basically, everybody has a slightly different dialect. At least, for the most part, it’s all XML over HTTP or HTTPS.

TC It sounds as if you’re operating from a premise that the W3C standards not going to succeed, or at least are not currently in use. How do you see standards eventually playing out?

LC I don’t know if I want to make predictions about whether they’re going to succeed or not. But, essentially, if I have a service and it’s useful—even if it’s not using the standard or if it’s close, or using a dialect of a standard—people are going to use it anyway. In some situations, there may be no other way to do it. If we want it and it’s useful for us, we’ll build the client side anyway. So, sure, it causes some friction, but it’s not a blocking obstacle.

TC There’s this notion that there’s a correct time to standardize. Standardizing too early tamps down on innovation and standardizing too late is almost impossible because the world has become too diverse. Where do you think we are along that spectrum? I sometimes have the feeling that the W3C folks are a little bit ahead of the curve in terms of how much they’re standardizing.

LC The standardization process usually takes quite a long time and I feel that we’re in the middle of that process right now.

I don’t follow the W3C that much because it’s not that relevant. I need to build what I need to build right now. That means I need to use services that were built years ago or even recently that aren’t following the standards.

That doesn’t necessarily mean that they’re coming too late because these things need to be shaken out, and maybe we haven’t even gotten there yet. And it doesn’t mean that it’s too early because I don’t know how much more innovation we need on the different ways to speak XML over HTTP.

TC Given that you’ve got some experience pulling together multiple services from multiple providers, do you have any advice for people who might be thinking about building applications this way?

LC To reiterate, don’t expect to pick one standard and use only that. There’s not a big market for these things in which functionality A is provided by 10 different companies, five of them using SOAP and five of them using REST or whatever. The actual situation is that we need a particular service and there is only one source to get it from. That source speaks its own dialect of XML over HTTP, so that’s what we’ve got to use. My advice would be that you can’t decide just to use SOAP.

It obviously depends on the requirements of your project, but you have to be flexible on what kind of technology you’re going to use to speak with each of the parties that you need to speak to. You must also address the system design issues, whether it be transactional processing or error handling. They usually get a little bit nastier when you have to deal with all these services.

Under that same heading is performance. You’re going to have latency, you’re going to have multithreading issues, and all the fundamental problems and benefits that go with those things.

TC Tell us how you might architect your own application for dealing with a number of different service providers. Would you try to internally map everything to a single model, converting things into the lingua franca inside your own application?

LC We have internal software interfaces that abstract away the particulars of the service that we’re talking to. But these abstractions aren’t perfect and we have to handle the quirks that you can’t hide at higher layers, anyway. As an organizing principle, however, we do have all these services that we categorize in the same bucket and try to treat as much as possible in the same way, except when we can’t.

TC Your initial way of describing a component was almost as a stand-alone entity. But a lot of the working components over the past decade actually live in the context of a relatively complex framework. EJB (Enterprise JavaBeans) and ActiveX controls fall into this category, where they’re not just redistributable units of functionality but part of this ecosystem.

LC I can understand why they were designed that way. It’s a lot easier to do. If you say there’s a container or there’s an application server that these things have to run in, then it becomes much easier to implement. But I have not had good experiences with that.

That’s why I feel the Web services components have a better chance of surviving, because the users don’t have to decide that they’re going to go with the J2EE (Java 2 Platform, Enterprise Edition) app server, or whatever big sort of framework they need to be in.

Users of Web services in most cases need a pretty simple client to be built to talk to the Web service. You don’t have to make this massive technology choice if you want to use company A’s Web service. Some manufacturers will provide a component that can live in many different frameworks. But with Web services, that’s kind of the point right off. You just need to do XML over HTTP.

TC Would life have been easier for you if there were some common frameworks in place, even in what I think of as the transport levels, where you’ve got your REST and your XML-RPC and your SOAP, or one layer up in terms of services or common interfaces?

LC A standard way of doing all these things would have made life easier. But I am naturally skeptical about standardizing these higher-level things, maybe because I haven’t seen something like that really catch on. For example, you see the success of various ways of authenticating people or securing channels, say through SSL. But it’s harder for me to think of lots of examples of a standard way of getting a bank account balance or of paying someone money. For now, if you want to take a credit card transaction, sure, there are a few ways to do it, but it isn’t because the W3C said this is how you do it.

TC What differences do you see between SOAP and REST? Do you have an affinity for one over the other?

LC As you can probably tell, I’m not very religious about these things. But from our experience, SOAP is often much more complicated to do something very simple. It’s pretty heavy.

There is a whole philosophy in the subculture around REST. It seems much simpler, much lighter weight. But I’ve noticed that when people are implementing it, they don’t really stick to that philosophy completely.

TC Do you think that’s because it’s too hard to actually build something real by following that “religion”?

LC There may be something to that. But I also think the people working on it may not understand the religion. Even if a couple of people in the organization do and that’s why they chose it, not necessarily all the engineers will. Pretty soon they’ve added a new function that works, but in some subtle or sometimes not-so-subtle way, it violates the REST philosophy.

We’re using one service that is completely nonstandard—a proprietary way of using HTTP. I would prefer that they use SOAP because we could then use our SOAP library to talk to it. But if they had used SOAP or something very standard, the same engineers would have designed it and it would have been just as quirky. The quirks are at a higher level and they’re more fundamental than the transport protocol.

TC Suppose you were going to provide a service. Would you choose one of these ways of doing it or would you just go with an amalgam of stuff?

LC I would probably choose SOAP because I feel there are more client libraries out there to use it. I might want to choose REST, however, because I like a lot of things about it: It’s much simpler, it doesn’t have a lot of overhead, it’s much lighter weight, and it’s pretty easy to debug. My choice would probably depend specifically on the service.

TC Do you have a sense of where service providers are going at the moment?

LC It seems that there’s momentum behind SOAP. There are lots of SOAP libraries in every language and all kinds of platforms. And everybody seems to know what SOAP is. Even people who don’t like SOAP complain about it and then use it—sort of like Windows, I guess.

TC Do you use Web services as a way of communicating between components within your application?

LC We have some services designed that aren’t distributed. We haven’t even chosen the dialect. We chose some simple lightweight thing that was totally proprietary to us, and we were very, very happy with it.

At this point my bias might be to use something REST-like internally. But if I had to expose something externally to customers, I might choose SOAP, because there are lots of client libraries for it.

TC One of the things that has made component systems come into existence is this desire for people to be able to package up functionality and sell it. How is that progressing in the Web services world? Are people starting to sell services? What are the models for selling services?

LC We’re definitely paying for some of the services that we’re using. It’s essentially call- or operation- or quota-based. For one penny you get this. We’re willing to pay, and the services keep track of billing with different levels of sophistication. But people are doing it. Absolutely.

Originally published in Queue vol. 4, no. 5—
Comment on this article in the ACM Digital Library