Discipline and Focus

Interviews

July 14, 2008
Volume 4, issue 6

Discipline and Focus - Transcript

Transcript of interview with Werner Vogels, CTO of Amazon.

MICHAEL VIZARD: Hello, and welcome to ACM Queuecasts. I'm your host, Mike Vizard, and joining me today to discuss large scale computing initiatives is Werner Vogels, CTO of Amazon. Welcome, Werner. How are you doing?

WERNER VOGELS: Hi, Mike. Thanks for having me.

MV: Great. Given the asynchronous nature of billions of transactions floating around amazon.com, what are some of the bigger technical challenges you face in making sure that all the relevant applications are always up to date with each other?

WV: I think first of all, it's important to realize that a large-scale system such as Amazon is actually built out of many different applications, and each of those applications has different requirements with respect to availability and consistency. Of course we would always like to achieve 100 percent availability, but that's not often possible. You have to then make tradeoffs in terms of consistency to actually achieve that. So whether you're taking an order, whether you are updating customer records, whether you are checking the availability of a particular item, whether you're creating a shipment order for a particular product, all of those things require a different approach to availability and consistency. One way that we've made this much easier for ourselves here at Amazon is that internally we are a services-oriented architecture, and I don't mean that in terms of the buzzword of the last five years. We've been this long before that word became public. What I mean by that is that within Amazon, all small pieces of business functionality are run as a separate service. For example, this availability of a particular item is a service that is a piece of software running somewhere that encapsulates data, that manages that data. And when, for example, we hit the gateway page of Amazon, it will go out to somewhere between 100 and 150 different services to construct this page for you. That means that we've localized the notion of availability and consistency towards each service because each service is responsible for its data encapsulation. And then, depending on what kind of service it is and what kind of consistency guarantees they require, you use different technologies. But it must be absolutely clear I think to everyone that we're not using two phase commit to update distributed resources all over the world.

MV: When you talk about service-oriented architecture, a lot of people today will equate that with, say, XML web services, but it sounds to me like you're actually talking about it more as the traditional model, where it's about the componentization of the applications and whether the interface is XML or whether it's Java or whatever it might be doesn't really matter.

WV: Indeed, it's really the latter pattern. And you should really see this as a particular pattern, where indeed we do data encapsulation and that the data is only accessible through a hardened interface towards the service. Customers of that service have a contract with the service that guarantees to provide this service to them through this interface, and in that way we can do perfect isolation of functionality, as well as data storage. It does mean that service owners, the ones that are responsible for the service, have to think about how to manage their data internally, but we never have anywhere direct database access, for example. All access to data is through the business logic that the service is. And I don't know, maybe on the wire that sometimes is XML, but that's just irrelevant for this case.

MV: How do you manage those types of servers and that level of infrastructure? There's been a lot of talk in the industry about the evolution of autonomic computing and I guess the next generation of lights-out management. How do you think about provisioning those servers and managing them and keeping them up without having to hire an army of IT guys.

WV: Yeah. So, I think there are two answers to your question there. The first part addresses autonomic computing, which is I think just like Web services or service-oriented architecture, one of those buzzwords, you know? Autonomic computing means many different things to many people. At Amazon, we really don't use the word "autonomic computing," but everything is indeed centered around these services. Now, you have to realize... I have to back up a bit and explain how these services work at Amazon. A service is not just a software component in that sense, a distributed software component. It is also the model we use for software development in terms of team structure. So how the process goes is that you have a particular business problem or a technology problem that you want to solve. You build a small team. You give them this problem. And at Amazon, they're allowed to solve that problem in any way they see fit, as long as it is through this hardened API. They can pick the tools they want. They can do any design methodology they want as long as they deliver the actual functionality that they've been tasked with. Now, with this flexibility comes also the responsibility of actually operating the service. The same group of people that actually builds the service is also responsible for operating it. So there is no wall over which things are being thrown at Amazon, to which the operators who are then responsible for handling the service, but they are the same developers, the same group of people that has built a service are also responsible for operating it. That allows us to have a sort of a direct feedback loop with customers inside there. Developers are actually motivated to improve their service because they are completely end-to-end responsible for it. And in general, this is a model that works very well for us because it allows us to innovate very quickly. We use this, not only encapsulating then at the software level, but also at the team level. Now, how you manage hundreds of thousands of these services is that we like to believe that it's a bit like... I think Pat Halens(sp?) uses the metaphor of using a town as an example. So in a big town, you have zoning requirements. You have some general laws about the roads and things like that, but the way that you build your house and the way that you operate your house is all up to you. So this is a bit the way that Amazon functions, also. We have some requirements: that services has to be monitorable, that they have to be tractable in all sorts of different ways. But in essence, operation is all up to the service owners themselves. This allows for a large-let's say controlled chaos-which actually works very well because everybody's responsible for their own services.

MV: So in that model, an application developer can't just build something and then dump it and run, say that it's the fault of the hardware or the network that it's not running correctly.

WV: Indeed, because you are responsible for the functionality of what your service delivers, whether it was database failure, whether it was hardware failure, whether it was your software that failed, whether it was human error, it doesn't matter. It is all about the service that you deliver to the rest of the enterprise, whether it is to other services that can be aggregated into something else, or whether it is towards the rendering for the user, it's all about the service that you deliver. And you guarantee a certain up-time, and you just have to meet those, whether it's hardware or not.

MV: Whether we talk about JAVA virtual machines or things like VMware, it feels like we're constantly trying to move the extraction layer of computing higher up the stack. How far do you think we'll go on that path, and will that change the way we think about systems and network management as we go forward because everything becomes an abstraction of something else?

WV: That's a more philosophical question, of course. I don't really know where we're going. But I can make a general observation, I think. What we're seeing here at Amazon is that maybe in the past we viewed only hardware or only the network as being unreliable.

MV: Yeah.

WV: I think slowly what we're seeing is that that's not the only level of unreliability, for example, that you have to deal with. There's the operating system, which is not necessarily as reliable as we want. Then there is the middle-ware, whether we fill it ourselves or whether you buy this from other people, that it's not as reliable or doesn't give you really the tools to do the availability versus consistency tradeoffs that you want. So if you really want to build a large, reliable distributed system, you have to involve the application into that. And so there are many things which you cannot just solve in a standard middle-ware layer and hope that everything on top of that runs reliably. To really find solutions for something of the scale of Amazon, you have to do a lot of custom development that targets the specific reliability and scalability requirements that we have. Do virtual machines play a role in that? I think if I look at the current excitement surrounding virtual machine monitors there are a number of areas where they can be applicable. I think one of them is for resource sharing. Then there is the isolation for security purposes, or for just system management. I think at Amazon, we're not as much concerned about the first part, which is the resource sharing. Given our services model, we already have a good isolation level there, and we would really like to have not too many shared resources between different services. So sharing a CPU or sharing a set of CPUs between services is not a good plan. You really won't have that isolation level down there. So it leaves us with isolation for security and isolation for easy deployment, and I think the latter two make virtual machines interesting for us, especially I think the latter one, where we really enjoy using virtual machines for actually doing fast deployment of new service instances, in cases where our services can be self-scalable and can be dialed up and dialed down, depending on the load on the site.

MV: So it sounds like you guys have thought a lot about the modular architecture you needed to build around a large-scale system. Rather than just letting it kind of sprawl out there over a period of years, everything is kind of tightly organized around people and machines, so if there's one failure, it's not catastrophic.

WV: Absolutely. First thing, I think there's a whole list of good practices that we have in terms of design, in terms of architecture, in terms of building. And one of those points -- one of the bullets on that list is that you have to design for failure, meaning that failure of components, whether they're hardware, software, humans, is a fact of life, and you have to architect as if they are continuously happening to you. And if you do that and you happen to hit a good streak, then you're fine. But failure in any large-scale system is the normal case, not the exception. So build, for example, for fast recovery. That's an essential part. You know, stuff fails, comes back up, and you have to make sure that it can be inserted back into the functioning set as soon as possible.

MV: If you only have 10 problems a day, you're having a good day, then. How do you manage storage in that environment, because as we all know as we go forward, storage is becoming more complex, certainly more important, and you've got this whole diverse set of storage subsystems, and you're trying to manage things across the life cycle of the data. How do you deal with that across different vendors?

WV: Let me step back a bit. We have to kind of think about what storage means. Are we really talking about block level storage, using storage area networks or ANSCSI(?) and things like that? One important thing to realize there is that with our ownership model, we really don't like to push off the responsibility towards a vendor for our reliability. So, no matter whether you're using a database that is more highly available than others appear to be, whether you're using a storage area network-in the end, it's the application's responsibility to be always up and running. So we never defer responsibility to the storage subsystem, for example. And we're at Amazon, you know, using standard databases, in-memory databases, fast indices, and we've also built a number of storage subsystems ourselves that allow us, for example, to make the tradeoff between consistency and availability. We have some highly scalable available systems, but that now and then they give you no for an answer because you want to have them highly available. We have a number of other approaches in which we're capable of relaxing the consistency requirements or the consistency semantics of the system such that we can always make progress under all circumstances.

MV: It sounds like you just don't accept what the vendors hand you for the gear. You do a lot of work to customize it and extend it where necessary to really make it work for your purposes.

WV: You absolutely have to. I think if you talk to anybody in industry that is responsible for running a very large-scale, geographically distributed, distributed system, such as Amazon is, relying on third parties, on vendors, to actually deliver this availability for you is very dangerous. We've seen that there are a number of vendors out there that are exceptional in providing highly available systems in very contained environments. There aren't that many systems out there of the scale of Amazon. The problems that we have-I won't say the problems-the challenges that we have in delivering this very highly available system...there are not that many others that have these kinds of challenges. And so third party software is clearly not geared to meeting our challenges there. And this is one of the real excitement areas at Amazon: building things that are so far out that nobody else in the world is doing this.

MV: Given the size and scope of Amazon, there's a lot of talk in the industry about good computing. Most of the talk is around scientific applications. Do you see good computing playing a role at Amazon in the future?

WV: Well, again there, you know, grid is such a buzzword. What is grid? Is grid this MPI-based, federated, parallel computing engine, such as the Globus guys did, or earlier the Combo guys? I don't think that really matches Amazon's structure, the way that we do things. We don't have that massive parallel computing going on. However, I think it's not that difficult, for example, to look at the collection of servers that take user requests-you know, takes a user request, goes out to all the services, collects the information, renders it, brings it back to the server-as being a sort of grid. But in the end, whether you call it grid or not, there's hardly any grid computing technology that we use for this particular case. So Amazon has, you know, if you look at all these different user sets, there's a lot of diversity. So we have our buying customers, the customers that visit our retail site, especially now during the Christmas holidays. But Amazon is also a platform so it's a platform on which other people are selling things. There are small sellers, so if you go to the website, if you look for a book, you often have this one little line that says, you know, "12 items also new and used sold by other vendors." There are larger merchants that are integrated with amazon.com, for example, Nordstrom or JOI Electronics. Then there are the larger merchants that actually we operate websites for, such as starter.com. And then we have the whole category of developers and associates. They're all different customer sets, and they all have different requirements. And for all of them we have different collections of servers that actually process their requests and then interact with the platform.

MV: So are you going to wind up managing more as you go forward, you know, not only the Amazon systems, but the systems of your partners and suppliers in some way, so that you now take large-scale networks to a completely different level, where you're creating a system that encompasses not just Amazon, but just about everybody that does business with Amazon?

WV: Well, actually that's not just in the future: It's already happening now. And I know this is not something that is commonly known out there, but Amazon is really a technology platform on which amazon.com and amazon.co.uk and all the international sites are just one customer of this platform. And so there are many others out there that use a few of the services that Amazon delivers integrated with their site. And there's many big sites that just run Amazon's technology. Do we manage them for them? Yes, absolutely, that's the business we are in. And for others, they're just integrated with Amazon. That means that there is an interaction where basically we send them feeds of transactions, and they send us feeds of items and offerings and of the availability back, and that's kind of the interaction. And for others, we also do fulfillment. For example, we process the shipments and send them out to customers.

MV: So you're really an information technology services company in many ways. And given as our last question here, what do you think is your best advice for people who are trying to build that same kind of infrastructure and who are charged with running that? What did you learn over the years that you wish you knew when you first got started?

WV: Well, first of all, I've not been here that terribly long, so I have trouble answering that question. There's a philosophy we use here at Amazon I think that is really important for us, and that's that it is really the customer that matters. Often you get really hung up on technology, and remember, all of this talk about grids, autonomic computing, storage techniques, all of this is all because we're actually wanting to serve our customers better. And if we're not doing that, then, you know, all of this talk about availability versus consistency actually doesn't mean anything. So we really try here to look at the customer and then walk backwards from the customer requirements until we have technology that exactly matches what we're trying to do for the customer. We try not to get hung up too much about, you know, how beautiful technology is and how excited as engineers we get into building them or running them because in the end, it means nothing if you are not making your customers' lives better. And I think that's, for me at least as a technologist, a very important thing, you know? You do this because in the end you want to try to achieve something for people. And that drives a lot of the technology thinking here at Amazon.

MV: Great. Werner, our thanks for being on the show today.

WV: Mike, it was my pleasure. Thank you.

MV: And this is Mike Vizard signing off for ACM Queuecasts. Thank you and have a good morning, evening, afternoon, whereever you are. Bye. (End Vogels Queuecast)

Originally published in Queue vol. 4, no. 6—
Comment on this article in the ACM Digital Library