Source blog: All Things Distributed
District heating: Using data centers to heat communities
An inside look at the Tallaght District Heating Scheme, where Heat Works is using recycled heat from an AWS data center to warm a community in Dublin, Ireland.
What I've been reading since re:Invent
After a busy conference season, I've taken some time to catch up on reading and make a dent in the pile of books on my nightstand. Here's what I've started, finished, and picked up since re:Invent.
Tech predictions for 2024 and beyond
The coming years will be filled with innovation in areas designed to democratize access to technology and help us keep up with the increasing pace of every-day life -- and it starts with Generative AI.
Standing on the shoulders of giants: Colm on constant work
The Builders' Library gathers the expertise of Amazon's most experience builders in one place. One article that really got me thinking was Colm MacCárthaigh's "Reliability, constant work, and a good cup of coffee" which is about an anti-fragility pattern that he developed for building simple, more robust, and cost-effective systems.
Farewell EC2-Classic, it?s been swell
It is time to celebrate EC2-Classic, which defined early cloud computing, and which is finally retiring.
Building and operating a pretty big storage system called S3
Three distinct perspectives on scale that come along with building and operating a storage system the size of S3.
A few words on taking notes
As we are about to start the planning meetings for 2024 at AWS, I?ve been thinking a lot about how I take notes.
Monoliths are not dinosaurs
Building evolvable software systems is a strategy, not a religion. And revisiting your architectures with an open mind is a must.
Can autonomous trucks transform the global supply chain?
For the season finale of Now Go Build, I traveled to Arizona, to see first-hand how autonomous trucking has the potential to revolutionize the global supply chain.
How AI coding companions will change the way developers work
Developer tools are one area where generative AI is already having a tangible impact on productivity and speed, and it's the reason I'm excited about Amazon CodeWhisperer.
AI doesn't plant trees
I went on a journey with the Vietnam AWS team to plant a 10000 trees
Demystifying LLMs with Amazon distinguished scientists
To learn more about large language models (LLMs), foundation models, and other advances in ML, I sat with two of Amazon?s distinguished scientists, Sudipta Sengupta and Dan Roth.
An introduction to generative AI with Swami Sivasubramanian
The VP of database, analytics and machine learning services at AWS, Swami Sivasubramanian, walks me through the broad landscape of generative AI, what we?re doing at Amazon to make large language and foundation models more accessible, and how custom silicon can help to bring down costs, speed up training, and increase energy efficiency for our customers.
Is Australia the new epicenter for healthtech startups?
Australia is home to a flourishing startup scene. It's flush with companies focused on improving healthcare outcomes, not only for people in Australia, but everywhere in the world. Cloud technologies are helping to accelerate both research and innovation in this space, and it all starts with the brain.
Bringing Digital People to life with autonomous animation
As we find ourselves spending more time talking with and through machines than we do face-to-face, the popularity of conversational AI and large language models show that there is an appetite for interfaces that feel more like conversations with people. However, what's still missing is the connection that's established with visual and non-verbal cues. The folks at Soul Machines believe that autonomously animated Digital People are the answer.
Melbourne gets a Region, a big trip, and a brain mapping startup
Innovation is borderless. Great ideas (and of course, great engineering) aren?t bound by geography or the lines on a map. Today, we launch the new AWS Asia Pacific (Melbourne) Region, expanding our footprint in Australia, and bringing AWS cloud services closer to companies like Omniscient Neurotechnology, a brain mapping startup that has the potential to revolutionise neurosurgery and treatments for mental health conditions.
An album for each year - 2022 version
A list of my favorite albums from 1958 through 2022. One per year; no repeats.
An album for each year - 2022 version
A list of my favorite albums from 1958 through 2022. One per year; no repeats.
An album for each year - 2022 version
A list of my favorite albums from 1958 through 2022. One per year; no repeats.
Tech predictions for 2023 and beyond
As access to advanced technology becomes even more ubiquitous?as every facet of life becomes data that we can analyze?we will start to see a torrent of innovation, and this will proliferate in 2023.
Tech predictions for 2023 and beyond
As access to advanced technology becomes even more ubiquitous?as every facet of life becomes data that we can analyze?we will start to see a torrent of innovation, and this will proliferate in 2023.
Tech predictions for 2023 and beyond
As access to advanced technology becomes even more ubiquitous?as every facet of life becomes data that we can analyze?we will start to see a torrent of innovation, and this will proliferate in 2023.
On site in Denmark: Behind the scenes with Veo
In the fourth episode of season 3 of Now Go Build, I visit with Veo in Denmark.
On site in Denmark: Behind the scenes with Veo
In the fourth episode of season 3 of Now Go Build, I visit with Veo in Denmark.
On site in Denmark: Behind the scenes with Veo
In the fourth episode of season 3 of Now Go Build, I visit with Veo in Denmark.
¡Bienvenida España! Introducing the new AWS Europe (Spain) Region
The new AWS Europe (Spain) Region is ready for use. Now, go build!
¡Bienvenida España! Introducing the new AWS Europe (Spain) Region
The new AWS Europe (Spain) Region is ready for use. Now, go build!
¡Bienvenida España! Introducing the new AWS Europe (Spain) Region
The new AWS Europe (Spain) Region is ready for use. Now, go build!
The Distributed Computing Manifesto
Today, I am publishing the Distributed Computing Manifesto, a canonical document from the early days of Amazon that transformed the architecture of Amazon's ecommerce platform. It highlights the challenges we were facing at the end of the 20th century, and hints at where we were headed.
The Distributed Computing Manifesto
Today, I am publishing the Distributed Computing Manifesto, a canonical document from the early days of Amazon that transformed the architecture of Amazon's ecommerce platform. It highlights the challenges we were facing at the end of the 20th century, and hints at where we were headed.
The Distributed Computing Manifesto
Today, I am publishing the Distributed Computing Manifesto, a canonical document from the early days of Amazon that transformed the architecture of Amazon's ecommerce platform. It highlights the challenges we were facing at the end of the 20th century, and hints at where we were headed.
The new AWS Europe (Zurich) Region and 16 years of Swiss innovation
Starting today the new AWS Europe (Zurich) Region is now open for use!
The new AWS Europe (Zurich) Region and 16 years of Swiss innovation
Starting today the new AWS Europe (Zurich) Region is now open for use!
The new AWS Europe (Zurich) Region and 16 years of Swiss innovation
Starting today the new AWS Europe (Zurich) Region is now open for use!
Accelerating innovation in Thailand with the AWS Asia Pacific (Bangkok) Region
AWS announced a new region in Asia in Thailand
Accelerating innovation in Thailand with the AWS Asia Pacific (Bangkok) Region
AWS announced a new region in Asia in Thailand
Accelerating innovation in Thailand with the AWS Asia Pacific (Bangkok) Region
AWS announced a new region in Asia in Thailand
On site in Vermont: BETA's aerial maneuvers in the cloud
In the third episode of Season 3 of Now Go Build I visit BETA in Vermont
On site in Vermont: BETA's aerial maneuvers in the cloud
In the third episode of Season 3 of Now Go Build I visit BETA in Vermont
On site in Vermont: BETA's aerial maneuvers in the cloud
In the third episode of Season 3 of Now Go Build I visit BETA in Vermont
On site in Hawai'i: How Terraformation is using AWS technology to reforest the world
In the second episode of Season 3 of Now Go Build I visit Terraformation in Hawaii.
On site in Hawai'i: How Terraformation is using AWS technology to reforest the world
In the second episode of Season 3 of Now Go Build I visit Terraformation in Hawaii.
On site in Hawai'i: How Terraformation is using AWS technology to reforest the world
In the second episode of Season 3 of Now Go Build I visit Terraformation in Hawaii.
Now Go Build -- Season 3
Today the first episode of season3 of Now Go Build is available
Now Go Build - Season 3
Today the first episode of Season 3 of Now Go Build is available.
Now Go Build - Season 3
Today the first episode of Season 3 of Now Go Build is available.
Scaling globally with the new AWS Middle East (UAE) Region
`Starting today the new AWS Middle East (UAE) Region is now open for use!
Curious about automated reasoning
Diving into Automated Reasoning and Formal Verification with Dr. Byron Cook in this new video series for those Curious about new technologies
Curious about quantum computing
Diving into Quantum computing with Prof. John Preskill in this new video series for those Curious about new technologies
Expanding the cloud: Introducing the AWS Asia Pacific (Jakarta) Region
AWS ada untuk Indonesia - Starting today the new AWS Asia Pacific (Jakarta) Region is now open!.
Tech Predictions for 2022 and Beyond
2022 will be an exciting year for technology, with it pushing all of us, and our planet, forward in the process.
Announcing the AWS Robotics Startup Accelerator: It?s still Day One for robotics
The AWS Robotics Startup Accelerator aims to help robotics startups adopt and use AWS to boost their robotics development.
Amazon MemoryDB for Redis ? Where speed meets consistency
We dive into the design of Amazon MemoryDB for Redis, a strongly-consistent, Redis-compatible, in-memory database service for ultra-fast performance.
Our investment in Canada ? a new Region coming soon!
AWS plans to open a new infrastructure region in Canada - AWS Canada West (Calgary).
Kia ora, Auckland! ? An AWS Region is coming to New Zealand.
AWS plans to open a new infrastructure region in Asia Pacific in Auckland, New Zealand.
Using the cloud to solve the world's hardest problems.
In the age of cloud, problem-solving has evolved, especially when it comes to the world's hardest problems.
Now Go Build - Season 2
Season 2 of Now Go Build has been released on Youtube and Prime Video
Continuous Configuration at the Speed of Sound.
With truly dynamic configuration the source of application truth lives in an independent configuration management system, and is polled by the consuming application(s).
Amazon on AWS: Seamlessly integrating physical and emerging digital technologies.
We go behind the scenes to look at how Amazon Robotics uses AWS IoT to manage hundred of thousands of robots.
Introducing AWS Bugbust
AWS BugBust is the first global bug-busting challenge for developers to fix bugs in a fun and competitive environment. Using modern tools like CodeGuru, we?re able to transform the way we go about bug bashes, improving developer productivity with ML. Our challenge has the ambitious goal of fixing one million bugs and helping AWS customers save $100 million in technical debt.
It?s Officially Startup Season in Space
We launched our AWS Space Accelerator to catalyze space innovation and support some of the most innovative startups that are helping shape the future of aerospace. Meet the 10 startups to take part in the first cohort of the AWS Space Accelerator.
Supercharging Application Delivery
The goal of AWS Proton is simple: customers should be able to adopt, customize and evolve best practices and technologies for delivering their modern applications to the cloud, and not worry about how they roll this out ? potentially to thousands of developers ? across their organization.
Continuing our investment in the Middle East: A New AWS Region coming to UAE in first half 2022
I am excited to share that AWS plans to open a new infrastructure region in the Middle East in the United Arab Emirates (UAE). Launching in the first half of 2022, the new AWS Middle East (UAE) Region will consist of three Availability Zones and become AWS?s second region in the Middle East with the existing AWS Region in Bahrain.
Amazon Timestream - Time series is the new black
We dive into the world of time series data with a look at the design of Amazon Timestream.
Diving Deep on S3 Consistency
I recently posted about Amazon S3 and how it’s evolved over the last 15 years since we launched the service in 2006 as “storage for the internet.” We built S3 because we knew customers wanted to store backups, videos, and images for applications like e-commerce web sites. Our top design priorities at the time were security, elasticity, reliability, durability, performance and cost because that’s what customers told us was most important to them for these types of applications.
A new era of DevOps, powered by machine learning
AWS is on a journey to revolutionize DevOps using the latest technologies. We are starting to treat DevOps, and the toolchains around it, as a data science problem ? And when we think of it this way, code, logs, and application metrics are all data that we can optimize with machine learning (ML).
Happy 15th Birthday Amazon S3 -- the service that started it all
15 years is a long time in the world of technology. Back when S3 launched on March 14, 2006 (also known as ?Pi Day? to some), iPhones didn?t exist, neither did tweets or likes, Facebook was still only used by a few colleges and universities, and you couldn?t hail a ride or order lunch with an app. Back then, Amazon was ~2% of its size today, and was growing faster than traditional IT systems could support. We had to rethink everything previously known about building scalable systems. Storage was one of our biggest pain points, and the traditional systems we used just weren?t fitting the needs of the Amazon.com retail business.
Happy 15th Birthday Amazon S3 -- the service that started it all
15 years is a long time in the world of technology. Back when S3 launched on March 14, 2006 (also known as “Pi Day” to some), iPhones didn’t exist, neither did tweets or likes, Facebook was still only used by a few colleges and universities, and you couldn’t hail a ride or order lunch with an app. Back then, Amazon was ~2% of its size today, and was growing faster than traditional IT systems could support. We had to rethink everything previously known about building scalable systems. Storage was one of our biggest pain points, and the traditional systems we used just weren’t fitting the needs of the Amazon.com retail business.
Celebrating 10 years in Japan with the new AWS Asia Pacific (Osaka) Region
I am pleased to share that the AWS Asia Pacific (Osaka) Region has transitioned from a Local Region into a standard region with three Availability Zones (AZs), a broader (and growing) service portfolio, free tier option, and new pricing options. The region joins the existing 25 AZs in eight AWS Regions across Asia Pacific in Beijing, Hong Kong, Mumbai, Ningxia, Seoul, Singapore, Sydney, and Tokyo. Globally, AWS has 80 Availability Zones across 25 geographic regions, with plans to launch 15 more Availability Zones and five more AWS Regions in Australia, India, Indonesia, Spain and Switzerland.
Celebrating 10 years in Japan with the new AWS Asia Pacific (Osaka) Region
I am pleased to share that the AWS Asia Pacific (Osaka) Region has transitioned from a Local Region into a standard region with three Availability Zones (AZs), a broader (and growing) service portfolio, free tier option, and new pricing options. The region joins the existing 25 AZs in eight AWS Regions across Asia Pacific in Beijing, Hong Kong, Mumbai, Ningxia, Seoul, Singapore, Sydney, and Tokyo. Globally, AWS has 80 Availability Zones across 25 geographic regions, with plans to launch 15 more Availability Zones and five more AWS Regions in Australia, India, Indonesia, Spain and Switzerland.
From Schooling to Space: Eight Predictions on How Technology Will Continue to Change Our Lives in the Coming Year
2021 is going to be a launchpad for change, and here?s what?s coming 2020 was a year unlike any other. Businesses large and small, governments new and old all had to completely change what they do and how they operate. Helping us to manage this dramatic change was technology. Whether it was Blackboard continuing our children?s education, Zoom becoming our business boardroom (and our pub), or Netflix being our night out at the movies, we relied on technology to help feed our families, teach our children, collaborate with co-workers, even entertain ourselves after yet another day in the house.
From Schooling to Space: Eight Predictions on How Technology Will Continue to Change Our Lives in the Coming Year
2020 was a year unlike any other. Businesses large and small, governments new and old all had to completely change what they do and how they operate. Helping us to manage this dramatic change was technology. Whether it was Blackboard continuing our children’s education, Zoom becoming our business boardroom (and our pub), or Netflix being our night out at the movies, we relied on technology to help feed our families, teach our children, collaborate with co-workers, even entertain ourselves after yet another day in the house. Rather than slow us down, 2020 accelerated our shift to a digital world and I anticipate we won’t go back any time soon.
Strengthening our commitment to Australia
Our customers never stop building. And because they never stop, their need for security, availability, performance, scalability, and flexibility over how they choose to run their workloads never stop. At AWS, we are customer obsessed, so we never stop building either. That is why we have created more than 175 services, which we offer from 24 AWS Regions around the world to give our customers the best possible cloud experience, no matter where they are. We are excited to be announcing plans to launch a second infrastructure region in Australia in the second half of 2022. The new AWS Asia Pacific Region will be located in the Australian city of Melbourne and will deliver even lower latency to customers and their end users in Asia Pacific.
Strengthening our commitment to Australia
Our customers never stop building. And because they never stop, their need for security, availability, performance, scalability, and flexibility over how they choose to run their workloads never stop. At AWS, we are customer obsessed, so we never stop building either. That is why we have created more than 175 services, which we offer from 24 AWS Regions around the world to give our customers the best possible cloud experience, no matter where they are. We are excited to be announcing plans to launch a second infrastructure region in Australia in the second half of 2022.
How the Seahawks are using a data lake to improve their game
Growing up in the Netherlands, American football was largely a foreign concept to me. My version of football was The Beautiful Game, or as most Americans know it, soccer. Football, futbol, soccer, or whatever else you call it, will always be something I?m deeply passionate about, especially my hometown team, Ajax. When I joined Amazon and Seattle became my new home, I began to see how my colleagues shared this same level of passion for American football ? and particularly the fervent fans known as ?The 12s? of the local team, the Seattle Seahawks. As I started to better understand this version of football, it was easy for me to get excited about the game as well as what was happening behind the scenes.
How the Seahawks are using a data lake to improve their game
Growing up in the Netherlands, American football was largely a foreign concept to me. My version of football was The Beautiful Game, or as most Americans know it, soccer. Football, futbol, soccer, or whatever else you call it, will always be something I’m deeply passionate about, especially my hometown team, Ajax. When I joined Amazon and Seattle became my new home, I began to see how my colleagues shared this same level of passion for American football – and particularly the fervent fans known as “The 12s” of the local team, the Seattle Seahawks. As I started to better understand this version of football, it was easy for me to get excited about the game as well as what was happening behind the scenes.
Understanding Climate Change Using High Performance Computing and Machine Learning
D Watson-Parris and NASA Worldview As the COVID pandemic continues to sequester many of us to our homes, our everyday behaviors have come mostly to a collective halt. The immediate effects are obvious, as cities, roads, and public spaces have emptied. Reports of nature intermingling with spaces once claimed by humans have amazed audiences worldwide. Coyotes casually strolling by the Golden Gate Bridge and through the streets of San Francisco, the canals of Venice running clear and teeming with fish, and the [Himalayas visible from India(https://www.insider.com/himalayas-seen-from-india-pollution-drop-coronavirus-lockdown-2020-4) for the first time in three decades are just a few of the examples made famous by popular culture.
Understanding Climate Change Using High Performance Computing and Machine Learning
As the COVID pandemic continues to sequester many of us to our homes, our everyday behaviors have come mostly to a collective halt. The immediate effects are obvious, as cities, roads, and public spaces have emptied. Reports of nature intermingling with spaces once claimed by humans have amazed audiences worldwide. Coyotes casually strolling by the Golden Gate Bridge and through the streets of San Francisco, the canals of Venice running clear and teeming with fish, and the [Himalayas visible from India(https://www.insider.com/himalayas-seen-from-india-pollution-drop-coronavirus-lockdown-2020-4) for the first time in three decades are just a few of the examples made famous by popular culture. At the same time, with tragic wildfires ravaging the Pacific Coast and an already record-setting 2020 Atlantic hurricane season underway, many are feeling a weighty pull towards action for the environment.
Grüezi Schwiiz! Bonjour la Suisse! Buongiorno Svizzera! An AWS Region comes to Switzerland
Today I?m happy to announce our plans to open a new AWS Region in Switzerland in the second half of 2022. When the AWS Europe (Zurich) Region is launched, developers, start-ups, and enterprises, as well as government, education, and non-profit organizations will be able to run their applications and serve end users across the region from data centers located in Switzerland.
Grüezi Schwiiz! Bonjour la Suisse! Buongiorno Svizzera! An AWS Region comes to Switzerland
Today I’m happy to announce our plans to open a new AWS Region in Switzerland in the second half of 2022. When the AWS Europe (Zurich) Region is launched, developers, start-ups, and enterprises, as well as government, education, and non-profit organizations will be able to run their applications and serve end users across the region from data centers located in Switzerland.
Reinventing virtualization with the AWS Nitro System
Running a business at the scale of Amazon, we often have to solve problems that no other company has faced before. The disadvantage of this is that there is no ?how to? guide for us?a lot is unknown. However, the advantage is that when we solve a new problem, it?s an opportunity to reinvent our services and create new benefits for our customers. Indeed, we have created some of our most innovative and successful ideas when we have entered unchartered territory. When you?re a customer-centric company, you often find yourself in the great unknown because customers will always want more and better.
Reinventing virtualization with the AWS Nitro System
Running a business at the scale of Amazon, we often have to solve problems that no other company has faced before. The disadvantage of this is that there is no “how to” guide for us—a lot is unknown. However, the advantage is that when we solve a new problem, it’s an opportunity to reinvent our services and create new benefits for our customers. Indeed, we have created some of our most innovative and successful ideas when we have entered unchartered territory. When you’re a customer-centric company, you often find yourself in the great unknown because customers will always want more and better.
Now Go Build - Season 2
I have always been very fortunate to meet our AWS customers where they have most impact, at their customers. Many of these AWS customers are solving really hard human problems, in ways that is extremely inspiring for any builder, like me. That became the inspiration for the Now Go Build series that chronicles my conversations with these innovators and their customers. In the first season, we had a wonderful diverse group of young businesses showing their impact on the world. From providing identity to smallholder farms in Indonesia to healthcare in Brazil and conservation in South Africa. The second season that we are launching today will again have four wonderful stories.
Now Go Build - Season 2
I have always been very fortunate to meet our AWS customers where they have most impact, at their customers. Many of these AWS customers are solving really hard human problems, in ways that is extremely inspiring for any builder, like me. That became the inspiration for the Now Go Build series that chronicles my conversations with these innovators and their customers. In the first season, we had a wonderful diverse group of young businesses showing their impact on the world. From providing identity to smallholder farms in Indonesia to healthcare in Brazil and conservation in South Africa. The second season that we are launching today will again have four wonderful stories.
The OS Classics
A few days ago I was fortunate to pick up a copy of a book that had a major impact on my early career as kernel engineer; The Design and Implementation of the 4.3 BSD UNIX Operating System by Samuel J. Leffler, Marshall Kirk McKusick, Michael J. Karels and John S. Quarterman. It was the first authoritative description of Berkeley UNIX, its design and implementation. The book covers the internal structure of the 4.3 BSD systems and the concepts, data structures and algorithms used in implementing the system facilities. But most importantly it was written by practitioners and builders and as such gave insights that academic text book would never give you.
The OS Classics
A few days ago I was fortunate to pick up a copy of a book that had a major impact on my early career as kernel engineer; The Design and Implementation of the 4.3 BSD UNIX Operating System by Samuel J. Leffler, Marshall Kirk McKusick, Michael J. Karels and John S. Quarterman. It was the first authoritative description of Berkeley UNIX, its design and implementation. The book covers the internal structure of the 4.3 BSD systems and the concepts, data structures and algorithms used in implementing the system facilities. But most importantly it was written by practitioners and builders and as such gave insights that academic text book would never give you.
The journey to modern manufacturing with AWS
One of the most rewarding parts of my job is getting to watch different industries implement new technologies that improve and transform business operations. Manufacturing, in particular, has always captivated my attention in this respect. When I think about how Amazon?s globally connected distribution network has changed in the last decade alone, it?s incredible. From the Internet of Things (IoT) to Artificial Intelligence (AI) and task automation to predictive maintenance technology, the advancements in this space are creating a world of new opportunity. But this is complicated by that fact that many manufacturers have been around for decades or longer.
The journey to modern manufacturing with AWS
One of the most rewarding parts of my job is getting to watch different industries implement new technologies that improve and transform business operations. Manufacturing, in particular, has always captivated my attention in this respect. When I think about how Amazon’s globally connected distribution network has changed in the last decade alone, it’s incredible. From the Internet of Things (IoT) to Artificial Intelligence (AI) and task automation to predictive maintenance technology, the advancements in this space are creating a world of new opportunity. But this is complicated by that fact that many manufacturers have been around for decades or longer. Some of their equipment was designed before the internet even existed.
Reaffirming our commitment to Italy: Introducing the AWS Europe (Milan) Region
The global healthcare pandemic has been like nothing many of us in Europe have ever known. During this time, many organizations have been contemplating their role in the COVID-19 crisis, and how they can best serve their communities. I can tell you it has been no different for us at Amazon Web Services (AWS). We are focused on where we can make the biggest difference, to help the global communities in which we all live and work. This is why today we are announcing that the AWS Europe (Milan) Region is now open. The opening of the AWS (Milan) Region demonstrates our ongoing commitment to the people of Italy and the long-term potential we believe there is in the country.
Confermando il nostro impegno in Italia: Apertura della Regione AWS Europe (Milano)
La maggior parte di noi, in Europa, non aveva mai conosciuto prima una pandemia globale come quella in corso. Durante questo periodo, molte organizzazioni stanno riflettendo sul proprio ruolo nella crisi COVID-19 e su quale può essere il modo migliore per supportare la propria comunità. Posso dirvi che per noi di Amazon Web Services (AWS) non è stato diverso. Ci siamo concentrati su come e dove avremmo potuto fare la differenza più grande aiutando le comunità globali in cui viviamo e lavoriamo. Con questo obiettivo in mente, oggi annunciamo l'apertura della Regione AWS Europe (Milano). Il lancio della Regione AWS in Italia conferma il nostro costante impegno per gli italiani e rafforza ulteriormente il nostro sostegno al grande potenziale del paese.
Confermando il nostro impegno in Italia: Apertura della Regione AWS Europe (Milano)
La maggior parte di noi, in Europa, non aveva mai conosciuto prima una pandemia globale come quella in corso. Durante questo periodo, molte organizzazioni stanno riflettendo sul proprio ruolo nella crisi COVID-19 e su quale può essere il modo migliore per supportare la propria comunità. Posso dirvi che per noi di Amazon Web Services (AWS) non è stato diverso. Ci siamo concentrati su come e dove avremmo potuto fare la differenza più grande aiutando le comunità globali in cui viviamo e lavoriamo. Con questo obiettivo in mente, oggi annunciamo l'apertura della Regione AWS Europe (Milano).
Reaffirming our commitment to Italy: Introducing the AWS Europe (Milan) Region
The global healthcare pandemic has been like nothing many of us in Europe have ever known. During this time, many organizations have been contemplating their role in the COVID-19 crisis, and how they can best serve their communities. I can tell you it has been no different for us at Amazon Web Services (AWS). We are focused on where we can make the biggest difference, to help the global communities in which we all live and work. This is why today we are announcing that the AWS Europe (Milan) Region is now open.
Continuing our investment in Africa: Introducing the AWS Africa (Cape Town) Region
As COVID-19 has disrupted life as we know it, I have been inspired by the stories of organizations around the world using AWS in very important ways to help combat the virus and its impact. Whether it is supporting the medical relief effort, advancing scientific research, spinning up remote learning programs, or standing-up remote working platforms, we have seen how providing access to scalable, dependable, and highly secure computing power is vital to keep organizations moving forward. This is why, today, we are announcing the AWS Africa (Cape Town) Region is now open.
Continuing our investment in Africa: Introducing the AWS Africa (Cape Town) Region
As COVID-19 has disrupted life as we know it, I have been inspired by the stories of organizations around the world using AWS in very important ways to help combat the virus and its impact. Whether it is supporting the medical relief effort, advancing scientific research, spinning up remote learning programs, or standing-up remote working platforms, we have seen how providing access to scalable, dependable, and highly secure computing power is vital to keep organizations moving forward. This is why, today, we are announcing the AWS Africa (Cape Town) Region is now open.
When scaling your workload is a matter of saving lives
On March 16, 2020, at 9:26 PM, I received an urgent email from my friend DJ Patil, former White House Chief Data Scientist, Head of Technology for Devoted Health, a Senior Fellow at the Belfer Center at the Harvard Kennedy School, and Advisor to Venrock Partners. You don?t get that many titles after your name unless you?re pretty good at something. For DJ, that ?something? is math and computer science. DJ was writing to me from the California crisis command center. He explained that he was working with governors from across the country to model the potential impact of COVID-19 for scenario planning.
When scaling your workload is a matter of saving lives
On March 16, 2020, at 9:26 PM, I received an urgent email from my friend DJ Patil, former White House Chief Data Scientist, Head of Technology for Devoted Health, a Senior Fellow at the Belfer Center at the Harvard Kennedy School, and Advisor to Venrock Partners. You don’t get that many titles after your name unless you’re pretty good at something. For DJ, that “something” is math and computer science. DJ was writing to me from the California crisis command center. He explained that he was working with governors from across the country to model the potential impact of COVID-19 for scenario planning.
How Amazon is solving big-data challenges with data lakes
Back when Jeff Bezos filled orders in his garage and drove packages to the post office himself, crunching the numbers on costs, tracking inventory, and forecasting future demand was relatively simple. Fast-forward 25 years, Amazon's retail business has more than 175 fulfillment centers (FC) worldwide with over 250,000 full-time associates shipping millions of items per day. Amazon's worldwide financial operations team has the incredible task of tracking all of that data (think petabytes). At Amazon's scale, a miscalculated metric, like cost per unit, or delayed data can have a huge impact (think millions of dollars). The team is constantly looking for ways to get more accurate data, faster.
How Amazon is solving big-data challenges with data lakes
Back when Jeff Bezos filled orders in his garage and drove packages to the post office himself, crunching the numbers on costs, tracking inventory, and forecasting future demand was relatively simple. Fast-forward 25 years, Amazon's retail business has more than 175 fulfillment centers (FC) worldwide with over 250,000 full-time associates shipping millions of items per day. Amazon's worldwide financial operations team has the incredible task of tracking all of that data (think petabytes). At Amazon's scale, a miscalculated metric, like cost per unit, or delayed data can have a huge impact (think millions of dollars). The team is constantly looking for ways to get more accurate data, faster.
The power of relationships in data
Have you ever received a call from your bank because they suspected fraudulent activity? Most banks can automatically identify when spending patterns or locations have deviated from the norm and then act immediately. Many times, this happens before victims even noticed that something was off. As a result, the impact of identity theft on a person's bank account and life can be managed before it's even an issue. Having a deep understanding of the relationships in your data is powerful like that. Consider the relationships between diseases and gene interactions. By understanding these connections, you can search for patterns within protein pathways to find other genes that may be associated with a disease.
The power of relationships in data
Have you ever received a call from your bank because they suspected fraudulent activity? Most banks can automatically identify when spending patterns or locations have deviated from the norm and then act immediately. Many times, this happens before victims even noticed that something was off. As a result, the impact of identity theft on a person's bank account and life can be managed before it's even an issue. Having a deep understanding of the relationships in your data is powerful like that. Consider the relationships between diseases and gene interactions. By understanding these connections, you can search for patterns within protein pathways to find other genes that may be associated with a disease.
AWS's HPC leadership recognized by industry experts with HPCwire awards
During AWS re:Invent 2019, we announced a number of High Performance Computing (HPC) innovations including the Amazon EC2 M6g, C6g, and R6g instances powered by next-generation Arm-based AWS Graviton2 Processors. We also recently announced that new AMD-powered, compute-optimized EC2 instances are in the works. Today, I'm happy to share some exciting news about our HPC solutions. On November 18, AWS won six HPCwire Readers' and Editors' Choice Awards at SC19, the International Conference for High Performance Computing, Networking, Storage, and Analysis.
AWS's HPC leadership recognized by industry experts with HPCwire awards
During AWS re:Invent 2019, we announced a number of High Performance Computing (HPC) innovations including the Amazon EC2 M6g, C6g, and R6g instances powered by next-generation Arm-based AWS Graviton2 Processors. We also recently announced that new AMD-powered, compute-optimized EC2 instances are in the works. Today, I'm happy to share some exciting news about our HPC solutions. On November 18, AWS won six HPCwire Readers' and Editors' Choice Awards at SC19, the International Conference for High Performance Computing, Networking, Storage, and Analysis.
¡Hola España! An AWS Region is coming to Spain!
Today, I am happy to announce our plans to open a new AWS Region in Spain in late 2022 or early 2023! I'm excited by the opportunities the availability of hyper scale infrastructure will bring to Spanish organizations of all sizes. When the AWS Europe (Spain) Region is launched, developers, startups, and enterprises, as well as government, education, and non-profit organizations will be able to run their applications and serve end users across the region from data centers located in Spain. Currently, AWS provides 69 Availability Zones across 22 infrastructure regions worldwide, with announced plans for thirteen more Availability Zones and four more Regions in Indonesia, Italy, South Africa, and Spain in the next few years.
¡Hola España! An AWS Region is coming to Spain!
Today, I am happy to announce our plans to open a new AWS Region in Spain in late 2022 or early 2023! I'm excited by the opportunities the availability of hyper scale infrastructure will bring to Spanish organizations of all sizes. When the AWS Europe (Spain) Region is launched, developers, startups, and enterprises, as well as government, education, and non-profit organizations will be able to run their applications and serve end users across the region from data centers located in Spain. Currently, AWS provides 69 Availability Zones across 22 infrastructure regions worldwide, with announced plans for thirteen more Availability Zones and four more Regions in Indonesia, Italy, South Africa, and Spain in the next few years.
Act locally, connect globally with IoT and edge computing
There are places so remote, so harsh that humans can't safely explore them (for example, hundreds of miles below the earth, areas that experience extreme temperatures, or on other planets). These places might have important data that could help us better understand earth and its history, as well as life on other planets. But they usually have little to no internet connection, making the challenge of exploring environments inhospitable for humans seem even more impossible. How do we push the boundaries of what's possible? The answer to this question is actually on your phone, your smartwatch, and billions of other places on earth?it's the Internet of Things (IoT).
Act locally, connect globally with IoT and edge computing
There are places so remote, so harsh that humans can't safely explore them (for example, hundreds of miles below the earth, areas that experience extreme temperatures, or on other planets). These places might have important data that could help us better understand earth and its history, as well as life on other planets. But they usually have little to no internet connection, making the challenge of exploring environments inhospitable for humans seem even more impossible. How do we push the boundaries of what's possible? The answer to this question is actually on your phone, your smartwatch, and billions of other places on earth?it's the Internet of Things (IoT).
Modern applications at AWS
Innovation has always been part of the Amazon DNA, but about 20 years ago, we went through a radical transformation with the goal of making our iterative process?"invent, launch, reinvent, relaunch, start over, rinse, repeat, again and again"?even faster. The changes we made affected both how we built applications and how we organized our company. Back then, we had only a small fraction of the number of customers that Amazon serves today. Still, we knew that if we wanted to expand the products and services we offered, we had to change the way we approached application architecture. The giant, monolithic "bookstore" application and giant database that we used to power Amazon.com limited our speed and agility.
Modern applications at AWS
Innovation has always been part of the Amazon DNA, but about 20 years ago, we went through a radical transformation with the goal of making our iterative process?"invent, launch, reinvent, relaunch, start over, rinse, repeat, again and again"?even faster. The changes we made affected both how we built applications and how we organized our company. Back then, we had only a small fraction of the number of customers that Amazon serves today. Still, we knew that if we wanted to expand the products and services we offered, we had to change the way we approached application architecture. The giant, monolithic "bookstore" application and giant database that we used to power Amazon.com limited our speed and agility.
Expanding the cloud to the Middle East: Introducing the AWS Middle East (Bahrain) Region
I'm happy to announce today that the new AWS Middle East (Bahrain) Region is now open! This is our first AWS Region in the Middle East and I'm excited by the opportunities the availability of hyper scale infrastructure will bring to organizations of all sizes. Starting today, developers, startups, and enterprises, as well as government, education, and non-profit organizations can run their applications and serve end users across the region from data centers located in the Middle East. With this launch, our infrastructure now spans 69 Availability Zones across 22 geographic regions around the world. We have also announced plans for nine more Availability Zones in three more AWS Regions in Indonesia, Italy, and South Africa coming online in the next few years.
Expanding the cloud to the Middle East: Introducing the AWS Middle East (Bahrain) Region
I'm happy to announce today that the new AWS Middle East (Bahrain) Region is now open! This is our first AWS Region in the Middle East and I'm excited by the opportunities the availability of hyper scale infrastructure will bring to organizations of all sizes. Starting today, developers, startups, and enterprises, as well as government, education, and non-profit organizations can run their applications and serve end users across the region from data centers located in the Middle East. With this launch, our infrastructure now spans 69 Availability Zones across 22 geographic regions around the world. We have also announced plans for nine more Availability Zones in three more AWS Regions in Indonesia, Italy, and South Africa coming online in the next few years.
Amazon Aurora development team wins the 2019 ACM SIGMOD Systems Award
A few months ago, I wrote the post "Amazon Aurora ascendant: How we designed acloud-native relational database," and now I'm excited to share some news about the people behind the service. This week, the developers of Amazon Aurora have won the 2019 Association for Computing Machinery's (ACM) Special Interest Group on Management of Data (SIGMOD) Systems Award. The award recognizes "an individual or set of individuals for the development of a software or hardware system whose technical contributions have had significant impact on the theory or practice of large-scale data management systems."
Amazon Aurora development team wins the 2019 ACM SIGMOD Systems Award
A few months ago, I wrote the post "Amazon Aurora ascendant: How we designed acloud-native relational database," and now I'm excited to share some news about the people behind the service. This week, the developers of Amazon Aurora have won the 2019 Association for Computing Machinery's (ACM) Special Interest Group on Management of Data (SIGMOD) Systems Award. The award recognizes "an individual or set of individuals for the development of a software or hardware system whose technical contributions have had significant impact on the theory or practice of large-scale data management systems."
Proving security at scale with automated reasoning
Customers often ask me how AWS maintains security at scale as we continue to grow so rapidly. They want to make sure that their data is secure in the AWS Cloud, and they want to understand how to better secure themselves as they grow.
Proving security at scale with automated reasoning
Customers often ask me how AWS maintains security at scale as we continue to grow so rapidly. They want to make sure that their data is secure in the AWS Cloud, and they want to understand how to better secure themselves as they grow.
Proving security at scale with automated reasoning
Customers often ask me how AWS maintains security at scale as we continue to grow so rapidly. They want to make sure that their data is secure in the AWS Cloud, and they want to understand how to better secure themselves as they grow.
Increasing access to blockchain and ledger databases
Last year, I spent some time in Jakarta visiting HARA, an AWS customer. They've created a way to connect small farms in developing nations to banks and distributers of goods, like seeds, fertilizer, and tools. Traditionally, rural farms have been ignored by the financial world, because they don't normally have the information required to open an account or apply for credit. With HARA, this hard-to-obtain data on small farms is collected and authenticated, giving these farmers access to resources they've never had before. A major component to the system that HARA created is blockchain. This is a technology used to build applications where multiple parties can interact through a peer-to-peer-network and record immutable transactions with no central trusted authority.
Increasing access to blockchain and ledger databases
Last year, I spent some time in Jakarta visiting HARA, an AWS customer. They've created a way to connect small farms in developing nations to banks and distributers of goods, like seeds, fertilizer, and tools. Traditionally, rural farms have been ignored by the financial world, because they don't normally have the information required to open an account or apply for credit. With HARA, this hard-to-obtain data on small farms is collected and authenticated, giving these farmers access to resources they've never had before. A major component to the system that HARA created is blockchain. This is a technology used to build applications where multiple parties can interact through a peer-to-peer-network and record immutable transactions with no central trusted authority.
Expanding the AWS Cloud ? Introducing the AWS Asia Pacific (Hong Kong) Region
Today, I am happy to introduce the new AWS Asia Pacific (Hong Kong) Region. AWS customers can now use this Region to serve their end users in Hong Kong SAR at a lower latency, and to comply with any data locality requirements. The AWS Asia Pacific (Hong Kong) Region is the eighth active AWS Region in Asia Pacific and mainland China along with Beijing, Mumbai, Ningxia, Seoul, Singapore, Sydney, and Tokyo. With this launch, AWS now spans 64 Availability Zones within 21 geographic regions around the world, and has announced plans for 12 more Availability Zones and four more AWS Regions in Bahrain, Cape Town, Jakarta, and Milan.
Expanding the AWS Cloud ? Introducing the AWS Asia Pacific (Hong Kong) Region
Today, I am happy to introduce the new AWS Asia Pacific (Hong Kong) Region. AWS customers can now use this Region to serve their end users in Hong Kong SAR at a lower latency, and to comply with any data locality requirements. The AWS Asia Pacific (Hong Kong) Region is the eighth active AWS Region in Asia Pacific and mainland China along with Beijing, Mumbai, Ningxia, Seoul, Singapore, Sydney, and Tokyo.
Halo Jakarta! An AWS Region is coming to Indonesia!
Today, I am excited to announce our plans to open a new AWS Region in Indonesia! The new AWS Asia Pacific (Jakarta) Region will be composed of three Availability Zones, and will give AWS customers and partners the ability to run workloads and store data in Indonesia. The AWS Asia Pacific (Jakarta) Region will be our ninth Region in Asia Pacific. It joins existing Regions in Beijing, Mumbai, Ningxia, Seoul, Singapore, Sydney, and Tokyo, as well as an upcoming Region in Hong Kong SAR. AWS customers are already using 61 Availability Zones across 20 infrastructure Regions worldwide. Today's announcement brings the total number of global Regions (operational and announced) up to 25.
Halo Jakarta! An AWS Region is coming to Indonesia!
Today, I am excited to announce our plans to open a new AWS Region in Indonesia! The new AWS Asia Pacific (Jakarta) Region will be composed of three Availability Zones, and will give AWS customers and partners the ability to run workloads and store data in Indonesia. The AWS Asia Pacific (Jakarta) Region will be our ninth Region in Asia Pacific. It joins existing Regions in Beijing, Mumbai, Ningxia, Seoul, Singapore, Sydney, and Tokyo, as well as an upcoming Region in Hong Kong SAR. AWS customers are already using 61 Availability Zones across 20 infrastructure Regions worldwide.
Redefining application communications with AWS App Mesh
At re:Invent 2018, AWS announced the AWS App Mesh public preview, a service mesh that allows you to easily monitor and control communications across applications. Today, I'm happy to announce that App Mesh is generally available for use by customers. New architectural patterns Many customers are modernizing their existing applications to become more agile and innovate faster. Architectural patterns like microservices enable teams to independently test services and continuously deliver changes to applications. This approach optimizes team productivity by allowing development teams to experiment and iterate faster. It also allows teams to rapidly scale how they build and run their applications.
Redefining application communications with AWS App Mesh
At re:Invent 2018, AWS announced the AWS App Mesh public preview, a service mesh that allows you to easily monitor and control communications across applications. Today, I'm happy to announce that App Mesh is generally available for use by customers. New architectural patterns Many customers are modernizing their existing applications to become more agile and innovate faster. Architectural patterns like microservices enable teams to independently test services and continuously deliver changes to applications. This approach optimizes team productivity by allowing development teams to experiment and iterate faster. It also allows teams to rapidly scale how they build and run their applications.
Amazon Aurora ascendant: How we designed a cloud-native relational database
Relational databases have been around for a long time. The relational model of data was pioneered back in the 1970s by E.F. Codd. The core technologies underpinning the major relational database management systems of today were developed in the 1980?1990s. Relational database fundamentals, including data relationships, ACID (Atomicity, Consistency, Isolation, Durability) transactions, and the SQL query language, have stood the test of time. Those fundamentals helped make relational databases immensely popular with users everywhere. They remain a cornerstone of IT infrastructure in many companies. This is not to say that a system administrator necessarily enjoys dealing with relational databases.
Amazon Aurora ascendant: How we designed a cloud-native relational database
Relational databases have been around for a long time. The relational model of data was pioneered back in the 1970s by E.F. Codd. The core technologies underpinning the major relational database management systems of today were developed in the 1980?1990s. Relational database fundamentals, including data relationships, ACID (Atomicity, Consistency, Isolation, Durability) transactions, and the SQL query language, have stood the test of time. Those fundamentals helped make relational databases immensely popular with users everywhere. They remain a cornerstone of IT infrastructure in many companies. This is not to say that a system administrator necessarily enjoys dealing with relational databases. For decades, managing a relational database has been a high-skill, labor-intensive task.
Expanding the AWS Cloud ? Introducing the AWS Europe (Stockholm) Region
In April 2017, Amazon Web Services announced that it would launch a new AWS infrastructure region Region in Sweden. Today, I'm happy to announce that the AWS Europe (Stockholm) Region, our 20th Region globally, is now generally available for use by customers. The AWS Europe (Stockholm) Region is our fifth European Region, joining Dublin, Frankfurt, London, and Paris. With this launch, AWS now provides 60 Availability Zones, with another 12 zones and four Regions expected to come online by 2020 in Bahrain, Cape Town, Hong Kong, and Milan. Starting today, developers, startups, and enterprises?as well as government, education, and non-profit organizations?can use the new AWS Europe (Stockholm) Region.
Expanding the AWS Cloud ? Introducing the AWS Europe (Stockholm) Region
In April 2017, Amazon Web Services announced that it would launch a new AWS infrastructure region Region in Sweden. Today, I'm happy to announce that the AWS Europe (Stockholm) Region, our 20th Region globally, is now generally available for use by customers. The AWS Europe (Stockholm) Region is our fifth European Region, joining Dublin, Frankfurt, London, and Paris. With this launch, AWS now provides 60 Availability Zones, with another 12 zones and four Regions expected to come online by 2020 in Bahrain, Cape Town, Hong Kong, and Milan.
Amazon Redshift and the art of performance optimization in the cloud
People often ask me if developing for the cloud is any different from developing on-premises software. It really is. In this post, I show some of the reasons why that's true, using the Amazon Redshift team and the approach they have taken to improve the performance of their data warehousing service as an example. The Amazon Redshift team has delivered remarkable gains using a few simple engineering techniques: Leveraging fleet telemetry Verifying benchmark claims Optimizing performance for bursts of user activity Leveraging fleet telemetry The biggest difference between developing for the cloud and developing on-premises software is that in the cloud, you have much better access to how your customers are using your services.
Amazon Redshift and the art of performance optimization in the cloud
People often ask me if developing for the cloud is any different from developing on-premises software. It really is. In this post, I show some of the reasons why that's true, using the Amazon Redshift team and the approach they have taken to improve the performance of their data warehousing service as an example. The Amazon Redshift team has delivered remarkable gains using a few simple engineering techniques: Leveraging fleet telemetry Verifying benchmark claims Optimizing performance for bursts of user activity
Ciao Milano! ? An AWS Region is coming to Italy!
Today, I am happy to announce our plans to open a new AWS Region in Italy! The AWS Europe (Milan) Region is the 25th AWS Region that we've announced globally. It's the sixth AWS Region in Europe, joining existing regions in France, Germany, Ireland, the UK, and the new Region that we recently announced in Sweden. The AWS Europe (Milan) Region will have three Availability Zones and be ready for customers in early 2020. Currently we have 57 Availability Zones across 19 technology infrastructure Regions. As of this announcement, another five AWS Regions and 15 Availability Zones are coming over the next year in Bahrain, Hong Kong SAR, Italy, South Africa, and Sweden.
Ciao Milano! ? An AWS Region is coming to Italy!
Today, I am happy to announce our plans to open a new AWS Region in Italy! The AWS Europe (Milan) Region is the 25th AWS Region that we've announced globally. It's the sixth AWS Region in Europe, joining existing regions in France, Germany, Ireland, the UK, and the new Region that we recently announced in Sweden.
Expanding the Cloud ? The Second AWS GovCloud (US) Region, AWS GovCloud (US-East)
Today, I'm happy to announce that the AWS GovCloud (US-East) Region, our 19th global infrastructure Region, is now available for use by customers in the US. With this launch, AWS now provides 57 Availability Zones, with another 12 zones and four Regions in Bahrain, Cape Town, Hong Kong SAR, and Stockholm expected to come online by 2020. The AWS GovCloud (US-East) Region is our second AWS GovCloud (US) Region, joining AWS GovCloud (US-West) to further help US government agencies, the contractors that serve them, and organizations in highly regulated industries move more of their workloads to the AWS Cloud by implementing a number of US government-specific regulatory requirements.
Expanding the Cloud ? The Second AWS GovCloud (US) Region, AWS GovCloud (US-East)
Today, I'm happy to announce that the AWS GovCloud (US-East) Region, our 19th global infrastructure Region, is now available for use by customers in the US. With this launch, AWS now provides 57 Availability Zones, with another 12 zones and four Regions in Bahrain, Cape Town, Hong Kong SAR, and Stockholm expected to come online by 2020.
Expanding the AWS Cloud ? An AWS Region is coming to South Africa!
Expanding the AWS Cloud?An AWS Region is coming to South Africa! Today, I am excited to announce our plans to open a new AWS Region in South Africa! AWS is committed to South Africa's transformation. The AWS Africa (Cape Town) Region is another milestone of our growth and part of our long-term investment in and commitment to the country. It is our first Region in Africa, and we're shooting to have it ready in the first half of 2020. The new AWS Africa (Cape Town) Region will have three Availability Zones and provide lower latency to end users across Sub-Saharan Africa.
Expanding the AWS Cloud ? An AWS Region is coming to South Africa!
Today, I am excited to announce our plans to open a new AWS Region in South Africa! AWS is committed to South Africa's transformation. The AWS Africa (Cape Town) Region is another milestone of our growth and part of our long-term investment in and commitment to the country. It is our first Region in Africa, and we're shooting to have it ready in the first half of 2020.
Make your voice count by simply saying, "Alexa, let's chat."
A while back I wrote about the Alexa Prize, a university competition where participating teams are creating socialbots focused on advancing computer to human interaction. We are now in year two, heading into the final stretch for 2018 and I thought I would give everyone an update. For those who aren't familiar, Alexa Prize teams use customer feedback to advance several areas of conversational AI, with the grand challenge being a socialbotthat can engage coherently for 20 minutes in a fun, high-quality conversation on popular topics such as entertainment, sports, technology, and fashion. Teams are thinking big about how to make strides in areas including knowledge acquisition, natural language understanding and generation, context modeling, common sense reasoning, and dialog planning.
Make your voice count by simply saying, "Alexa, let's chat."
A while back I wrote about the Alexa Prize, a university competition where participating teams are creating socialbots focused on advancing computer to human interaction. We are now in year two, heading into the final stretch for 2018 and I thought I would give everyone an update. For those who aren't familiar, Alexa Prize teams use customer feedback to advance several areas of conversational AI, with the grand challenge being a socialbotthat can engage coherently for 20 minutes in a fun, high-quality conversation on popular topics such as entertainment, sports, technology, and fashion.
A one size fits all database doesn't fit anyone
A common question that I get is why do we offer so many database products? The answer for me is simple: Developers want their applications to be well architected and scale effectively. To do this, they need to be able to use multiple databases and data models within the same application. Seldom can one database fit the needs of multiple distinct use cases. The days of the one-size-fits-all monolithic database are behind us, and developers are now building highly distributed applications using a multitude of purpose-built databases. Developers are doing what they do best: breaking complex applications into smaller pieces and then picking the best tool to solve each problem.
A one size fits all database doesn't fit anyone
A common question that I get is why do we offer so many database products? The answer for me is simple: Developers want their applications to be well architected and scale effectively. To do this, they need to be able to use multiple databases and data models within the same application.
The workplace of the future
This article titled "Die Arbeitswelt der Zukunft" appeared in German last week in the "Digitalisierung" column of Wirtschaftwoche. The workplace of the future We already have an idea of how digitalization, and above all new technologies like machine learning, big-data analytics or IoT, will change companies' business models and are already changing them on a wide scale. So now's the time to examine more closely how different facets of the workplace will look and the role humans will have. In fact, the future is already here but it's still not evenly distributed. Science fiction author William Gibson said that nearly 20 years ago.
The workplace of the future
This article titled “Die Arbeitswelt der Zukunft” appeared in German last week in the “Digitalisierung” column of Wirtschaftwoche. The workplace of the future We already have an idea of how digitalization, and above all new technologies like machine learning, big-data analytics or IoT, will change companies' business models ? and are already changing them on a wide scale.
Changing the calculus of containers in the cloud
I wrote to you over two years ago about what happens under the hood of Amazon ECS. Last year at re:Invent, we launched AWS Fargate, and today, I want to explore how Fargate fundamentally changes the landscape of container technology. I spend a lot of time talking to our customers and leaders at Amazon about innovation. One of the things I've noticed is that ideas and technologies which dramatically change the way we do things are rarely new. They're often the combination of an existing concept with an approach, technology, or capability in a particular way that's never been successfully tried before.
Changing the calculus of containers in the cloud
I wrote to you over two years ago about what happens under the hood of Amazon ECS. Last year at re:Invent, we launched AWS Fargate, and today, I want to explore how Fargate fundamentally changes the landscape of container technology. I spend a lot of time talking to our customers and leaders at Amazon about innovation.
Looking back at 10 years of compartmentalization at AWS
At AWS, we don't mark many anniversaries. But every year when March 14th comes around, it's a good reminder that Amazon S3 originally launched on Pi Day, March 14, 2006. The Amazon S3 team still celebrate with homemade pies! March 26, 2008 doesn't have any delicious desserts associated with it, but that's the day when we launched Availability Zones for Amazon EC2. A concept that has changed infrastructure architecture is now at the core of both AWS and customer reliability and operations. Powering the virtual instances and other resources that make up the AWS Cloud are real physical data centers with AWS servers in them.
Looking back at 10 years of compartmentalization at AWS
At AWS, we don't mark many anniversaries. But every year when March 14th comes around, it's a good reminder that Amazon S3 originally launched on Pi Day, March 14, 2006. The Amazon S3 team still celebrate with homemade pies! March 26, 2008 doesn't have any delicious desserts associated with it, but that's the day when we launched Availability Zones for Amazon EC2.
Infinitely scalable machine learning with Amazon SageMaker
In machine learning, more is usually more. For example, training on more data means more accurate models. At AWS, we continue to strive to enable builders to build cutting-edge technologies faster in a secure, reliable, and scalable fashion. Machine learning is one such transformational technology that is top of mind not only for CIOs and CEOs, but also developers and data scientists. Last re:Invent, to make the problem of authoring, training, and hosting ML models easier, faster, and more reliable, we launched Amazon SageMaker. Now, thousands of customers are trying Amazon SageMaker and building ML models on top of their data lakes in AWS.
Infinitely scalable machine learning with Amazon SageMaker
In machine learning, more is usually more. For example, training on more data means more accurate models. At AWS, we continue to strive to enable builders to build cutting-edge technologies faster in a secure, reliable, and scalable fashion. Machine learning is one such transformational technology that is top of mind not only for CIOs and CEOs, but also developers and data scientists.
Unlocking Enterprise systems using voice
At Amazon, we are heavily invested in machine learning (ML), and are developing new tools to help developers quickly and easily build, train, and deploy ML models. The power of ML is in its ability to unlock a new set of capabilities that create value for consumers and businesses. A great example of this is the way we are using ML to deal with one of the world's biggest and most tangled datasets: human speech. Voice-driven conversation has always been the most natural way for us to communicate. Conversations are personal and they convey context, which helps us to understand each other.
Unlocking Enterprise systems using voice
At Amazon, we are heavily invested in machine learning (ML), and are developing new tools to help developers quickly and easily build, train, and deploy ML models. The power of ML is in its ability to unlock a new set of capabilities that create value for consumers and businesses. A great example of this is the way we are using ML to deal with one of the world's biggest and most tangled datasets: human speech.
Rethinking the 'production' of data
This article titled "Daten müssen strategischer Teil des Geschäfts werden" appeared in German last week in the "IT und Datenproduktion" column of Wirtschaftwoche. How companies can use ideas from mass production to create business with data Strategically, IT doesn't matter. That was the provocative thesis of a much-talked-about article from 2003 in the Harvard Business Review by the US publicist Nicolas Carr. Back then, companies spent more than half of their entire investment for their IT, in a non-differentiating way. In a world in which tools are equally accessible for every company, they wouldn't offer any competitive advantage so went the argument.
Rethinking the 'production' of data
This article titled “Daten müssen strategischer Teil des Geschäfts werden” appeared in German last week in the “IT und Datenproduktion” column of Wirtschaftwoche. How companies can use ideas from mass production to create business with data Strategically, IT doesn't matter. That was the provocative thesis of a much-talked-about article from 2003 in the Harvard Business Review by the US publicist Nicolas Carr.
'Paris s'éveille'! Introducing the AWS EU (Paris) Region
Today, I'm happy to announce that the AWS EU (Paris) Region, our 18th technology infrastructure Region globally, is now generally available for use by customers worldwide. With this launch, AWS now provides 49 Availability Zones, with another 12 Availability Zones and four Regions in Bahrain, Hong Kong, Sweden, and a second AWS GovCloud (US) Region expected to come online by early 2019. In France, you can find one of the most vibrant startup ecosystems in the world, a strong research community, excellent energy, telecom, and transportation infrastructure, a very strong agriculture and food industry, and some of the most influential luxury brands in the world.
'Paris s'éveille'! Introducing the AWS EU (Paris) Region
Today, I'm happy to announce that the AWS EU (Paris) Region, our 18th technology infrastructure Region globally, is now generally available for use by customers worldwide. With this launch, AWS now provides 49 Availability Zones, with another 12 Availability Zones and four Regions in Bahrain, Hong Kong, Sweden, and a second AWS GovCloud (US) Region expected to come online by early 2019.
Expanding the AWS Cloud: Introducing the AWS China (Ningxia) Region
Today, I am happy to announce the general availability of AWS China (Ningxia) Region, operated by Ningxia Western Cloud Data Technology Co. Ltd. (NWCD). This is our 17th Region globally, and the second in China. To comply with China's legal and regulatory requirements, AWS has formed a strategic technology collaboration with NWCD to operate and provide services from the AWS China (Ningxia) Region. Founded in 2015, NWCD is a licensed data center and cloud services provider, based in Ningxia, China. Coupled with the AWS China (Beijing) Region operated by Sinnet, the AWS China (Ningxia) Region, operated by NWCD, serves as the foundation for new cloud initiatives in China, especially in Western China.
Expanding the AWS Cloud: Introducing the AWS China (Ningxia) Region
Today, I am happy to announce the general availability of AWS China (Ningxia) Region, operated by Ningxia Western Cloud Data Technology Co. Ltd. (NWCD). This is our 17th Region globally, and the second in China. To comply with China's legal and regulatory requirements, AWS has formed a strategic technology collaboration with NWCD to operate and provide services from the AWS China (Ningxia) Region.
Accelerate Machine Learning with Amazon SageMaker
Applications based on machine learning (ML) can provide tremendous business value. However, many developers find them difficult to build and deploy. As there are few individuals with this expertise, an easier process presents a significant opportunity for companies who want to accelerate their ML usage. Though the AWS Cloud gives you access to the storage and processing power required for ML, the process for building, training, and deploying ML models has unique challenges that often block successful use of this powerful new technology. The challenges begin with collecting, cleaning, and formatting training data. After the dataset is created, you must scale the processing to handle the data, which can often be a blocker.
Accelerate Machine Learning with Amazon SageMaker
Applications based on machine learning (ML) can provide tremendous business value. However, many developers find them difficult to build and deploy. As there are few individuals with this expertise, an easier process presents a significant opportunity for companies who want to accelerate their ML usage. Though the AWS Cloud gives you access to the storage and processing power required for ML, the process for building, training, and deploying ML models has unique challenges that often block successful use of this powerful new technology.
Scaling Amazon ElastiCache for Redis with Online Cluster Resizing
Amazon ElastiCache embodies much of what makes fast data a reality for customers looking to process high volume data at incredible rates, faster than traditional databases can manage. Developers love the performance, simplicity, and in-memory capabilities of Redis, making it among the most popular NoSQL key-value stores. Redis's microsecond latency has made it a de facto choice for caching. Its support for advanced data structures (for example, lists, sets, and sorted sets) also enables a variety of in-memory use cases such as leaderboards, in-memory analytics, messaging, and more. Four years ago, as part of our AWS fast data journey, we introduced Amazon ElastiCache for Redis, a fully managed, in-memory data store that operates at microsecond latency.
Scaling Amazon ElastiCache for Redis with Online Cluster Resizing
Amazon ElastiCache embodies much of what makes fast data a reality for customers looking to process high volume data at incredible rates, faster than traditional databases can manage. Developers love the performance, simplicity, and in-memory capabilities of Redis, making it among the most popular NoSQL key-value stores. Redis's microsecond latency has made it a de facto choice for caching.
A Decade of Dynamo: Powering the next wave of high-performance, internet-scale applications
Today marks the 10 year anniversary of Amazon's Dynamo whitepaper, a milestone that made me reflect on how much innovation has occurred in the area of databases over the last decade and a good reminder on why taking a customer obsessed approach to solving hard problems can have lasting impact beyond your original expectations. It all started in 2004 when Amazon was running Oracle's enterprise edition with clustering and replication. We had an advanced team of database administrators and access to top experts within Oracle. We were pushing the limits of what was a leading commercial database at the time and were unable to sustain the availability, scalability and performance needs that our growing Amazon business demanded.
A Decade of Dynamo: Powering the next wave of high-performance, internet-scale applications
Today marks the 10 year anniversary of Amazon's Dynamo whitepaper, a milestone that made me reflect on how much innovation has occurred in the area of databases over the last decade and a good reminder on why taking a customer obsessed approach to solving hard problems can have lasting impact beyond your original expectations.
As-Salaam-Alaikum: The cloud arrives in the Middle East!
Today, I am excited to announce plans for Amazon Web Services (AWS) to bring an infrastructure Region to the Middle East! This move is another milestone in our global expansion and mission to bring flexible, scalable, and secure cloud computing infrastructure to organizations around the world. Based in Bahrain, this will be the first Region for AWS in the Middle East. The Region will be in the heart of Gulf Cooperation Council (GCC) countries, and we're aiming to have it ready by early 2019. This Region will consist of three Availability Zones at launch, and it will provide even lower latency to users across the Middle East.
As-Salaam-Alaikum: The cloud arrives in the Middle East!
Today, I am excited to announce plans for Amazon Web Services (AWS) to bring an infrastructure Region to the Middle East! This move is another milestone in our global expansion and mission to bring flexible, scalable, and secure cloud computing infrastructure to organizations around the world. Based in Bahrain, this will be the first Region for AWS in the Middle East.
AI for everyone - How companies can benefit from the advance of machine learning
This article titled "Wie Unternehmen vom Vormarsch des maschinellen Lernens profitieren können" appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. When a technology has its breakthrough, can often only be determined in hindsight. In the case of artificial intelligence (AI) and machine learning (ML), this is different. ML is that part of AI that describes rules and recognizes patterns from large amounts of data in order to predict future data. Both concepts are virtually omnipresent and at the top of most buzzword rankings. Personally, I think and this is clearly linked to the rise of AI and ML that there has never been a better time than today to develop smart applications and use them.
AI for everyone - How companies can benefit from the advance of machine learning
This article titled “Wie Unternehmen vom Vormarsch des maschinellen Lernens profitieren können” appeared in German last week in the “Digitaliserung” column of Wirtschaftwoche. When a technology has its breakthrough, can often only be determined in hindsight. In the case of artificial intelligence (AI) and machine learning (ML), this is different. ML is that part of AI that describes rules and recognizes patterns from large amounts of data in order to predict future data.
Improving Customer Service with Amazon Connect and Amazon Lex
Customer service is central to the overall customer experience that all consumers are familiar with when communicating with companies. That experience is often tested when we need to ask for help or have a question to be answered. Unfortunately, we've become accustomed to providing the same information multiple times, waiting on hold, and generally spending a lot more time than we expected to resolve our issue when we call customer service. When you call for customer assistance, you often need to wait for an agent to become available after navigating a set of menus. This means that you're going to wait on hold regardless of whether your issue is simple or complex.
Improving Customer Service with Amazon Connect and Amazon Lex
Customer service is central to the overall customer experience that all consumers are familiar with when communicating with companies. That experience is often tested when we need to ask for help or have a question to be answered. Unfortunately, we've become accustomed to providing the same information multiple times, waiting on hold, and generally spending a lot more time than we expected to resolve our issue when we call customer service.
Stop waiting for perfection and learn from your mistakes
This article titled "Wartet nicht auf Perfektion lernt aus euren Fehlern!" appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. "Man errs as long as he doth strive." Goethe, the German prince of poets, knew that already more than 200 years ago. His words still ring true today, but with a crucial difference: Striving alone is not enough. You have to strive faster than the rest. And while there's nothing wrong with striving for perfection, in today's digital world you can no longer wait until your products are near perfection before offering them to your customers. If so, you will fall behind in your market.
Stop waiting for perfection and learn from your mistakes
This article titled “Wartet nicht auf Perfektion ? lernt aus euren Fehlern!” appeared in German last week in the “Digitaliserung” column of Wirtschaftwoche. "Man errs as long as he doth strive." Goethe, the German prince of poets, knew that already more than 200 years ago. His words still ring true today, but with a crucial difference: Striving alone is not enough.
Amazon DynamoDB Accelerator (DAX): Speed Up DynamoDB Response Times from Milliseconds to Microseconds without Application Rewrite.
Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX), a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. You can add DAX to your existing DynamoDB applications with just a few clicks in the AWS Management Console no application rewrites required. DynamoDB has come a long way in the 5 years since we announced its availability in January 2012. As we said at the time, DynamoDB was a result of 15 years of learning in the area of large scale non-relational databases and cloud services.
Amazon DynamoDB Accelerator (DAX): Speed Up DynamoDB Response Times from Milliseconds to Microseconds without Application Rewrite.
Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX), a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. You can add DAX to your existing DynamoDB applications with just a few clicks in the AWS Management Console ? no application rewrites required.
Expanding the Cloud An AWS Region is coming to Hong Kong
Today, I am very excited to announce our plans to open a new AWS Region in Hong Kong! The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong. The new AWS Asia Pacific (Hong Kong) Region will have three Availability Zones and be ready for customers for use in 2018. Over the past decade, we have seen tremendous growth at AWS. As a result, we have opened 43 Availability Zones across 16 AWS Regions worldwide. Last year, we opened new regions in Korea, India, the US, Canada, and the UK.
Expanding the Cloud ? An AWS Region is coming to Hong Kong
Today, I am very excited to announce our plans to open a new AWS Region in Hong Kong! The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong. The new AWS Asia Pacific (Hong Kong) Region will have three Availability Zones and be ready for customers for use in 2018.
Unlocking the Value of Device Data with AWS Greengrass.
Unlocking the value of data is a primary goal that AWS helps our customers to pursue. In recent years, an explosion of intelligent devices have created oceans of new data across many industries. We have seen that such devices can benefit greatly from the elastic resources of the cloud. This is because data gets more valuable when it can be processed together with other data. At the same time, it can be valuable to process some data right at the source where it is generated. Some applications medical equipment, industrial machinery, and building automation are just a few can't rely exclusively on the cloud for control, and require some form of local storage and execution.
Unlocking the Value of Device Data with AWS Greengrass.
Unlocking the value of data is a primary goal that AWS helps our customers to pursue. In recent years, an explosion of intelligent devices have created oceans of new data across many industries. We have seen that such devices can benefit greatly from the elastic resources of the cloud. This is because data gets more valuable when it can be processed together with other data.
Weekend Reading: Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases.
In many, high-throughput, OLTP style applications the database plays a crucial role to achieve scale, reliability, high-performance and cost efficiency. For a long time, these requirements were almost exclusively served by commercial, proprietary databases. Soon after the launch of Amazon Relation Dabase Service (RDS) AWS customers were giving us feedback that they would love to migrate to RDS but what they would love even more was if we could also unshackle them from the high-cost, punitive licensing schemes that came with the proprietary databases. They would love to migrate to an open-source style database like MySQL or PostgresSQL if only such a database could meet the enterprise-grade reliability and performance these high-scale applications required.
Weekend Reading: Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases.
In many high-throughput OLTP style applications, the database plays a crucial role in achieving scale, reliability, high-performance, and cost efficiency. For a long time, these requirements were almost exclusively served by commercial, proprietary databases. Soon after the launch of the AWS Relational Database Service (RDS) customers gave us feedback that they would love to migrate to RDS.
Faster, higher, stronger: How the digitalization of industry is redefining value creation
This article titled "Wie die Digitalisierung Wertschöpfung neu definiert" appeared in German last week in the "Größer, höher, weiter (bigger, higher, further)" column of Wirtschaftwoche. Germany's "hidden champions" family-owned companies, engineering companies, specialists are unique in the world. They stand for quality, reliability and a high degree of know-how in manufacturing. Hidden champions play a significant role in the German economy; as a result, Germany has become one of the few countries in Western Europe where manufacturing accounts for more than 20% of GDP. By contrast, neighboring countries have seen a continuous decline in their manufacturing base. What's more, digital technologies and business models that are focused on Industry 4.0 (i.e., the term invented in Germany to refer to the digitalization of production) have the potential to reinforce Germany's lead even more.
Faster, higher, stronger: How the digitalization of industry is redefining value creation
This article titled “Wie die Digitalisierung Wertschöpfung neu definiert” appeared in German last week in the “Größer, höher, weiter (bigger, higher, further)” column of Wirtschaftwoche. Germany's "hidden champions" ? family-owned companies, engineering companies, specialists ? are unique in the world. They stand for quality, reliability and a high degree of know-how in manufacturing.
Coming to STATION F: The first Mentor's Office powered by AWS!
I am excited to announce that AWS is opening its first Mentor's Office at STATION F in Paris! The Mentor's Office is a workplace exclusively dedicated to meetings between AWS experts and the startups. STATION F is the world's biggest startup campus. With this special offer starting at the end of June, at the campus opening, AWS increases the support already available to startup customers in France. All year long, AWS experts will deliver technical and business assistance to startups based on campus. AWS Solutions Architects will meet startup members for face-to-face sessions, to share guidance on how cloud services can be used for their specific use cases, workloads, or applications.
Coming to STATION F: The first Mentor's Office powered by AWS!
I am excited to announce that AWS is opening its first Mentor's Office at STATION F in Paris! The Mentor's Office is a workplace exclusively dedicated to meetings between AWS experts and the startups. STATION F is the world's biggest startup campus. With this special offer starting at the end of June, at the campus opening, AWS increases the support already available to startup customers in France.
Back-to-Basics Weekend Reading: Twenty years of functional MRI: The science and the stories.
I will be returning this weekend to the US from a very successful AWS Summit in Sydney, so I have ample time to read during travels. This weekend however I would like to take a break from reading historical computer science material, to catch up on another technology I find fascinating, that of functional Magnetic Resonace Imaging, aka fMRI. fMRI is a functional imagine technology, meaning that it just records the state of the brain at one particular point in time, but the changing state over a period of time. The basic technology records brain activity by measuring changes in blood flow through the brain.
Back-to-Basics Weekend Reading: Twenty years of functional MRI: The science and the stories.
I will be returning this weekend to the US from a very successful AWS Summit in Sydney, so I have ample time to read during travels. This weekend however I would like to take a break from reading historical computer science material, to catch up on another technology I find fascinating, that of functional Magnetic Resonace Imaging, aka fMRI.
Välkommen till Stockholm ? An AWS Region is coming to the Nordics
Today, I am very excited to announce our plans to open a new AWS Region in the Nordics! The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden. The new AWS EU (Stockholm) Region will have three Availability Zones and will be ready for customers to use in 2018.
Välkommen till Stockholm An AWS Region is coming to the Nordics
Today, I am very excited to announce our plans to open a new AWS Region in the Nordics! The new region will give Nordic-based businesses, government organisations, non-profits, and global companies with customers in the Nordics, the ability to leverage the AWS technology infrastructure from data centers in Sweden. The new AWS EU (Stockholm) Region will have three Availability Zones and will be ready for customers to use in 2018. Over the past decade, we have seen tremendous growth at AWS. As a result, we have opened 42 Availability Zones across 16 AWS Regions worldwide. Last year, we opened new regions in Canada, India, Korea, the UK, and the US.
Back-to-Basics Weekend Reading: An Implementation of a Log-Structured File System
This weekend I am travelling to Australia for the first AWS Summit of 2017. I find on such a long trip, to keep me from getting distracted, I need an exciting paper that is easy to read. Last week's 'Deep Learning' overview would have not met those requirements. One topic that always gets me excited is how to take computer science research and implement it in production systems. There are often so many obstacles that we do not see much of this work happening. For example when building Dynamo, where we put a collection of different research technologies together in production, we struggled with all the assumptions the researchers had made.
Back-to-Basics Weekend Reading: An Implementation of a Log-Structured File System
This weekend I am travelling to Australia for the first AWS Summit of 2017. I find on such a long trip, to keep me from getting distracted, I need an exciting paper that is easy to read. Last week’s ‘Deep Learning’ overview would have not met those requirements. One topic that always gets me excited is how to take computer science research and implement it in production systems.
Back-to-Basics Weekend Reading: Deep learning in neural networks
In the past few years, we have seen an explosion in the use of Deep Learning as its software platforms mature and the supporting hardware, especially GPUs with larger memories, become widely available. Even though this is a recent development, Deep Learning has deep historical roots, tracing back all the way to the sixties, or maybe even earlier. By reading up on its history, we get a better understanding of the current state of the art of Deep Learning algorithms and the Neural Networks that you build with them. There is such a broad set of papers to read if we would want to dive deep into the history, that it would take us multiple weekends.
Back-to-Basics Weekend Reading: Deep learning in neural networks
In the past few years, we have seen an explosion in the use of ‘Deep Learning’ as its software platforms and the supporting hardware mature, especially as GPUs with larger memories become widely available. Even though this is a recent development, ‘Deep Learning’ has entrenched historical roots, tracing back all the way to the sixties or possibly earlier.
Amazon Makes it Free for Developers to Build and Host Most Alexa Skills Using AWS
Amazon today announced a new program that will make it free for tens of thousands of Alexa developers to build and host most Alexa skills using Amazon Web Services (AWS). Many Alexa skill developers currently take advantage of the AWS Free Tier, which offers one million AWS Lambda requests and up to 750 hours of Amazon Elastic Compute Cloud (Amazon EC2) compute time per month at no charge. However, if developers exceed the AWS Free Tier limits, they may incur AWS usage fees each month. Now, developers with a live Alexa skill can apply to receive a $100 AWS promotional credit and can earn an additional $100 per month in AWS promotional credits if they incur AWS usage charges for their skill making it free for developers to build and host most Alexa skills.
Amazon Makes it Free for Developers to Build and Host Most Alexa Skills Using AWS
Amazon today announced a new program that will make it free for tens of thousands of Alexa developers to build and host most Alexa skills using Amazon Web Services (AWS). Many Alexa skill developers currently take advantage of the AWS Free Tier, which offers one million AWS Lambda requests and up to 750 hours of Amazon Elastic Compute Cloud (Amazon EC2) compute time per month at no charge.
How companies can become magnets for digital talent
This article titled "Wie Unternehmen digitale Talente anziehen" appeared in German last week in the "Tipps für Arbeitgeber" section of Wirtschaftwoche. The rise in digital business models is a huge challenge for recruiting and talent selection. The sort of skills businesses need today are in short supply. How companies can prepare themselves to attract the best talents for shaping their digital business. Digitalization offers almost endless possibilities to communicate faster, work more efficiently, and be more creative in real-time. But groundbreaking digital business models need pioneers: creators, forward-looking thinkers and inventors who don't hesitate to leave the beaten path, embody ownership, and who understand how to translate customers' wishes into superb new products, services and solutions that evolve with speed.
How companies can become magnets for digital talent
This article titled “Wie Unternehmen digitale Talente anziehen” appeared in German last week in the “Tipps für Arbeitgeber” section of Wirtschaftwoche. The rise in digital business models is a huge challenge for recruiting and talent selection. The sort of skills businesses need today are in short supply. How companies can prepare themselves to attract the best talents for shaping their digital business.
Back-to-Basic Weekend Reading: The Foundations of Blockchain
More and more we see stories appearing, like this one in HBR by MIT Media Lab's Joi Ito and crew. It praises the power of blockchain as a disruptive technology, on par with how "the internet" changed everything. I am always surprised to see that these far-reaching predictions are made, without diving into the technology itself. This weekend I would like to read about some of the technologies that predate blockchain, as they are its fundamental building blocks. Blockchain technology first came on the scene in 2008, as a core component of the bitcoin cryptocurrency. Blockchain provides transactional, distributed ledger functionality that can operate without a centralized, trusted authority.
Back-to-Basics Weekend Reading: The Foundations of Blockchain
More and more we see stories appearing, like this one in HBR by MIT Media Lab’s Joi Ito and crew. It praises the power of blockchain as a disruptive technology, on par with how “the internet” changed everything. I am always surprised to see that these far-reaching predictions are made, without diving into the technology itself.
Back-to-Basic Weekend Reading: Why Do Computers Stop and What Can Be Done About It?
"Everything fails, all the time." A humble computer scientist once said. With all the resources we have today, it is easier for us to achieve fault-tolerance than it was many decades ago when computers began playing a role in critical systems such as health care, air traffic control and financial market systems. In the early days, the thinking was to use a hardware approach to achieve fault-tolerance. It was not until the mid-nineties that software fault-tolerance became more acceptable. Tandem Computer was one of the pioneers in building these fault-tolerant, mission-critical systems. They used a shared-nothing multi-cpu approach. This is where each CPU had its own memory- and io-bus, and all were connected through a replicated shared bus, over which the independent OS instances could communication and run in lock step.
Back-to-Basics Weekend Reading: Why Do Computers Stop and What Can Be Done About It?
“Everything fails, all the time.” A humble computer scientist once said. With all the resources we have today, it is easier for us to achieve fault-tolerance than it was many decades ago when computers began playing a role in critical systems such as health care, air traffic control and financial market systems.
Back-to-Basic Weekend Reading: Byzantine Generals
In Reliable Distribution Systems, we need to handle different failure scenarios. Many of those deal with message loss and process failure. However, there is a class of scenarios that deal with malfunctioning processes, which send out conflicting information. The challenge is developing algorithms that can reach an agreement in the presence of these failures. Lamport described that he was frustrated with the attention that Dijkstra had gotten for describing a computer science problem as the story of dining philosophers. He decided the best way to attract attention to a particular distributed systems problem was to present it in terms of a story; hence, the Byzantine Generals.
Back-to-Basics Weekend Reading: Byzantine Generals
In Reliable Distributed Systems, we need to handle different failure scenarios. Many of those deal with message loss and process failure. However, there is a class of scenarios that deal with malfunctioning processes, which send out conflicting information. The challenge is developing algorithms that can reach an agreement in the presence of these failures.
Back-to-Basic Weekend Reading: Monte-Carlo Methods
I always enjoy looking for solutions to difficult challenges in non-obvious places. That is probably why I like using probabilistic techniques for problems that appear to be hard, or impossible to solve deterministically. The probabilistic approach may not result in the perfect result, but it may get you very close, and much faster than deterministic techniques (which may even be computationally impossible). Some of the earliest approaches using probabilities in physics experiments resulted in the Monte Carlo methods. Their essential idea is using randomness to solve problems that might be deterministic in principle. These are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results.
Back-to-Basic Weekend Reading: Monte-Carlo Methods
I always enjoy looking for solutions to difficult challenges in non-obvious places. That is probably why I like using probabilistic techniques for problems that appear to be hard, or impossible to solve deterministically. The probabilistic approach may not result in the perfect result, but it may get you very close, and much faster than deterministic techniques (which may even be computationally impossible).
Back-to-Basics Weekend Reading - Bloom Filters
Listening to the "Algorithms to Live By" audio on my commute this morning I once again was struck by the beauty of Bloom Filters. So I think it is time to resurrect the Back-to-Basics Weekend Reading series, as I will be re-reading some really fundamental CS papers this weekend. In the past I have done some weekend reading about Counting Bloom Filters, but now I am going even more fundamental. Bloom Filters, conceived by Burton Bloom in 1970, are probabilistic data structures to test whether an item is in a set. False positives are possible but false negatives are not. So if a bit in the filter is not set you can be sure the item is not in the set, if it is set, the mapped item may be in the set.
Back-to-Basics Weekend Reading - Bloom Filters
Listening to the “Algorithms to Live By” audio on my commute this morning, once again I was struck by the beauty of Bloom Filters. So, I decided it is time to resurrect the ‘Back-to-Basics Weekend Reading’ series, as I will be re-reading some fundamental CS papers this weekend. In the past, I have done some weekend reading about Counting Bloom Filters, but now I am going even more fundamental, and I invite you to join me.
A survival strategy for the digital transformation
This article titled “Überlebensstrategie für die digitale Transformation” appeared in German last week in the “Die Zukunft beginnt heute (the future starts today)” section of Wirtschaftwoche. Smaller companies have a lot to gain in the digital era ? provided they adopt the right mindset. The winners will be those that view their business from the eyes of their customers and understand that fast-paced innovation is the key to long-term growth.
A survival strategy for the digital transformation
This article titled "Überlebensstrategie für die digitale Transformation" appeared in German last week in the "Die Zukunft beginnt heute (the future starts today)" section of Wirtschaftwoche. Smaller companies have a lot to gain in the digital era provided they adopt the right mindset. The winners will be those that view their business from the eyes of their customers and understand that fast-paced innovation is the key to long-term growth. With this mindset they can take on even the largest enterprises who are slow to adapt to the fast moving digital reality. The digital era is here. Companies that haven't realized that by now will fall behind.
Expanding the AWS Cloud: Introducing the AWS Europe (London) Region
In November 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in the United Kingdom. Today, I'm happy to announce that the AWS Europe (London) Region, our 16th technology infrastructure region globally, is now generally available for use by customers worldwide. UK companies are using AWS to innovate across diverse industries, such as energy, manufacturing, medicaments, retail, media, and financial services and the UK is home to some of the world's most forward-thinking businesses. These include startups like Fanduel, JustEat, and Monzo to enterprises such as British Gas, Trainline, Perkins, News UK, the Financial Times. The British Government is also helping to drive innovation and has embraced a cloud-first policy for technology adoption.
Expanding the AWS Cloud Introducing the AWS Europe (London) Region
In November 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in the United Kingdom. Today, I'm happy to announce that the AWS Europe (London) Region, our 16th technology infrastructure region globally, is now generally available for use by customers worldwide. UK companies are using AWS to innovate across diverse industries, such as energy, manufacturing, medicaments, retail, media, and financial services and the UK is home to some of the world's most forward-thinking businesses.
Expanding the AWS Cloud: Introducing the AWS Canada (Central) Region
Earlier this year, Amazon Web Services (AWS) announced it would launch a new AWS infrastructure region in Montreal, Quebec. Today, I'm happy to share that the Canada (Central) Region is available for use by customers worldwide. The AWS Cloud now operates in 40 Availability Zones within 15 geographic regions around the world, with seven more Availability Zones and three more regions coming online in China, France, and the U.K. in the coming year. The Canadian opportunity Canada has set forth a bold innovation agenda grounded in entrepreneurship, scientific research, growing small and medium-sized businesses with a focus on environmentally friendly technologies, and the transition to a digital economy.
Expanding the AWS Cloud: Introducing the AWS Canada (Central) Region
Earlier this year, Amazon Web Services (AWS) announced it would launch a new AWS infrastructure region in Montreal, Quebec. Today, I'm happy to share that the Canada (Central) Region is available for use by customers worldwide. The AWS Cloud now operates in 40 Availability Zones within 15 geographic regions around the world, with seven more Availability Zones and three more regions coming online in China, France, and the U.
Transforming Development with AWS
In my keynote at AWS re:Invent today, I announced 13 new features and services (in addition to the 15 we announced yesterday). My favorite parts of James Bond movies is are where 007 gets to visit Q to pick up and learn about new tools of the trade: super-powered tools with special features which that he can use to complete his missions, and, in some cases, get out of some nasty scrapes. Bond always seems to have the perfect tool for every situation that he finds himself in. * At AWS, we want to be the Q for developers, giving them the super-powered tools and services with deep features in the Cloud.
Transforming Development with AWS
In my keynote at AWS re:Invent today, I announced 13 new features and services (in addition to the 15 we announced yesterday). My favorite parts of James Bond movies is are where 007 gets to visit Q to pick up and learn about new tools of the trade: super-powered tools with special features which that he can use to complete his missions, and, in some cases, get out of some nasty scrapes.
Bringing the Magic of Amazon AI and Alexa to Apps on AWS.
From the early days of Amazon, Machine learning (ML) has played a critical role in the value we bring to our customers. Around 20 years ago, we used machine learning in our recommendation engine to generate personalized recommendations for our customers. Today, there are thousands of machine learning scientists and developers applying machine learning in various places, from recommendations to fraud detection, from inventory levels to book classification to abusive review detection. There are many more application areas where we use ML extensively: search, autonomous drones, robotics in fulfillment centers, text processing and speech recognition (such as in Alexa) etc. Among machine learning algorithms, a class of algorithms called deep learning has come to represent those algorithms that can absorb huge volumes of data and learn elegant and useful patterns within that data: faces inside photos, the meaning of a text, or the intent of a spoken word.After over 20 ...
Bringing the Magic of Amazon AI and Alexa to Apps on AWS.
From the early days of Amazon, Machine learning (ML) has played a critical role in the value we bring to our customers. Around 20 years ago, we used machine learning in our recommendation engine to generate personalized recommendations for our customers. Today, there are thousands of machine learning scientists and developers applying machine learning in various places, from recommendations to fraud detection, from inventory levels to book classification to abusive review detection.
MXNet - Deep Learning Framework of Choice at AWS
Machine learning is playing an increasingly important role in many areas of our businesses and our lives and is being employed in a range of computing tasks where programming explicit algorithms is infeasible. At Amazon, machine learning has been key to many of our business processes, from recommendations to fraud detection, from inventory levels to book classification to abusive review detection. And there are many more application areas where we use machine learning extensively: search, autonomous drones, robotics in fulfillment centers, text and speech recognitions, etc. Among machine learning algorithms, a class of algorithms called deep learning hascome to represent those algorithms that can absorb huge volumes of data and learn elegant and useful patterns within that data: faces inside photos, the meaning of a text, or the intent of a spoken word.
MXNet - Deep Learning Framework of Choice at AWS
Machine learning is playing an increasingly important role in many areas of our businesses and our lives and is being employed in a range of computing tasks where programming explicit algorithms is infeasible. At Amazon, machine learning has been key to many of our business processes, from recommendations to fraud detection, from inventory levels to book classification to abusive review detection.
Spice up your Analytics: Amazon QuickSight Now Generally Available in N. Virginia, Oregon, and Ireland.
Previously, I wrote about Amazon QuickSight, a new service targeted at business users that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily, and at a low cost. QuickSight is a very fast, cloud-powered, business intelligence service for the 1/10th the cost of old-guard BI solutions. Today, I am very happy to announce that QuickSight is now generally available in the N. Virginia, Oregon, and Ireland regions. When we announced QuickSight last year, we set out to help all customersregardless of their technical skillsmake sense out of their ever-growing data. As I mentioned, we live in a world where massive volumes of data are being generated, every day, from connected devices, websites, mobile apps, and customer applications running on top of AWS infrastructure.
Spice up your Analytics: Amazon QuickSight Now Generally Available in N. Virginia, Oregon, and Ireland.
Previously, I wrote about Amazon QuickSight, a new service targeted at business users that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily, and at a low cost. QuickSight is a very fast, cloud-powered, business intelligence service for the 1/10th the cost of old-guard BI solutions.
Meet the Teams Competing for the Alexa Prize
On September 29, 2016, Amazon announced the Alexa Prize, a $2.5 million university competition to advance conversational AI through voice. We received applications from leading universities across 22 countries. Each application was carefully reviewed by senior Amazon personnel against a rigorous set of criteria covering scientific contribution, technical merit, novelty, and ability to execute. Teams of scientists, engineers, user experience designers, and product managers read, evaluated, discussed, argued, and finally selected the twelve teams who would be invited to participate in the competition. Today, were excited to announce the 12 teams selected to compete with an Amazon sponsorship. In alphabetical order, they are: Carnegie-Mellon University: CMU Magnus Carnegie-Mellon University: TBD Czech Technical University, Prague: eClub Prague Heriot-Watt University, UK: WattSocialBot Princeton University: Princeton Alexa Rensselaer Polytechnic Institute: BAKAbot University of California, Berkeley: Machine Learning @ Berkeley University of California, Santa Cruz: SlugBots University of Edinburgh, UK: Edina University of ...
Meet the Teams Competing for the Alexa Prize
On September 29, 2016, Amazon announced the Alexa Prize, a $2.5 million university competition to advance conversational AI through voice. We received applications from leading universities across 22 countries. Each application was carefully reviewed by senior Amazon personnel against a rigorous set of criteria covering scientific contribution, technical merit, novelty, and ability to execute.
Welcoming Adrian Cockcroft to the AWS Team.
I am excited that Adrian Cockcroft will be joining AWS as VP of Cloud Architecture. Adrian has played a crucial role in developing the cloud ecosystem as Cloud Architect at Netflix and later as a Technology Fellow at Battery Ventures. Prior to this, he held positions as Distinguished Engineer at eBay and Sun Microsystems. One theme that has been consistent throughout his career is that Adrian has a gift for seeing the bigger engineering picture. At Netflix, Adrian played a key role in the company's much-discussed migration to a "cloud native" architecture, and the open sourcing of the widely used (and award-winning) NetflixOSS platform.
Welcoming Adrian Cockcroft to the AWS Team.
I am excited that Adrian Cockcroft will be joining AWS as VP of Cloud Architecture. Adrian has played a crucial role in developing the cloud ecosystem as Cloud Architect at Netflix and later as a Technology Fellow at Battery Ventures. Prior to this, he held positions as Distinguished Engineer at eBay and Sun Microsystems.
Expanding the AWS Cloud: Introducing the AWS US East (Ohio) Region
Today I am very happy to announce the opening of the new US East (Ohio) Region. The Ohio Region is the fifth AWS region in the US. It brings the worldwide total of AWS Availability Zones (AZs) to 38, and the number of regions globally to 14. The pace of expansion at AWS is accelerating, and Ohio is our third region launch this year. In the remainder of 2016 and inq 2017, we will launch another four AWS regions in Canada, China, the United Kingdom, and France, adding another nine AZs to our global infrastructure footprint. We strive to place customer feedback first in our considerations for where to open new regions.
Expanding the AWS Cloud: Introducing the AWS US East (Ohio) Region
Today I am very happy to announce the opening of the new US East (Ohio) Region. The Ohio Region is the fifth AWS region in the US. It brings the worldwide total of AWS Availability Zones (AZs) to 38, and the number of regions globally to 14. The pace of expansion at AWS is accelerating, and Ohio is our third region launch this year.
Accelerating Data: Faster and More Scalable ElastiCache for Redis
Fast Data is an emerging industry term for information that is arriving at high volume and incredible rates, faster than traditional databases can manage. Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis, a fully managed in-memory data store that operates at sub-millisecond latency. Since then weve introduced Amazon Kinesis for real-time streaming data, AWS Lambda for serverless processing, Apache Spark analytics on EMR, and Amazon QuickSight for high performance Business Intelligence. While caching continues to be a dominant use of ElastiCache for Redis, we see customers increasingly use it as an in-memory NoSQL database.
Accelerating Data: Faster and More Scalable ElastiCache for Redis
Fast Data is an emerging industry term for information that is arriving at high volume and incredible rates, faster than traditional databases can manage. Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis, a fully managed in-memory data store that operates at sub-millisecond latency.
Introducing the Alexa Prize, Its Day One for Voice
In the past voice interfaces were seen as gimmicks, or a nuisance for driving hands-free. The Amazon Echo and Alexa have completely changed that perception. Voice is now seen as potentially the most important interface to interact with the digitally connected world. From home automation to commerce, from news organizations to government agencies, from financial services to healthcare, everyone is working on the best way is to interact with their services if voice is the interface. Especially for the exciting case where voice is the only interface. Voice makes access to digital services far more inclusive than traditional screen-based interaction, for example, an aging population may be much more comfortable interacting with voice-based systems than through tablets or keyboards.
Introducing the Alexa Prize, It?s Day One for Voice
In the past voice interfaces were seen as gimmicks, or a nuisance for driving ?hands-free.? The Amazon Echo and Alexa have completely changed that perception. Voice is now seen as potentially the most important interface to interact with the digitally connected world. From home automation to commerce, from news organizations to government agencies, from financial services to healthcare, everyone is working on the best way is to interact with their services if voice is the interface.
Allez, rendez-vous à Paris An AWS Region is coming to France!
Today, I am very excited to announce our plans to open a new AWS Region in France! Based in the Paris area, the region will provide even lower latency and will allow users who want to store their content in datacenters in France to easily do so. The new region in France will be ready for customers to use in 2017. Over the past 10 years, we have seen tremendous growth at AWS. As a result, we have opened 35 Availability Zones (AZs), across 13 AWS Regions worldwide. We have announced several additional regions in Canada, China, Ohio, and the United Kingdom all expected in the coming months.
Allez, rendez-vous à Paris ? An AWS Region is coming to France!
Today, I am very excited to announce our plans to open a new AWS Region in France! Based in the Paris area, the region will provide even lower latency and will allow users who want to store their content in datacenters in France to easily do so. The new region in France will be ready for customers to use in 2017. Over the past 10 years, we have seen tremendous growth at AWS. As a result, we have opened 35 Availability Zones (AZs), across 13 AWS Regions worldwide. We have announced several additional regions in Canada, China, Ohio, and the United Kingdom ?
A Hungry Neighbor is an Angry Neighbor
I am very grateful that I have had the opportunity to meet with President Shimon Peres several times. Especially the first time, which was a 1:1 in his presidential residence, was an unforgettable experience. After I explained in 5 minutes the power of cloud for unlocking digital business building for everyone, he went on a lecture of half an hour how bringing economic prosperity to the region was crucial to achieving a long lasting peace. " A hungry neighbor is an angry neighbor". He strongly believed peace in the Middle East was attainable, and I have no reason to doubt him.
A Hungry Neighbor is an Angry Neighbor
I am very grateful that I have had the opportunity to meet with President Shimon Peres several times. Especially the first time, which was a 1:1 in his presidential residence, was an unforgettable experience. After I explained in 5 minutes the power of cloud for unlocking digital business building for everyone, he went on a lecture of half an hour how bringing economic prosperity to the region was crucial to achieving a long lasting peace.
New Ways to Discover and Use Alexa Skills
Introducing New Features That Make It Easier for Customers to Discover and Use Your Alexa Skills Alexa, Amazons cloud-based voice service, powers voice experiences on millions of devices, including Amazon Echo and Echo Dot, Amazon Tap, Amazon Fire TV devices, and devices like Triby that use the Alexa Voice Service. One year ago, Amazon opened up Alexa to developers, enabling you to build Alexa skills with the Alexa Skills Kit and integrate Alexa into your own products with the Alexa Voice Service. Today, tens of thousands of developers are building skills for Alexa, and there are over 1,400 skills for Alexa including Lyft and Honeywell, which were added today.
New Ways to Discover and Use Alexa Skills
Introducing New Features That Make It Easier for Customers to Discover and Use Your Alexa Skills Alexa, Amazon?s cloud-based voice service, powers voice experiences on millions of devices, including Amazon Echo and Echo Dot, Amazon Tap, Amazon Fire TV devices, and devices like Triby that use the Alexa Voice Service. One year ago, Amazon opened up Alexa to developers, enabling you to build Alexa skills with the Alexa Skills Kit and integrate Alexa into your own products with the Alexa Voice Service.
Expanding the Cloud: Introducing the AWS Asia Pacific (Mumbai) Region
In June 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in India. Today, Im happy to announce that the Asia Pacific (Mumbai) Region is generally available for use by customers worldwide. The opportunity to revolutionize A region in India has been highly sought after by companies around the world who want to participate in one of the most significant economic opportunities in the world India, a rising economy that holds tremendous promise for growth, a thriving technology hub with a rich eco-system of technology talent, and more. Rapid economic growth in India is creating several business opportunities such as distributed ledger technology with blockchains that could drive efficiencies in the real estate market, Fin-Tech innovations such as P2P mobile apps that have the power to change the social economic lives of people through financial inclusion, applying the sharing economy from cabs to other ...
Expanding the Cloud: Introducing the AWS Asia Pacific (Mumbai) Region
In June 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in India. Today, I?m happy to announce that the Asia Pacific (Mumbai) Region is generally available for use by customers worldwide. The opportunity to revolutionize A region in India has been highly sought after by companies around the world who want to participate in one of the most significant economic opportunities in the world ? India, a rising economy that holds tremendous promise for growth, a thriving technology hub with a rich eco-system of technology talent, and more.
Serverless Reference Architectures with AWS Lambda
Building your applications with only managed components has become very popular, and AWS Lambda plays a crucial role in that. I see a tremendous interest in examples how to build such applications, and articles such as "The Serverless Start-Up - Down With Servers!" about teletext.io are read eagerly around the globe. If you are looking for more examples there are the Lambda Serverless Reference Architectures that can serve as the blueprint for building your own serverless applications. Mobile Backend Serverless Reference Architecture The Mobile Backend reference architecture demonstrates how to use AWS Lambda along with other services to build a serverless backend for a mobile application.
Serverless Reference Architecture with AWS Lambda
Building your applications with only managed components has become very popular, and AWS Lambda plays a crucial role in that. I see a tremendous interest in examples how to build such applications, and articles such as "The Serverless Start-Up - Down With Servers!" about teletext.io are read eagerly around the globe. If you are looking for more examples there are the Lambda Serverless Reference Architectures that can serve as the blueprint for building your own serverless applications. Mobile Backend Serverless Reference Architecture The Mobile Backend reference architecture demonstrates how to use AWS Lambda along with other services to build a serverless backend for a mobile application.
Serverless Reference Architectures with AWS Lambda
Building your applications with only managed components has become very popular, and AWS Lambda plays a crucial role in that. I see a tremendous interest in examples how to build such applications, and articles such as “The Serverless Start-Up - Down With Servers!” about teletext.io are read eagerly around the globe.
10 Lessons from 10 Years of Amazon Web Services
The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Looking back over the past 10 years, there are hundreds of lessons that weve learned about building and operating services that need to be secure, reliable, scalable, with predictable performance at the lowest possible cost. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. As weve said many times before, There is no compression algorithm for experience. With over a million active customers per month, who in turn may serve hundreds of millions of their own customers, there is no lack of opportunities to gain more experience and perhaps no better environment for continuous improvement in the way we serve our customers.
10 Lessons from 10 Years of Amazon Web Services
The epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Looking back over the past 10 years, there are hundreds of lessons that we?ve learned about building and operating services that need to be secure, reliable, scalable, with predictable performance at the lowest possible cost.
Expanding the Cloud: Introducing the AWS Asia Pacific (Seoul) Region
In November, Amazon Web Services announced that it would launch a new AWS infrastructure region in South Korea. Today, Im happy to announce that the Asia Pacific (Seoul) Region is now generally available for use by customers worldwide. A region in South Korea has been highly requested by companies around the world who want to take full advantage of Koreas world-leading Internet connectivity and provide their customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, and more. Weve also been hearing many requests from Korean companies, including large enterprises like Samsung and Mirae Asset. For example, Samsung Electronic Printing used AWS to deploy its Printing Apps Center in a way that didnt require them to invest up-front capital and kept total costs quite low.
Expanding the Cloud: Introducing the AWS Asia Pacific (Seoul) Region
In November, Amazon Web Services announced that it would launch a new AWS infrastructure region in South Korea. Today, I?m happy to announce that the Asia Pacific (Seoul) Region is now generally available for use by customers worldwide. A region in South Korea has been highly requested by companies around the world who want to take full advantage of Korea?s world-leading Internet connectivity and provide their customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, and more.
London Calling! An AWS Region is coming to the UK!
Yesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific. Customers can choose between 11 regions around the world today and, in addition to Korea, we are adding regions in India, a second region in China, and Ohio in 2016. Today, I am excited to add the United Kingdom to that list! The AWS UK region will be our third in the European Union (EU), and we're shooting to have it ready by the end of 2016 (or early 2017). This region will provide even lower latency and strong data sovereignty to local users.
London Calling! An AWS Region is coming to the UK!
Yesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific. Customers can choose between 11 regions around the world today and, in addition to Korea, we are adding regions in India, a second region in China, and Ohio in 2016.
Expanding the Cloud: Introducing Amazon QuickSight
We live in a world where massive volumes of data are generated from websites, connected devices and mobile apps. In such a data intensive environment, making key business decisions such as running marketing and sales campaigns, logistic planning, financial analysis and ad targeting require deriving insights from these data. However, the data infrastructure to collect, store and process data is geared toward developers (e.g., Amazon Redshift, DynamoDB, Amazon EMR) whereas insights need to be derived by not just developers but also non-technical business users. In AWS quest to enable the best data storage options for engineers, we have built several innovative database solutions like Amazon RDS, Amazon RDS for Aurora, Amazon DynamoDB, and Amazon Redshift.
Expanding the Cloud: Introducing Amazon QuickSight
We live in a world where massive volumes of data are being generated from websites, connected devices and mobile apps. In such a data intensive environment, making key business decisions such as running marketing and sales campaigns, logistic planning, financial analysis, and ad targeting require deriving insights from these data. However, the data infrastructure to collect, store, and process data is geared primarily towards developers and IT professionals (e.
The Startup Experience at AWS re:Invent
AWS re:Invent is just over one week awayas I prepare to head to Vegas, Im pumped up about the chance to interact with AWS-powered startups from around the world. One of my favorite parts of the week is being able to host three startup-focused sessions Thursday afternoon: The Startup Scene in 2016: a Visionary Panel [Thursday, 2:45PM] In this session, Ill moderate a diverse panel of technology experts wholl discuss emerging trends all startups should be aware of, including how local governments, microeconomic trends, evolving accelerator programs, and the AWS cloud are influencing the global startup scene. This panel will include: Tracy DiNunzio, Founder & CEO, Tradesy Michael DeAngelo, Deputy CIO, State of Washington Ben Whaley, Founder & Principal Consultant, WhaleTech LLC Jason Seats, Managing Director (Austin), & Partner, Techstars CTO-to-CTO Fireside Chat [Thursday, 4:15 PM] This is one of my favorite sessions as I get a chance ...
The Startup Experience at AWS re:Invent
AWS re:Invent is just over one week away?as I prepare to head to Vegas, I?m pumped up about the chance to interact with AWS-powered startups from around the world. One of my favorite parts of the week is being able to host three startup-focused sessions Thursday afternoon: The Startup Scene in 2016: a Visionary Panel [Thursday, 2:45PM]
The AWS Pop-up Lofts are opening in London and Berlin
Amazon Web Services (AWS) has been working closely with the startup community in London, and Europe, since we launched back in 2006. We have grown substantially in that time and today more than two thirds of the UKs startups with valuations of over a billion dollars, including Skyscanner, JustEat, Powa, Fanduel and Shazam, are all leveraging our platform to deliver innovative services to customers around the world. This week I will have the pleasure of meeting up with our startup customers to we celebrate the opening of the first of the AWS Pop-up Lofts to open outside of the US in one of the greatest cities in the World, London.
The AWS Pop-up Lofts are opening in London and Berlin
Amazon Web Services (AWS) has been working closely with the startup community in London, and Europe, since we launched back in 2006. We have grown substantially in that time and today more than two thirds of the UK?s startups with valuations of over a billion dollars, including Skyscanner, JustEat, Powa, Fanduel and Shazam, are all leveraging our platform to deliver innovative services to customers around the world.
Titan Graph Database Integration with DynamoDB: World-class Performance, Availability, and Scale for New Workloads
Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. The importance of relationships Relationships are a fundamental aspect of both the physical and virtual worlds. Modern applications need to quickly navigate connections in the physical world of people, cities, and public transit stations as well as the virtual world of search terms, social posts, and genetic code, for example. Developers need efficient methods to store, traverse, and query these relationships.
Titan Graph Database Integration with DynamoDB: World-class Performance, Availability, and Scale for New Workloads
Today, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. The importance of relationships
Under the Hood of Amazon EC2 Container Service
In my last post about Amazon EC2 Container Service (Amazon ECS), I discussed the two key components of running modern distributed applications on a cluster: reliable state management and flexible scheduling. Amazon ECS makes building and running containerized applications simple, but how that happens is what makes Amazon ECS interesting. Today, I want to explore the Amazon ECS architecture and what this architecture enables. Below is a diagram of the basic components of Amazon ECS: How we coordinate the cluster Lets talk about what Amazon ECS is actually doing. The core of Amazon ECS is the cluster manager, a backend service that handles the tasks of cluster coordination and state management.
Under the Hood of Amazon EC2 Container Service
In my last post about Amazon EC2 Container Service (Amazon ECS), I discussed the two key components of running modern distributed applications on a cluster: reliable state management and flexible scheduling. Amazon ECS makes building and running containerized applications simple, but how that happens is what makes Amazon ECS interesting. Today, I want to explore the Amazon ECS architecture and what this architecture enables.
Back-to-Basics Weekend Reading - Data Compression
Data compression today is still as important as it was in the early days of computing. Although in those days all computer and storage resources were very limited, the objects in use were much smaller than today. We have seen a shift from generic compression to compression for specific file types, especially those in images, audio and video. In this weekend's back to basic reading we go back in time, 1987 to be specific, when Leweler and Hirschberg wrote a survey paper that covers the 40 years of data compression research. It covers all the areas that we like in a back to basics paper, it does not present the most modern results but it gives you a great understanding of the fundamentals.
Back-to-Basics Weekend Reading - Data Compression
Data compression today is still as important as it was in the early days of computing. Although in those days all computer and storage resources were very limited, the objects in use were much smaller than today. We have seen a shift from generic compression to compression for specific file types, especially those in images, audio and video.
Embrace event-driven computing: Amazon expands DynamoDB with streams, cross-region replication, and database triggers
In just three short years, Amazon DynamoDB has emerged as the backbone for many powerful Internet applications such as AdRoll, Druva, DeviceScape, and Battlecamp. Many happy developers are using DynamoDB to handle trillions of requests every day. I am excited to share with you that today we are expanding DynamoDB with streams, cross-region replication, and database triggers. In this blog post, I will explain how these three new capabilities empower you to build applications with distributed systems architecture and create responsive, reliable, and high-performance applications using DynamoDB that work at any scale. DynamoDB Streams enables your application to get real-time notifications of your tables item-level changes.
Embrace event-driven computing: Amazon expands DynamoDB with streams, cross-region replication, and database triggers
In just three short years, Amazon DynamoDB has emerged as the backbone for many powerful Internet applications such as AdRoll, Druva, DeviceScape, and Battlecamp. Many happy developers are using DynamoDB to handle trillions of requests every day. I am excited to share with you that today we are expanding DynamoDB with streams, cross-region replication, and database triggers.
Amazon announces the Alexa Skills Kit, Enabling Developers to Create New Voice Capabilities
Today, Amazon announced the Alexa Skills Kit (ASK), a collection of self-service APIs and tools that make it fast and easy for developers to create new voice-driven capabilities for Alexa. With a few lines of code, developers can easily integrate existing web services with Alexa or, in just a few hours, they can build entirely new experiences designed around voice. No experience with speech recognition or natural language understanding is requiredAmazon does all the work to hear, understand, and process the customers spoken request so you dont have to. All of the code runs in the cloud nothing is installed on any user device.
Amazon announces the Alexa Skills Kit, Enabling Developers to Create New Voice Capabilities
Today, Amazon announced the Alexa Skills Kit (ASK), a collection of self-service APIs and tools that make it fast and easy for developers to create new voice-driven capabilities for Alexa. With a few lines of code, developers can easily integrate existing web services with Alexa or, in just a few hours, they can build entirely new experiences designed around voice.
Back-to-Basics Weekend Reading - The Working Set Model for Program Behavior
This weekend we go back in time all the way to the beginning of operating systems research. In the first SOSP conference in 1967 there were several papers that laid the foundation for the development of structured operating systems. There was the of course the lauded paper on the THE operating system by Dijkstra but for this weekend I picked the paper on memory locality by Peter Denning as this work laid the groundwork for the development of virtual memory systems. The Working Set Model for Program Behavior, Peter J.
Back-to-Basics Weekend Reading - The Working Set Model for Program Behavior
This weekend we go back in time all the way to the beginning of operating systems research. In the first SOSP conference in 1967 there were several papers that laid the foundation for the development of structured operating systems. There was the of course the lauded paper on the THE operating system by Dijkstra but for this weekend I picked the paper on memory locality by Peter Denning as this work laid the groundwork for the development of virtual memory systems.
Back-to-Basics Weekend Reading - Survey of Local Algorithms
As we know the run time of most algorithms increases when the input set increases in size. There is one noticeable exception: there is a class of distributed algorithms, dubbed local algorithms, that run in constant time, independently of the size of the network. Being highly scalable and fault tolerant, such algorithms are ideal in the operation of large-scale distributed systems.
Back-to-Basics Weekend Reading - Survey of Local Algorithms
As we know the run time of most algorithms increases when the input set increases in size. There is one noticeable exception: there is a class of distributed algorithms, dubbed local algorithms, that run in constant time, independently of the size of the network. Being highly scalable and fault tolerant, such algorithms are ideal in the operation of large-scale distributed systems. Furthermore, even though the model of local algorithms is very limited, in recent years we have seen many positive results for nontrivial problems. In this weekend's paper Jukka Suomela surveys the state-of-the-art in the field, covering impossibility results, deterministic local algorithms, randomized local algorithms, and local algorithms for geometric graphs.
Join me at the AWS Summit in Paris, Tel Aviv, Berlin, Amsterdam or New York
An important way of engaging with AWS customers is through the AWS Global Summit Series. All AWS Summits feature a keynote address highlighting the latest announcements from AWS and customer testimonials, technical sessions led by AWS engineers, and hands-on technical training. You will learn best practices for deploying applications on AWS, optimizing performance, monitoring cloud resources, managing security, cutting costs, and more.
Join me at the AWS Summit in Paris, Tel Aviv, Berlin, Amsterdam or New York
An important way of engaging with AWS customers is through the AWS Global Summit Series. All AWS Summits feature a keynote address highlighting the latest announcements from AWS and customer testimonials, technical sessions led by AWS engineers, and hands-on technical training. You will learn best practices for deploying applications on AWS, optimizing performance, monitoring cloud resources, managing security, cutting costs, and more. You will also have opportunities to meet AWS staff and partners to get your technical questions answered. At the Summit we focus on education and helping our customers, there are deep technical developer sessions, broad sessions on architectural principles, sessions for enterprise decision makers and how to best exploit AWS in a public sector or education settings.
The AWS Pop-up Loft opens in New York City
Over a year ago the AWS team opened a “pop-up loft” in San Francisco at 925 Market Street. The goal of opening the loft was to give developers an opportunity to get in-person support and education on AWS, to network, get some work done, or just hang out with peers. It became a great success; every time when I visit the loft there is a great buzz with people getting advice from our solution architects, getting training or attending talks and demos.
The AWS Pop-up Loft opens in New York City
Over a year ago the AWS team opened a "pop-up loft" in San Francisco at 925 Market Street. The goal of opening the loft was to give developers an opportunity to get in-person support and education on AWS, to network, get some work done, or just hang out with peers. It became a great success; every time when I visit the loft there is a great buzz with people getting advice from our solution architects, getting training or attending talks and demos. It became such a hit among developers that we decided to reopen the loft last year August after its initial run of 4 weeks, making sure everyone would have continued access to this important resource.
Expanding the Cloud: Amazon Machine Learning Service, the Amazon Elastic Filesystem and more
Today was a big day for the Amazon Web Services teams as a whole range of new services and functionality was delivered to our customers. Here is a brief recap of it: The Amazon Machine Learning service As I wrote last week machine learning is becoming an increasingly important tool to build advanced data driven applications.
Expanding the Cloud: Amazon Machine Learning Service, the Amazon Elastic Filesystem and more
Today was a big day for the Amazon Web Services teams as a whole range of new services and functionality was delivered to our customers. Here is a brief recap of it: The Amazon Machine Learning service As I wrote last week machine learning is becoming an increasingly important tool to build advanced data driven applications. At Amazon we have hundreds of teams using machine learning and by making use of the Machine Learning Service we can significantly speed up the time they use to bring their technologies into production. And you no longer need to be a machine learning expert to be able to use it.
State Management and Scheduling with the Amazon EC2 Container Service
Last November, I had the pleasure of announcing the preview of Amazon EC2 Container Service (ECS) at re:Invent. At the time, I wrote about how containerization makes it easier for customers to decompose their applications into smaller building blocks resulting in increased agility and speed of feature releases. I also talked about some of the challenges our customers were facing as they tried to scale container-based applications including challenges around cluster management.
State Management and Scheduling with the Amazon EC2 Container Service
Last November, I had the pleasure of announcing the preview of Amazon EC2 Container Service (ECS) at re:Invent. At the time, I wrote about how containerization makes it easier for customers to decompose their applications into smaller building blocks resulting in increased agility and speed of feature releases. I also talked about some of the challenges our customers were facing as they tried to scale container-based applications including challenges around cluster management. Today, I want to dive deeper into some key design decisions we made while building Amazon ECS to address the core problems our customers are facing. Running modern distributed applications on a cluster requires two key components - reliable state management and flexible scheduling.
Back-to-Basics Weekend Reading - Machine Learning
Machine learning is a scientific discipline that explores the construction and study of algorithms that can learn from data. Such algorithms operate by building a model from example inputs and using that to make predictions or decisions, rather than following strictly static program instructions. Machine Learning is playing an increasing important role in many areas of our businesses and our lives.
Back-to-Basics Weekend Reading - Machine Learning
Machine learning is a scientific discipline that explores the construction and study of algorithms that can learn from data. Such algorithms operate by building a model from example inputs and using that to make predictions or decisions, rather than following strictly static program instructions. Machine Learning is playing an increasing important role in many areas of our businesses and our lives. ML is used for predictive analytics and predictive modeling, e.g. making predictions about the likelihood that a certain event is going to happen (will this customer be interested in this item, is this message spam). At Amazon machine learning has been key to many of our business processes, from recommendations to fraud detection, from inventory levels to book classification to abusive review detection.
European Union Data Protection Authorities Approve Amazon Web Services Data Processing Agreement
As you all know security, privacy, and protection of our customers data is our number one priority and as such we work very closely with regulators to ensure that customers can be assured that they are getting the right protections when processing and storing data in the AWS. I am especially pleased that the group of European Union (EU) data protection authorities known as the Article 29 Working Party has approved the AWS Data Processing Agreement (DPA), assuring customers that it meets the high standards of EU data protection laws. The media alert below that went out today gives the details: European Union Data Protection Authorities Approve Amazon Web Services Data Processing Agreement Customers All Over the World Are Assured that AWS Agreement Meets Rigorous EU Privacy Laws Brussels March 31, 2015 Amazon Web Services (AWS) today announced that the group of European Union (EU) data protection authorities ...
European Union Data Protection Authorities Approve Amazon Web Services? Data Processing Agreement
As you all know security, privacy, and protection of our customer?s data is our number one priority and as such we work very closely with regulators to ensure that customers can be assured that they are getting the right protections when processing and storing data in the AWS. I am especially pleased that the group of European Union (EU) data protection authorities known as the Article 29 Working Party has approved the AWS Data Processing Agreement (DPA), assuring customers that it meets the high standards of EU data protection laws.
European Union Data Protection Authorities Approve Amazon Web Services Data Processing Agreement
As you all know security, privacy, and protection of our customers data is our number one priority and as such we work very closely with regulators to ensure that customers can be assured that they are getting the right protections when processing and storing data in the AWS. I am especially pleased that the group of European Union (EU) data protection authorities known as the Article 29 Working Party has approved the AWS Data Processing Agreement (DPA), assuring customers that it meets the high standards of EU data protection laws. The media alert below that went out today gives the details: European Union Data Protection Authorities Approve Amazon Web Services Data Processing Agreement Customers All Over the World Are Assured that AWS Agreement Meets Rigorous EU Privacy Laws Brussels March 31, 2015 Amazon Web Services (AWS) today announced that the group of European Union (EU) data protection authorities ...
Observations on the Importance of Cloud-based Analytics
Cloud computing is enabling amazing new innovations both in consumer and enterprise products, as it became the new normal for organizations of all sizes. So many exciting new areas are being empowered by cloud that it is fascinating to watch. AWS is enabling innovations in areas such as healthcare, automotive, life sciences, retail, media, energy, robotics that it is mind boggling and humbling.
Observations on the Importance of Cloud-based Analytics
Cloud computing is enabling amazing new innovations both in consumer and enterprise products, as it became the new normal for organizations of all sizes. So many exciting new areas are being empowered by cloud that it is fascinating to watch. AWS is enabling innovations in areas such as healthcare, automotive, life sciences, retail, media, energy, robotics that it is mind boggling and humbling. Despite all of the amazing innovations we have already seen, we are still on Day One in the Cloud; at AWS we will continue to use our inventive powers to build new tools and services to enable even more exciting innovations by our customers that will touch every area of our lives.
Join me at SXSW 2015
Every year I enjoy travelling to the South-by-South-West (SXSW) festival as it is ons of the biggest event with many Amazon customers present. Thousand of AWS customers and partners will be in Austin for SXSW Interactive and given the free flowing networking it is a very important feedback opportunity for us.
Join me at SXSW 2015
Every year I enjoy travelling to the South-by-South-West (SXSW) festival as it is ons of the biggest event with many Amazon customers present. Thousand of AWS customers and partners will be in Austin for SXSW Interactive and given the free flowing networking it is a very important feedback opportunity for us. But also many Amazon customers will be there for the Film and the Music festival, and I always enjoy getting feedback from those Amazon consumers and producers that are attending these festivals. The program is always a bit in flux, but here are the events in the beginning of the week that I am taking part in: Sunday 3/15 1-2pm - I will give a talk at Techstars on "The History of Microcroservices at Amazon".
Back-to-Basics Weekend Reading - Experience with Grapevine: The Growth of a Distributed System
Grapevine was one of the first systems designed to be fully distributed. It was built at the famous Xerox PARC (Palo Alto Research Center) Computer Science Laboratory as an exercise in discovering what is needed as the fundamental building blocks of a distributed system; messaging, naming, discovery, location, routing, authentication, encryption, replication, etc.
Back-to-Basics Weekend Reading - Experience with Grapevine: The Growth of a Distributed System
Grapevine was one of the first systems designed to be fully distributed. It was built at the famous Xerox PARC (Palo Alto Research Center) Computer Science Laboratory as an exercise in discovering what is needed as the fundamental building blocks of a distributed system; messaging, naming, discovery, location, routing, authentication, encryption, replication, etc. The origins of the system are described in Grapevine: An Exercise in Distributed Computing by researchers who all went on to become grandmasters in distributed computing: Andrew Birrell, Roy Levin, Roger Needham, and Mike Schroeder. For this weekend's reading we will use a followup paper that focusses on the learnings with running Grapevine for several years under substantial load.
Back-to-Basics Weekend Reading - Distributed Snapshots: Determining Global States of a Distributed System
Several problems in Distributed Systems can be seen as the challenge to determine a global state. In the classical “Time, Clocks and the Ordering of Events in a Distributed System” Lamport had laid out the principles and mechanisms to solve such problems, and the Distributed Snapshots algorithm, popularly know as the Chandy-Lamport algorithm, is an application of that work.
Back-to-Basics Weekend Reading - Distributed Snapshots: Determining Global States of a Distributed System
Several problems in Distributed Systems can be seen as the challenge to determine a global state. In the classical "Time, Clocks and the Ordering of Events in a Distributed System" Lamport had laid out the principles and mechanisms to solve such problems, and the Distributed Snapshots algorithm, popularly know as the Chandy-Lamport algorithm, is an application of that work. The fundamental techniques in the Distributed Snapshot paper are the secret sauce in many distributed algorithms for deadlock detection, termination detection, consistent checkpointing for fault tolerance, global predicate detection for debugging and monitoring, and distributed simulation. An interesting anecdote about the algorithm is told by Lamport: "The distributed snapshot algorithm described here came about when I visited Chandy, who was then at the University of Texas in Austin.
Back-to-Basics Weekend Reading - RAID: High-Performance, Reliable Secondary Storage
Disk arrays, which organize multiple, independent disks into a large, high-performance logical disk, were a natural solution to dealing with constraints on performance and reliability of single disk drives. The term “RAID” was invented by David Patterson, Garth A. Gibson, and Randy Katz at the University of California, Berkeley in 1987.
Back-to-Basics Weekend Reading - RAID: High-Performance, Reliable Secondary Storage
Disk arrays, which organize multiple, independent disks into a large, high-performance logical disk, were a natural solution to dealing with constraints on performance and reliability of single disk drives. The term "RAID" was invented by David Patterson, Garth A. Gibson, and Randy Katz at the University of California, Berkeley in 1987. In their June 1988 paper "A Case for Redundant Arrays of Inexpensive Disks (RAID)" they argued that the top performing mainframe disk drives of the time could be beaten on performance by an array of the inexpensive drives that had been developed for the growing personal computer market. Although failures would rise in proportion to the number of drives, by configuring for redundancy, the reliability of an array could far exceed that of any large single drive.
AWS Certification for DevOps Engineers
One of our guiding principles at AWS is to listen closely to our customers and the feedback that I am getting about our training and certification program is very positive. Many architects and engineers know the Cloud is the future of development and IT and the are gearing up to be as succesful as possible in this new normal.
AWS Certification for DevOps Engineers
One of our guiding principles at AWS is to listen closely to our customers and the feedback that I am getting about our training and certification program is very positive. Many architects and engineers know the Cloud is the future of development and IT and the are gearing up to be as succesful as possible in this new normal. This is why Im excited to announce the availability of a new Professional level certification from AWS that has been high on the list of our customers. With the growing adoption of cloud computing, we see more of our customers establishing DevOps practices within their IT organizations as a way to increase IT efficiency, improve agility, and in turn innovate faster for their own customers.
Back-to-Basics Weekend Reading - Exploring Complex Networks
After a year of absence I am bringing back the Back to Basic Weekend Reading Series. We'll continue to look at the fundamental works of Computer Science and Engineering, and other interesting technical works. We will start this year with a topic that spans many sciences: that of complex networks. It is relevant to everything from biology, life sciences, social sciences to computer engineering. There is no one better suitable to teach us about the fundamentals of complex networks than Steven Strogatz, the well known author and applied mathematics professor from Cornell University. You can find much of Steven's work on his personal website.
Back-to-Basics Weekend Reading - Exploring Complex Networks
After a year of absence I am bringing back the Back to Basic Weekend Reading Series. We’ll continue to look at the fundamental works of Computer Science and Engineering, and other interesting technical works. We will start this year with a topic that spans many sciences: that of complex networks. It is relevant to everything from biology, life sciences, social sciences to computer engineering.
The Easiest Way to Compute in the Cloud ? AWS Lambda
When AWS launched, it changed how developers thought about IT services: What used to take weeks or months of purchasing and provisioning turned into minutes with Amazon EC2. Capital-intensive storage solutions became as simple as PUTting and GETting objects in Amazon S3. At AWS we innovate by listening to and learning from our customers, and one of the things we hear from them is that they want it to be even simpler to run code in the cloud and to connect services together easily.
The Easiest Way to Compute in the Cloud AWS Lambda
When AWS launched, it changed how developers thought about IT services: What used to take weeks or months of purchasing and provisioning turned into minutes with Amazon EC2. Capital-intensive storage solutions became as simple as PUTting and GETting objects in Amazon S3. At AWS we innovate by listening to and learning from our customers, and one of the things we hear from them is that they want it to be even simpler to run code in the cloud and to connect services together easily. Customers want to focus on their unique application logic and business needs not on the undifferentiated heavy lifting of provisioning and scaling servers, keeping software stacks patched and up to date, handling fleet-wide deployments, or dealing with routine monitoring, logging, and web service front ends.
Expanding The Cloud - Introducing The Amazon EC2 Container Service
Today, I am excited to announce the Preview of the Amazon EC2 Container Service, a highly scalable, high performance container management service. We created EC2 Container Service to help customers run and manage Dockerized distributed applications. Benefits of Containers Customers have been using Linux containers for quite some time on AWS and have increasingly adopted microservice architectures.
Expanding The Cloud - Introducing The Amazon EC2 Container Service
Today, I am excited to announce the Preview of the Amazon EC2 Container Service, a highly scalable, high performance container management service. We created EC2 Container Service to help customers run and manage Dockerized distributed applications. Benefits of Containers Customers have been using Linux containers for quite some time on AWS and have increasingly adopted microservice architectures. The microservices approach to developing a single application is to divide the application into a set of small services, each running its own processes, which communicate with each other. Each small service can be scaled independently of the application and can be managed by different teams.
The Story of Apollo - Amazon?s Deployment Engine
Automated deployments are the backbone of a strong DevOps environment. Without efficient, reliable, and repeatable software updates, engineers need to redirect their focus from developing new features to managing and debugging their deployments. Amazon first faced this challenge many years ago. When making the move to a service-oriented architecture, Amazon refactored its software into small independent services and restructured its organization into small autonomous teams.
The Story of Apollo - Amazons Deployment Engine
Automated deployments are the backbone of a strong DevOps environment. Without efficient, reliable, and repeatable software updates, engineers need to redirect their focus from developing new features to managing and debugging their deployments. Amazon first faced this challenge many years ago. When making the move to a service-oriented architecture, Amazon refactored its software into small independent services and restructured its organization into small autonomous teams. Each team took on full ownership of the development and operation of a single service, and they worked directly with their customers to improve it. With this clear focus and control, the teams were able to quickly produce new features, but their deployment process soon became a bottleneck.
Don't Miss These Startup Activities at AWS re:Invent!
I?m excited to be heading to Las Vegas in less than two weeks for our annual re:Invent conference. One of the highlights for me is being able to host an extensive lineup of startup-focused events which take place at re:Invent on Thursday, November 13. Here?s a quick peak at the startup experience this year:
Don't Miss These Startup Activities at AWS re:Invent!
Im excited to be heading to Las Vegas in less than two weeks for our annual re:Invent conference. One of the highlights for me is being able to host an extensive lineup of startup-focused events which take place at re:Invent on Thursday, November 13. Heres a quick peak at the startup experience this year: Third Annual Startup Launches Im excited to host this event where five AWS-powered startups will make a significant, never-before-shared launch announcement on stage. Included in the announcements are special discounts on the newly-launched productsdiscounts only available to session attendees. And to top it all off, well have a happy hour immediately following the final launch announcement!
Expanding the Cloud ? Introducing the AWS EU (Frankfurt) Region
Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS region in Frankfurt, Germany. This is our 11th infrastructure region and was built to support the strong demand we are seeing in Europe and to give our customers the option to run infrastructure located in Germany.
Expanding the Cloud Introducing the AWS EU (Frankfurt) Region
Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS region in Frankfurt, Germany. This is our 11th infrastructure region and was built to support the strong demand we are seeing in Europe and to give our customers the option to run infrastructure located in Germany. The new Frankfurt region provides low millisecond latencies to major cities in continental Europe and is also run with carbon neutral power. With the launch of the new Frankfurt region customers now also have the ability to architect across multiple regions within the European Union. Many prominent German, and European, customers have been using AWS for quite some time already, including start-ups such as 6Wunderkinder, EyeEm, mytaxi, Onefootball, Soundcloud and Wooga, mid-market companies such as Airport Nuremburg, Euroforum, and Kärcher, and Enterprise companies such as Axel Springer, Hubert Burda Media, Kempinski Hotels, RTL, SAP, Software AG, and Talanx.
Document Model Support in DynamoDB: Flexibility, Availability, Performance, and Scale...Together at last
Today, I?m thrilled to announce several major features that significantly enhance the development experience on DynamoDB. We are introducing native support for document model like JSON into DynamoDB, the ability to add / remove global secondary indexes, adding more flexible scaling options, and increasing the item size limit to 400KB. These improvements have been sought by many applications developers, and we are happy to be bringing them to you.
Document Model Support in DynamoDB: Flexibility, Availability, Performance, and Scale...Together at last
Today, Im thrilled to announce several major features that significantly enhance the development experience on DynamoDB. We are introducing native support for document model like JSON into DynamoDB, the ability to add / remove global secondary indexes, adding more flexible scaling options, and increasing the item size limit to 400KB. These improvements have been sought by many applications developers, and we are happy to be bringing them to you. The best part is that we are also significantly expanding the free tier many of you already enjoy by increasing the storage to 25 GB and throughput to 200 million requests per month.
AWS Pop-up Loft 2.0: Returning to San Francisco on October 1st
It?s an exciting time in San Francisco as the return of the AWS Loft is fast approaching. We?ve been working round-the-clock, making updates to ensure the experience is more fulfilling and educational than in June. Today we?re excited to announce that? On Wednesday, October 1st, we?ll be returning to 925 Market Street!
AWS Pop-up Loft 2.0: Returning to San Francisco on October 1st
Its an exciting time in San Francisco as the return of the AWS Loft is fast approaching. Weve been working round-the-clock, making updates to ensure the experience is more fulfilling and educational than in June. Today were excited to announce that& On Wednesday, October 1st, well be returning to 925 Market Street! The AWS Loft is all about helping you scale and grow your business by offering free AWS technical resources. Youll have access to training including hands-on bootcamps and labs, and 1:1 sessions with AWS Solutions Architects.
Cloud computing in Europe should put power in the hands of the customer
This is an extended version of an article that appeared in the Guardian today We are rapidly entering into an era where massive computing power, digital storage and global network connections can be deployed by anyone as quickly and easily as turning on the lights. This is the promise ? and the reality ? of cloud computing which is driving tremendous change in the technology industry and transforming how we do business in Europe and around the world.
Cloud computing in Europe should put power in the hands of the customer
This is an extended version of an article that appeared in the Guardian today We are rapidly entering into an era where massive computing power, digital storage and global network connections can be deployed by anyone as quickly and easily as turning on the lights. This is the promise and the reality of cloud computing which is driving tremendous change in the technology industry and transforming how we do business in Europe and around the world. Cloud computing unlocks innovation within organisations of all types and sizes. No longer do they need to spend valuable human and capital resources on maintaining and procuring expensive technology infrastructure and datacenters, they can focus their most valuable resources on what they do best, building better products and services for their customers.
The AWS Activate CTO to CTO series on Medium
I’m excited to announce a new blog dedicated to AWS startups. We’re launching it on Medium, itself a startup on AWS. I kicked off the blog with a Q&A with the Medium CTO Don Neufeld. I really enjoyed Don’s answers to my questions and there are some real gems in here for startup CTOs.
The AWS Activate CTO to CTO series on Medium
I'm excited to announce a new blog dedicated to AWS startups. We're launching it on Medium, itself a startup on AWS. I kicked off the blog with a Q&A with the Medium CTO Don Neufeld. I really enjoyed Don's answers to my questions and there are some real gems in here for startup CTOs. Check it out. We'll be keeping this blog fresh with other startup spotlights and good technical content so follow the collection and keep up.
Expanding the Cloud: Docker Containers in Elastic Beanstalk
We launched Elastic Beanstalk in 2011 with support for Java web applications and Tomcat 6 in one region, and we’ve seen the service grow to 6 container types (Java/Tomcat, PHP, Ruby, Python, .NET, and Node.js) supported in 8 AWS regions around the world. The Elastic Beanstalk team spends a lot of time talking to AWS Developers, and in the last few months they’ve noticed a common theme in those conversations: developers tell us they’re interested in Docker, and ask if we are thinking about making it easy to run and scale Docker workloads in AWS.
Expanding the Cloud: Docker Containers in Elastic Beanstalk
We launched Elastic Beanstalk in 2011 with support for Java web applications and Tomcat 6 in one region, and we've seen the service grow to 6 container types (Java/Tomcat, PHP, Ruby, Python, .NET, and Node.js) supported in 8 AWS regions around the world. The Elastic Beanstalk team spends a lot of time talking to AWS Developers, and in the last few months they've noticed a common theme in those conversations: developers tell us they're interested in Docker, and ask if we are thinking about making it easy to run and scale Docker workloads in AWS. Several weeks ago we made it simple to yum install Docker on your EC2 Instances running Amazon Linux, and today Elastic Beanstalk introduces the ability to deploy, manage, and scale Docker Containers.
Customer Centricity at Amazon Web Services
In the 2013 Amazon Shareholder letter, Jeff Bezos spent time explaining the decision to pursue a customer-centric way in our business. As regular readers of this letter will know, our energy at Amazon comes from the desire to impress customers rather than the zeal to best competitors. We don?t take a view on which of these approaches is more likely to maximize business success.
Customer Centricity at Amazon Web Services
In the 2013 Amazon Shareholder letter, Jeff Bezos spent time explaining the decision to pursue a customer-centric way in our business. As regular readers of this letter will know, our energy at Amazon comes from the desire to impress customers rather than the zeal to best competitors. We dont take a view on which of these approaches is more likely to maximize business success. There are pros and cons to both and many examples of highly successful competitor-focused companies. We do work to pay attention to competitors and be inspired by them, but it is a fact that the customer-centric way is at this point a defining element of our culture.
Updated Lampson's Hints for Computer Systems Design
This year I have not been able to publish many back-to-basics readings, so I will not close the year with a recap of those. Instead I have a video of a wonderful presentation by Butler Lampson where he talks about the learnings of the past decades that helped him to update his excellent 1983 “Hints for computer system design”.
Updated Lampson's Hints for Computer Systems Design
This year I have not been able to publish many back-to-basics readings, so I will not close the year with a recap of those. Instead I have a video of a wonderful presentation by Butler Lampson where he talks about the learnings of the past decades that helped him to update his excellent 1983 "Hints for computer system design". The presentation was part of the Heidelberg Laureate Forum helt in September of this year. At the Forum many of the Abel, Fields and Turing Laureates held presentations. Our most famous computer scientists like Fernando Carbato, Stephen Cook, Edward Feigenbaum, Juris Hartmanis, John Hopcroft, Alan Kay, Vinton Cerf, etc.
Taking DynamoDB beyond Key-Value: Now with Faster, More Flexible, More Powerful Query Capabilities
We launched DynamoDB last year to address the need for a cloud database that provides seamless scalability, irrespective of whether you are doing ten transactions or ten million transactions, while providing rock solid durability and availability. Our vision from the day we conceived DynamoDB was to fulfil this need without limiting the query functionality that people have come to expect from a database.
Taking DynamoDB beyond Key-Value: Now with Faster, More Flexible, More Powerful Query Capabilities
We launched DynamoDB last year to address the need for a cloud database that provides seamless scalability, irrespective of whether you are doing ten transactions or ten million transactions, while providing rock solid durability and availability. Our vision from the day we conceived DynamoDB was to fulfil this need without limiting the query functionality that people have come to expect from a database. However, we also knew that building a distributed database that has unlimited scale and maintains predictably high performance while providing rich and flexible query capabilities, is one of the hardest problems in database development, and will take a lot of effort and invention from our team of distributed database engineers to solve.
Expanding the Cloud: Enabling Globally Distributed Applications and Disaster Recovery
As I discussed in my re:Invent keynote earlier this month, I am now happy to announce the immediate availability of Amazon RDS Cross Region Read Replicas, which is another important enhancement for our customers using or planning to use multiple AWS Regions to deploy their applications. Cross Region Read Replicas are available for MySQL 5.6 and enable you to maintain a nearly up-to-date copy of your master database in a different AWS Region. In case of a regional disaster, you can simply promote your read replica in a different region to a master and point your application to it to resume operations.
Expanding the Cloud: Enabling Globally Distributed Applications and Disaster Recovery
As I discussed in my re:Invent keynote earlier this month, I am now happy to announce the immediate availability of Amazon RDS Cross Region Read Replicas, which is another important enhancement for our customers using or planning to use multiple AWS Regions to deploy their applications. Cross Region Read Replicas are available for MySQL 5.
AWS re:Invent 2013
Today we are kicking off AWS re:Invent 2013. Over the course of the next three days, we will host more than 200 sessions, training bootcamps, and hands on labs taught by expert AWS staff as well as dozens of our customers. This years conference kicks off with a keynote address by AWS Senior Vice President Andy Jassy, followed by my keynote on Thursday morning. Tune in to hear the latest from AWS and our customers. If youre not already here in Vegas with us, you can sign up to watch the keynotes on live stream here. Outside of the keynotes, there are an incredible number of sessions offering a tailored experience whether you are a developer, startup, executive, partner, or other.
AWS re:Invent 2013
Today we are kicking off AWS re:Invent 2013. Over the course of the next three days, we will host more than 200 sessions, training bootcamps, and hands on labs taught by expert AWS staff as well as dozens of our customers. This year?s conference kicks off with a keynote address by AWS Senior Vice President Andy Jassy, followed by my keynote on Thursday morning.
Simplifying Mobile App Data Management with DynamoDB's Fine-Grained Access Control
Speed of development, scalability, and simplicity of management are among the critical needs of mobile developers. With the proliferation of mobile devices and users, and small agile teams that are tasked with building successful mobile apps that can grow from 100 users to 1 million users in a few days, scalability of the underlying infrastructure and simplicity of management are more important than ever. We created DynamoDB to make it easy to set up and scale databases so that developers can focus on building great apps without worrying about the muck of managing the database infrastructure. As I have mentioned previously, companies like Crittercism and Dropcam have already built exciting mobile businesses leveraging DynamoDB.
Simplifying Mobile App Data Management with DynamoDB's Fine-Grained Access Control
Speed of development, scalability, and simplicity of management are among the critical needs of mobile developers. With the proliferation of mobile devices and users, and small agile teams that are tasked with building successful mobile apps that can grow from 100 users to 1 million users in a few days, scalability of the underlying infrastructure and simplicity of management are more important than ever.
Back-to-Basics Weekend Reading - U-Net: A User-Level Network Interface
Many of you know Thorsten von Eicken as the founder of Rightscale, the company that has helped numerous organizations find their way onto AWS. In what seems almost a previous life by now Thorsten was one of the top young professors in Distributed Systems and I had the great pleasure of working with him at Cornell in the early 90's. What set Thorsten aside from so many other system research academics was his desire to build practical, working systems, a path that I followed as well. In the back to basics readings this week I am re-reading a paper from 1995 about the work that I did together with Thorsten on solving the problem of end-to-end low-latency communication on high-speed networks.
Back-to-Basics Weekend Reading - U-Net: A User-Level Network Interface
Many of you know Thorsten von Eicken as the founder of Rightscale, the company that has helped numerous organizations find their way onto AWS. In what seems almost a previous life by now Thorsten was one of the top young professors in Distributed Systems and I had the great pleasure of working with him at Cornell in the early 90’s.
AWS Activate Supporting Startups on AWS
I am very excited to announce AWS Activate, a program designed to provide startups with the resources they need to build applications on AWS. Startups will forever be a very important customer segment of AWS. They were among our first customers and along the way some amazing businesses have been built by these startups, many of which running for 100% on AWS. Startups operate in a world of high uncertainty and limited capital, so an elastic and on-demand infrastructure at low and variable cost aligns very naturally with their needs. By reducing the cost of failure and democratizing access to infrastructure, the cloud has enabled more startups to build, experiment, and scale.
AWS Activate ? Supporting Startups on AWS
I am very excited to announce AWS Activate, a program designed to provide startups with the resources they need to build applications on AWS. Startups will forever be a very important customer segment of AWS. They were among our first customers and along the way some amazing businesses have been built by these startups, many of which running for 100% on AWS.
Back-to-Basics Weekend Reading - Tor: The Second-Generation Onion Router
The anonymity routing network Tor is frequently in the news these days, which makes it a good case to read up on the fascinating technologies behind it. Tor stands for The Onion Router as its technology is based on the onion routing principles. These principles were first described by Goldschlag, et al., from the Naval Research Lab, in their 1996 paper on Hiding Routing Information. Almost immediately work started on addressing a number of omissions in the original work in what became known as the second-generation onion router. Tor is the implementation of such a second generation router and has a number of fascinating features.
Back-to-Basics Weekend Reading - Tor: The Second-Generation Onion Router
The anonymity routing network Tor is frequently in the news these days, which makes it a good case to read up on the fascinating technologies behind it. Tor stands for The Onion Router as its technology is based on the onion routing principles. These principles were first described by Goldschlag, et al.
Fear, Uncertainty and Desperation
Back-to-Basics Weekend Reading - A Decomposition Storage Model
Traditionally records in a database were stored as such: the data in a row was stored together for easy and fast retrieval. Not everybody agreed that the "N-ary Storage Model" (NSM) was the best approach for all workloads but it stayed dominant until hardware constraints, especially on caches, forced the community to revisit some of the alternatives. Combined with the rise of data warehouse workloads, where there is often significant redundancy in the values stored in columns, and database models based on column oriented storage took off. The first practical modern implementation is probably C-Store by Stonebraker, et al. in 2005.
Back-to-Basics Weekend Reading - A Decomposition Storage Model
Traditionally records in a database were stored as such: the data in a row was stored together for easy and fast retrieval. Not everybody agreed that the “N-ary Storage Model” (NSM) was the best approach for all workloads but it stayed dominant until hardware constraints, especially on caches, forced the community to revisit some of the alternatives.
Dutch Enterprises and The Cloud
This spring I travelled through Europe for the AWS Global Summit series. In my many conversations with customers, and with the media, I encountered surprise and excitement about the extent that European enterprises have already been using the Amazon Web Services for some time. Whether it is large telecommunications manufactures like Nokia Siemens Networks running their real-time data analytics for network operators on AWS, or a luxury hotel chain like Kempinski moving their core IT functions to AWS such that they can get out of the IT business, or a major newspaper corporation like News International, who plan to have 75% of their infrastructure running on AWS within 3 years to improve their agility, European enterprises have been moving to the cloud for some time to become more agile and competitive.
Dutch Enterprises and The Cloud
This spring I travelled through Europe for the AWS Global Summit series. In my many conversations with customers, and with the media, I encountered surprise and excitement about the extent that European enterprises have already been using the Amazon Web Services for some time.
DynamoDB for Location Data: Geospatial querying on DynamoDB datasets
Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data. The explosive growth in mobile devices and mobile apps is generating a huge amount of data, which has fueled the demand for big data services and for high scale databases. Meanwhile, mobile app developers have shown that they care a lot about getting to market quickly, the ability to easily scale their app from 100 users to 1 million users on day 1, and the extreme low latency database performance that is crucial to ensure a great end-user experience. These factors have made DynamoDB a compelling database for mobile developers, who happen to be among the biggest adopters of this technology.
DynamoDB for Location Data: Geospatial querying on DynamoDB datasets
Over the past few years, two important trends that have been disrupting the database industry are mobile applications and big data. The explosive growth in mobile devices and mobile apps is generating a huge amount of data, which has fueled the demand for big data services and for high scale databases.
Expanding the Cloud: More memory, more caching and more performance for your data
Today, we added two important choices for customers running high performance apps in the cloud: support for Redis in Amazon ElastiCache and a new high memory database instance (db.cr1.8xlarge) for Amazon RDS. As we prepared to launch these features, I was struck not only by the range of services we provide to enable customers to run fully managed, scalable, high performance database workloads, including Amazon RDS, Amazon DynamoDB, Amazon Redshift and Amazon ElastiCache, but also by the pace at which these services are evolving and improving. Since you now have lots of choices to address your high performance database needs, I decided to write this blog to help you select the most appropriate services for your workload using lessons I have learnt by scaling the infrastructure for Amazon.com.
Expanding the Cloud: More memory, more caching and more performance for your data
Today, we added two important choices for customers running high performance apps in the cloud: support for Redis in Amazon ElastiCache and a new high memory database instance (db.cr1.8xlarge) for Amazon RDS. As we prepared to launch these features, I was struck not only by the range of services we provide to enable customers to run fully managed, scalable, high performance database workloads, including Amazon RDS, Amazon DynamoDB, Amazon Redshift and Amazon ElastiCache, but also by the pace at which these services are evolving and improving.
Back-to-Basics Weekend Reading - An Introduction to Spatial Database Systems
Storing and querying datasets that contain objects in a geometric space have always required special treatment. The choice of data structures and query algorithms can easily make the different between a query that runs in seconds or in days. Much of the fundamental work has been done in the late eighties and early nineties, for examples around topological relations (disjoint, meet, equal, overlap, contains, etc.) , direction relations (north, north-east, etc.) and distance relations (far, near), and also with respect to spatial data structures (a great survey by Hanan Samet). With location becoming a more important attribute to many modern datasets a solid understanding of the tradeoffs is important.
Back-to-Basics Weekend Reading - An Introduction to Spatial Database Systems
Storing and querying datasets that contain objects in a geometric space have always required special treatment. The choice of data structures and query algorithms can easily make the different between a query that runs in seconds or in days. Much of the fundamental work has been done in the late eighties and early nineties, for examples around topological relations (disjoint, meet, equal, overlap, contains, etc.
Back-to-the-Future Weekend Reading - Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud
The intense travels around the world in the spring have kept me from keeping up on the historical reading that I would like to do, as such there have not been that many suggesting for the back-to-basics reading list. The fall is going be not that much different but I will make an effort to get back into a reading habit. I want to kick off the fall readings not with an historical paper but with two that detail GraphLab, an excellent framework for high performance machine learning that originally has been built by the Carlos Guestrin. GraphLab has been used to build several different data mining and graph processing toolkits and applications.
Back-to-the-Future Weekend Reading - Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud
The intense travels around the world in the spring have kept me from keeping up on the historical reading that I would like to do, as such there have not been that many suggesting for the back-to-basics reading list. The fall is going be not that much different but I will make an effort to get back into a reading habit.
Making Mobile App Development Easier with Cross Platform Mobile Push
This year as I hosted AWS Summits in 12 different cities around the world, I met thousands of developers who are building powerful new applications for smartphones, tablets and other connected devices, all running mobile cloud backends on AWS. These developers want to engage their users with timely, dynamic content even when the users havent opened their mobile apps. For example, baseball fans want to know as soon as their favorite team player hits a home run, so they can watch a video replay and catch the rest of the game. The rising proliferation of cheap and powerful sensors means not only apps but smart devices want to communicate important information.
Making Mobile App Development Easier with Cross Platform Mobile Push
This year as I hosted AWS Summits in 12 different cities around the world, I met thousands of developers who are building powerful new applications for smartphones, tablets and other connected devices, all running mobile cloud backends on AWS. These developers want to engage their users with timely, dynamic content even when the users haven?t opened their mobile apps.
Feeling the Customer Love for AWS
We work hard to meet our customers expectations and to innovate continuous on their behalf. This week at the Singapore AWS Summit we were fortunate that our customers Astro Radio from Kuala Lumpur were willing to join us on stage. Jayaram Gopinath Nagaraj and Kavitha Doraimaickam gave truly electrifying presentation about how AWS has transformed their radio stations. They also brought with them a video that showed their appreciation for how we enable them to innovate. It humbling and fun at the same time.
Feeling the Customer Love for AWS
We work hard to meet our customer’s expectations and to continue to innovate on their behalf. This week at the Singapore AWS Summit we were fortunate that our customers Astro Radio from Kuala Lumpur were willing to join us on stage. Jayaram Gopinath Nagaraj and Kavitha Doraimaickam gave a truly electrifying presentation about how AWS has transformed their radio stations.
AWS re:Invent 2013
The AWS re:Invent user conference last year in Las Vegas was by many described as the best technology conference they had been to in a long time. We had worked hard to give you great keynote sessions as well as deep technical content by AWS engineers, partners and customers. This year we will again work hard to create a conference that will exceed your expectations of a conference that is unique in its high quality content and engagement.You can choose from 175+ sessions, training bootcamps, hands-on labs, and hackathons to gain deeper skills and knowledge of the AWS Cloud. Bring your entire executive and technical teams and walk away with the skills and knowledge to refine your cloud strategy, improve developer productivity, increase application performance and security, and reduce infrastructure costs.
AWS re:Invent 2013
The AWS re:Invent user conference last year in Las Vegas was by many described as the best technology conference they had been to in a long time. We had worked hard to give you great keynote sessions as well as deep technical content by AWS engineers, partners and customers. This year we will again work hard to create a conference that will exceed your expectations of a conference that is unique in its high quality content and engagement.
Exerting Fine Grain Control Over Your Cloud Resources
I am thrilled that now both Amazon EC2 and Amazon RDS support resource-level permissions. As customers move increasing amounts of compute and database workloads over to AWS, they have expressed an increased desire for finer grain control over their underlying resources. You can now use these new features to define the permissions your AWS IAM users (and applications) have to perform actions on specific or groups of Amazon EC2 and Amazon RDS resources. You can apply user-defined tags to your EC2 and RDS resources to help organize resources according to whatever schema is most relevant for a particular organization be it an application stack, an organization unit, a cost center, or any other schema that might be appropriate.
Exerting Fine Grain Control Over Your Cloud Resources
I am thrilled that now both Amazon EC2 and Amazon RDS support resource-level permissions. As customers move increasing amounts of compute and database workloads over to AWS, they have expressed an increased desire for finer grain control over their underlying resources. You can now use these new features to define the permissions your AWS IAM users (and applications) have to perform actions on specific or groups of Amazon EC2 and Amazon RDS resources.
Back-to-Basics Weekend Reading - Auctions and bidding: A guide for computer scientists
I have just returned from the AWS Summits in New Zealand and Japan, which were both very well attended and, according to the feedback, very successful. While I was in New Zealand I had great discussion with the folks from Trade Me, the auction site which according to some counts for 70% of all NZ internet traffic. This resulted in some deep technical conversations later, over beer, with some colleagues and customers about the principles behinds different auction and bidding styles. I noticed that my basic knowledge there was rather rusty and I have decided to use this weekend to go a bit more in-depth in the various styles and techniques.
Back-to-Basics Weekend Reading - Auctions and bidding: A guide for computer scientists
I have just returned from the AWS Summits in New Zealand and Japan, which were both very well attended and, according to the feedback, very successful. While I was in New Zealand I had great discussion with the folks from Trade Me, the auction site which according to some counts for 70% of all NZ internet traffic.
Amazon Redshift and Designing for Security
Its been a few months since I last wrote about Amazon Redshift and I thought Id update you on some of the things we are hearing from customers. Since we launched, weve been adding over a hundred customers a week and are well over a thousand today. Thats pretty stunning. As far as I know, its unprecedented for this space. Weve enabled our customers to save tens of millions of dollars in up front capital expenses by using Amazon Redshift. Its clear that Amazon Redshifts message of price, performance and simplicity has resonated with our customers. Thats no surprise these are core principles for every AWS service.
Amazon Redshift and Designing for Security
It?s been a few months since I last wrote about Amazon Redshift and I thought I?d update you on some of the things we are hearing from customers. Since we launched, we?ve been adding over a hundred customers a week and are well over a thousand today. That?s pretty stunning. As far as I know, it?s unprecedented for this space.
DynamoDB Keeps Getting Better (and cheaper!)
We love getting feedback so we can deliver the improvements and new features that really matter to our customers. You can see from the pace at which we roll out new functionality that teams across AWS take this very seriously. One of the teams thats iterating quickly is DynamoDB. They recently launched Local Secondary Indexes and today they are releasing several new features that will help customers build faster, cheaper, and more flexible applications: Parallel Scans To be able to increase the throughput of table scans, the team has introduce new functionality that allows you to scan through the table with multiple threads concurrently.
DynamoDB Keeps Getting Better (and cheaper!)
We love getting feedback so we can deliver the improvements and new features that really matter to our customers. You can see from the pace at which we roll out new functionality that teams across AWS take this very seriously. One of the teams that?s iterating quickly is DynamoDB. They recently launched Local Secondary Indexes and today they are releasing several new features that will help customers build faster, cheaper, and more flexible applications:
Back-to-Basics Weekend Reading - Using continuations to implement thread management and communication in operating systems
I have returned from a great series of AWS Summits in NYC and in Europe so it is time to get back to some weekend reading. During the nineties much operating systems research focussed on microkernels, which resulted in a large collection of prototype systems: Mach 3.0, L3/L4, Plan 9, Xenokernel, Minix and others. Not many of those made into production, the version of Mach that rolled into Mac OS X through the XNU integration was an earlier, monolithic version. I believe commercially QNX has been the most successful microkernel. There was a wealth of interesting, fundamental research triggered by the concepts of microkernels: new communication paradigms, memory management structures, schedulers, etc.
Back-to-Basics Weekend Reading - Using continuations to implement thread management and communication in operating systems
I have returned from a great series of AWS Summits in NYC and in Europe so it is time to get back to some weekend reading. During the nineties much operating systems research focussed on microkernels, which resulted in a large collection of prototype systems: Mach 3.0, L3/L4, Plan 9, Xenokernel, Minix and others.
Expanding the Cloud: Faster, More Flexible Queries with DynamoDB
Today, Im thrilled to announce that we have expanded the query capabilities of DynamoDB. We call the newest capability Local Secondary Indexes (LSI). While DynamoDB already allows you to perform low-latency queries based on your tables primary key, even at tremendous scale, LSI will now give you the ability to perform fast queries against other attributes (or columns) in your table. This gives you the ability to perform richer queries while still meeting the low-latency demands of responsive, scalable applications. Our customers have been asking us to expand the query capabilities of DynamoDB and were excited to see how they use LSI.
Expanding the Cloud: Faster, More Flexible Queries with DynamoDB
Today, I?m thrilled to announce that we have expanded the query capabilities of DynamoDB. We call the newest capability Local Secondary Indexes (LSI). While DynamoDB already allows you to perform low-latency queries based on your table?s primary key, even at tremendous scale, LSI will now give you the ability to perform fast queries against other attributes (or columns) in your table.
Back-to-Basics Weekend Reading - Join Processing in Relational Databases
Joins are one of the fundamental relational database query operations. It is very hard to implement the join operation efficiently as there any many unknowns in the execution of the operation. In the early days much relation database research was done in understanding the complexity of performing joins, what exactly impacted their performance and which approach performed better under which conditions. In 1992 Priti Mishra and Margaret Eich conducted a survey on what was achieved until then in Join Processing and described in details the algorithms, the implementation complexity and the performance. Which make it a good back-to-basics paper to read this weekend.
Back-to-Basics Weekend Reading - Join Processing in Relational Databases
Joins are one of the fundamental relational database query operations. It is very hard to implement the join operation efficiently as there any many unknowns in the execution of the operation. In the early days much relation database research was done in understanding the complexity of performing joins, what exactly impacted their performance and which approach performed better under which conditions.
Back-to-Basics Weekend Reading - Practical Applications of Triggers and Constraints: Successes and Lingering Issues
At the end of the 80's Ceri and Widom were researching the fundamentals of integrity constraints in databases. In 2000 they were invited by the VLDB conference to review 10 years of work around Constraints and Triggers with an eye on the practical application of both abstractions. The resulting paper gives a good overview of the fundamentals of both concepts. Practical Applications of Triggers and Constraints: Success Stories and Lingering Issues, Stefano Ceri, Roberta Cochrane, and Jennifer. Widom, In 26th Very Large Data Bases Conference Proceedings, Cairo, September 2000, Pages 254-262
Back-to-Basics Weekend Reading - Practical Applications of Triggers and Constraints: Successes and Lingering Issues
At the end of the 80’s Ceri and Widom were researching the fundamentals of integrity constraints in databases. In 2000 they were invited by the VLDB conference to review 10 years of work around Constraints and Triggers with an eye on the practical application of both abstractions. The resulting paper gives a good overview of the fundamentals of both concepts.
Back-to-Basics Weekend Reading - Principles of Transaction-Oriented Database Recovery
I have been reading mainly newer papers in the beginning of this year, but it is time to get back to the basics and start reading some more historical papers again. From the time when researchers and engineers where laying the foundations for our current systems. A good early paper to start again is the Survey that Härder en Reuter did on Database Recovery in 1983. Principles of Transaction-Oriented Database Recovery, Theo Härder and Andreas Reuter, ACM Computing Surveys, Volume 15 Issue 4, December 1983, Pages 287-317
Back-to-Basics Weekend Reading - Principles of Transaction-Oriented Database Recovery
I have been reading mainly newer papers in the beginning of this year, but it is time to get back to the basics and start reading some more historical papers again. From the time when researchers and engineers where laying the foundations for our current systems. A good early paper to start again is the Survey that Härder en Reuter did on Database Recovery in 1983.
The Netflix OSS Cloud Prize
Netflix has over the years become one of the absolute best engineering powerhouses for building cloud-native applications. At AWS we are very proud to be their infrastructure partner and every day we learn from how they use our cloud services. Many of the observations I talk about in my 21st Century Application Architectures presentation come from seeing Netflix architects at work. Netflix has gone beyond just building great applications; they have made fundamental pieces of their cloud platform available as open source and many in the industry have responded to that with great enthusiasm, evidenced by the packed Netflix House in February where people came to hear more about NetflixOSS.
The Netflix OSS Cloud Prize
Netflix has over the years become one of the absolute best engineering powerhouses for building cloud-native applications. At AWS we are very proud to be their infrastructure partner and every day we learn from how they use our cloud services. Many of the observations I talk about in my ?21st Century Application Architectures? presentation come from seeing Netflix architects at work.
Beanstalk a la Node
I spent a lot of time talking to AWS developers, many working in the gaming and mobile space, and most of them have been finding Node.js well suited for their web applications. With its asynchronous, event-driven programming model, Node.js allows these developers to handle a large number of concurrent connections with low latencies. These developers typically use EC2 instances combined with one of our database services to create web services used for data retrievals or to create dynamic mobile interfaces. Today, AWS Elastic Beanstalk just added support for Node.js to help developers easily deploy and manage these web applications on AWS.
Elastic Beanstalk a la Node
I spent a lot of time talking to AWS developers, many working in the gaming and mobile space, and most of them have been finding Node.js well suited for their web applications. With its asynchronous, event-driven programming model, Node.js allows these developers to handle a large number of concurrent connections with low latencies.
DynamoDB One Year Later: Bigger, Better, and 85% Cheaper&
Time passes very quickly around here and I hadnt realized until recently that over a year has gone by since we launched DynamoDB. As I sat down with the DynamoDB team to review our progress over the last year, I realized that DynamoDB had surpassed even my own expectations for how easily applications could achieve massive scale and high availability with DynamoDB. Many of our customers have, with the click of a button, created DynamoDB deployments in a matter of minutes that are able to serve trillions of database requests per year. Ive written about it before, but I continue to be impressed by Shazams use of DynamoDB, which is an extreme example of how DynamoDBs fast and easy scalability can be quickly applied to building high scale applications.
DynamoDB One Year Later: Bigger, Better, and 85% Cheaper?
Time passes very quickly around here and I hadn?t realized until recently that over a year has gone by since we launched DynamoDB. As I sat down with the DynamoDB team to review our progress over the last year, I realized that DynamoDB had surpassed even my own expectations for how easily applications could achieve massive scale and high availability with DynamoDB.
Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution
Today Amazon Web Services launched AWS OpsWorks, a flexible application management solution with automation tools that enable you to model and control your applications and their supporting infrastructure. OpsWorks allows you to manage the complete application lifecycle, including resource provisioning, configuration management, application deployment, software updates, monitoring, and access control. As with all the AWS Application Management services AWS OpsWorks is provided at no additional charge. AWS customers only pay for those resources that they have used. Simplified Application Management OpsWorks is designed for IT administrators and ops-minded developers who want an easy way to manage applications of nearly any scale and complexity without sacrificing control.
Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution
Today Amazon Web Services launched AWS OpsWorks, a flexible application management solution with automation tools that enable you to model and control your applications and their supporting infrastructure. OpsWorks allows you to manage the complete application lifecycle, including resource provisioning, configuration management, application deployment, software updates, monitoring, and access control.
Amazon Redshift and Designing for Resilience
As you may remember from our announcement at re: Invent in November 2012, Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service that delivers fast query performance at less than one tenth the cost of most traditional data warehouse systems. Ive been eagerly waiting for Amazon Redshifts launch since we announced the service preview at re: Invent and Im delighted that its now available for all customers in the US East (N. Virginia) Region, with additional AWS Regions planned for the coming months. To get started with Amazon Redshift, visit: http://aws.amazon.com/redshift. Amazon Redshift and Resilience Previously, Ive written at length about how Amazon Redshift achieves high performance.
Amazon Redshift and Designing for Resilience
As you may remember from our announcement at re: Invent in November 2012, Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service that delivers fast query performance at less than one tenth the cost of most traditional data warehouse systems. I?ve been eagerly waiting for Amazon Redshift?s launch since we announced the service preview at re: Invent and I?m delighted that it?s now available for all customers in the US East (N.
Expanding the Cloud: The Amazon Elastic Transcoder
While I was returning from an exciting time in New Orleans watching the Super Bowl, AWS launched a very cool, brand new service: Amazon Elastic Transcoder. Amazon Elastic Transcoder is designed to be very easy to use, scalable and cost-effective video transcoding in the cloud. Jeff Barr did an excellent job running through the service on his blog and you can also check out the detail page. I spent a lot of time talking to AWS customers who were also in New Orleans, many of them who work with media, and all emphasized that transcoding fits naturally with services that we already offer like storage (Amazon S3 and Glacier) and delivery (Amazon CloudFront).
Expanding the Cloud: The Amazon Elastic Transcoder
While I was returning from an exciting time in New Orleans watching the Super Bowl, AWS launched a very cool, brand new service: Amazon Elastic Transcoder. Amazon Elastic Transcoder is designed to be very easy to use, scalable and cost-effective video transcoding in the cloud. Jeff Barr did an excellent job running through the service on his blog and you can also check out the detail page.
Back-to-Basics Weekend Reading - Epidemics
My paper to read this weekend was the Alan Demers' seminal paper on epidemic techniques for database replication. I realized that in 2004, before my Amazon days, I already wrote a blog post about the fundamental publications in the area of epidemics, so this seems like a good moment to revisit that with updated links, etc.
Back-to-Basics Weekend Reading - Epidemics
My paper to read this weekend was the Alan Demers' seminal paper on epidemic techniques for database replication. I realized that in 2004, before my Amazon days, I already wrote a blog post about the fundamental publications in the area of epidemics, so this seems like a good moment to revisit that with updated links, etc. History of Epidemics In the past 6-8 years we have been using various epidemic techniques in building our reliable and scalable distributed systems with great success. Now that industry is starting to deal with issues of scale that can almost only be solved by using epidemic techniques, it becomes important to produce some basic pointers to the origin of the use of epidemics in distributed systems.
My Best Christmas Present Root Domain Support for Amazon S3 Website Hosting
I have been a big fan of the Amazon S3 Static Website Hosting feature since its launch and this blog happily is being served from it. S3 is not only a highly reliable and available storage service but also one of the most powerful web serving engines that exists today. By storing your website in Amazon S3, you suddenly no longer have to worry about scaling, replication, performance, security, etc. All of that is handled seamlessly by S3. As such I am very happy that the Amazon S3 team has finally knocked off the last piece of dependency on an external infrastructure piece.
My Best Christmas Present ? Root Domain Support for Amazon S3 Website Hosting
I have been a big fan of the Amazon S3 Static Website Hosting feature since its launch and this blog happily is being served from it. S3 is not only a highly reliable and available storage service but also one of the most powerful web serving engines that exists today. By storing your website in Amazon S3, you suddenly no longer have to worry about scaling, replication, performance, security, etc.
An Album for Each Year - 2012 Version
About 5 years ago I joined a challenge to list "a favorite album for every year of your life." The challenge has two restrictions: only one album per year and there can be no repeats of artists. I added for myself the restriction that I should actually own the album, which restricts the set to choose from significantly and also makes for some peculiar choices. My list stopped in 2007, so now that 2012 is almost at its end it is a good moment to add the next 5 years to the list 1958: Jerry Lee Lewis, Great Balls of Fire 1959: Ray Charles, What I'd Say 1960: Miles Davis, Sketches of Spain 1961: Robert Johnson, King of the Delta Blues Singers 1962: Booker T & MG, Green Onions 1963: James Brown, Live at the Apollo 1964: John Coltrane, Love Supreme 1965: Bob Dylan, Highway 61 Revisted 1966: Cream, ...
An Album for Each Year - 2012 Version
About 5 years ago I joined a challenge to list "a favorite album for every year of your life." The challenge has two restrictions: only one album per year and there can be no repeats of artists. I added for myself the restriction that I should actually own the album, which restricts the set to choose from significantly and also makes for some peculiar choices.
The Back-to-Basics Readings of 2012
After the AWS re: Invent conference I spent two weeks in Europe for the last customer visits of the year. I have since returned and am now in New York City enjoying a few days of winding down the last activities of the year before spending the holidays here with family. Do not expect too many blog posts or twitter updates. Although there are still a few very exciting AWS news updates to happen this year. I thought this was a good moment to collect all the readings I suggested this year in one summary post. It was not until later in the year that I started to recording the readings here on the blog, so I hope this is indeed the complete list.
The Back-to-Basics Readings of 2012
After the AWS re: Invent conference I spent two weeks in Europe for the last customer visits of the year. I have since returned and am now in New York City enjoying a few days of winding down the last activities of the year before spending the holidays here with family.
Back-to-Basics Weekend Reading - Sparse Partitions
The amazing AWS re: Invent conference completed last night and I am on my way to Europe for a last visit to customers this year. I am carrying with me a more theoretical paper on the principles of distributed computing: Sparse Partitions by Awerbug and Peleg. It deals with the failure of control if networks grow larger and presents several solutions based on locality that have found practical applications. Sparse Partition, Baruch Awerbuch and David Peleg, Proceedings of the 31st Annual Symposium on Foundations of Computer Science (FOCS), 503-513, October 1990.
Back-to-Basics Weekend Reading - Sparse Partitions
The amazing AWS re: Invent conference completed last night and I am on my way to Europe for a last visit to customers this year. I am carrying with me a more theoretical paper on the principles of distributed computing: Sparse Partitions by Awerbug and Peleg. It deals with the failure of control if networks grow larger and presents several solutions based on locality that have found practical applications.
Expanding the Cloud Announcing Amazon Redshift, a Petabyte-scale Data Warehouse Service
Today, we are excited to announce the limited preview of Amazon Redshift, a fast and powerful, fully managed, petabyte-scale data warehouse service in the cloud. Amazon Redshift enables customers to obtain dramatically increased query performance when analyzing datasets ranging in size from hundreds of gigabytes to a petabyte or more, using the same SQL-based business intelligence tools they use today. Customers have been asking us for a data warehouse service for some time now and were excited to be able to deliver this to them. Amazon Redshift uses a variety of innovations to enable customers to rapidly analyze datasets ranging in size from several hundred gigabytes to a petabyte and more.
Expanding the Cloud ? Announcing Amazon Redshift, a Petabyte-scale Data Warehouse Service
Today, we are excited to announce the limited preview of Amazon Redshift, a fast and powerful, fully managed, petabyte-scale data warehouse service in the cloud. Amazon Redshift enables customers to obtain dramatically increased query performance when analyzing datasets ranging in size from hundreds of gigabytes to a petabyte or more, using the same SQL-based business intelligence tools they use today.
Back-to-Basics Weekend Reading - The Andrew File System
This weekend I am heading to Brussels for meetings with the European Commission, specifically with Vice-president Neelie Kroes who owns the Digital Agenda for the EU, about how to accelerate cloud usage in both business and government in Europe. I am bringing with me a paper with one of first distributed systems that had actually see wide-spread commercial deployment. The Andrew File System (AFS) was developed at CMU and was much more than just a distributed file systems and had a very interesting caching and volume replication architecture. Scale and performance in a distributed file system, John H. Howard, Michael L.
Back-to-Basics Weekend Reading - The Andrew File System
This weekend I am heading to Brussels for meetings with the European Commission, specifically with Vice-president Neelie Kroes who owns the Digital Agenda for the EU, about how to accelerate cloud usage in both business and government in Europe. I am bringing with me a paper with one of first distributed systems that had actually see wide-spread commercial deployment.
Expanding the Cloud introducing the Asia Pacific (Sydney) Region
Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS Region in Sydney, Australia. This new Asia Pacific (Sydney) Region has been highly requested by companies worldwide, and it provides low latency access to AWS services for those who target customers in Australia and New Zealand. The Region launches with two Availability Zones to help customers build highly available applications. I have visited Australia at least twice every year for the past four years and I have seen first-hand evidence of the tremendous interest there is in the AWS service. Many young businesses as well as established enterprises are already using AWS, many of them targeting customers globally.
Expanding the Cloud ? introducing the Asia Pacific (Sydney) Region
Today, Amazon Web Services has greater worldwide coverage with the launch of a new AWS Region in Sydney, Australia. This new Asia Pacific (Sydney) Region has been highly requested by companies worldwide, and it provides low latency access to AWS services for those who target customers in Australia and New Zealand.
Back-to-Basics Weekend Reading - Using Encryption for Authentication
Now that I am enjoying some time in Seattle with real weekends, I like to remind you that my reading list is called back-to-basics. My goal with reading these papers is that by revisiting the original problems systems researchers were trying to solve you get a much better understanding of the challenges we are often still faced with today. That means that many papers on these papers are "old", as I was recently told, even published before some of you were born :-). That might definitely be the case with this famous Needham - Schroeder paper from 1978. Roger Needham and Mike Schroeder were some of the first researchers to tackle the problem of secure communication over insecure networks; their protocols deal with how to authenticate both parties and how to establish a secure channel between the parties.
Back-to-Basics Weekend Reading - Using Encryption for Authentication
Now that I am enjoying some time in Seattle with real weekends, I like to remind you that my reading list is called back-to-basics. My goal with reading these papers is that by revisiting the original problems systems researchers were trying to solve you get a much better understanding of the challenges we are often still faced with today.
Improving the Cloud - More Efficient Queuing with SQS
The Amazon Simple Queue Service (SQS) is a highly scalable, reliable and elastic queuing service that 'just works'. Customers from various verticals (media, social gaming, mobile, news, advertisement) such as Netflix, Shazam and Scopely have used SQS in variety of use-cases requiring loose coupling and high performance. For example, AWS customers use SQS for asynchronous communication pipelines, buffer queues for databases, asynchronous work queues, and moving latency out of highly responsive requests paths. Today, the SQS team is launching two important features Long Polling and richer client functionality in the SQS SDK that we believe will extend the reach of SQS to new use cases by reducing the cost of high scale messaging for our customers.
Improving the Cloud - More Efficient Queuing with SQS
The Amazon Simple Queue Service (SQS) is a highly scalable, reliable and elastic queuing service that ‘just works’. Customers from various verticals (media, social gaming, mobile, news, advertisement) such as Netflix, Shazam and Scopely have used SQS in variety of use-cases requiring loose coupling and high performance. For example, AWS customers use SQS for asynchronous communication pipelines, buffer queues for databases, asynchronous work queues, and moving latency out of highly responsive requests paths.
Back-to-Basics Weekend Reading - Weighted Voting for Replicated Data
The last two weeks in Europe and Israel (The image above is from Tel Aviv) were intense so I didn't get to do much reading, hence I didn't post any reading suggestions. This weekend I pick a true back-to-basics paper to read; Dave Gifford's paper on Weighted Voted was the first to describe the "r+w" overlapping quorum approach to reason about the consistency of replicated data. Weighted voting for replicated data, David K. Gifford, Proceedings of the 7th ACM Symposium on Operating Systems Principles, December 10-12, 1979, Pacific Grove, CA USA
Back-to-Basics Weekend Reading - Weighted Voting for Replicated Data
The last two weeks in Europe and Israel (The image above is from Tel Aviv) were intense so I didn’t get to do much reading, hence I didn’t post any reading suggestions. This weekend I pick a true back-to-basics paper to read; Dave Gifford’s paper on Weighted Voted was the first to describe the “r+w” overlapping quorum approach to reason about the consistency of replicated data.
Back-to-Basics Weekend Reading - Automatic Reconfiguration in Autonet
I wrote this post last week but didn't get around to publishing it. I am heading to Europe this weekend for, among other things, Structure Europe and the AWS Summit in Tel Aviv. My time in India is almost over and I am heading back to Seattle. I am bringing with me on the plane a relatively unknown paper that I really enjoyed when it was first published. Autonet was a point-to-point network designed at Dec SRC research lab by Mike Schroeder's group. The original paper on Autonet can be found here. I am actually picking a follow-up paper to read this weekend, it deals with fault-tolerance of the network through automatic reconfiguration of its components.
Back-to-Basics Weekend Reading - Automatic Reconfiguration in Autonet
I wrote this post last week but didn’t get around to publishing it. I am heading to Europe this weekend for, among other things, Structure Europe and the AWS Summit in Tel Aviv. My time in India is almost over and I am heading back to Seattle. I am bringing with me on the plane a relatively unknown paper that I really enjoyed when it was first published.
Back-to-Basics Weekend Reading - Counting Bloom Filters
I am in India for the AWS Summits in Mumbai, Chennai and Bangalore (next week). As always in India I have an amazing time, the events are packed, the participants are extremely enthusiastic and eager to learn, the customers very appreciative and the food is just amazing. This weeks reading was triggered by a note from Matt Wood who ran into a great in-depth analysis of the Bloom Filter data structure by Michael Nielsen on his Data Driven Intelligence blog.
Back-to-Basics Weekend Reading - Counting Bloom Filters
I am in India for the AWS Summits in Mumbai, Chennai and Bangalore (next week). As always in India I have an amazing time, the events are packed, the participants are extremely enthusiastic and eager to learn, the customers very appreciative and the food is just amazing. This weeks reading was triggered by a note from Matt Wood who ran into a great in-depth analysis of the Bloom Filter data structure by Michael Nielsen on his Data Driven Intelligence blog. I love probabilistic data structures and Bloom filters have unique properties of possible false positives, but no false-negatives. They have been used in many network devices, network protocols and distributed applications where a question like "have I possibly seen this before" needs to be able to operate at very large scale.
Expanding the Cloud ? Provisioned IOPS for Amazon RDS
Following the huge success of being able to provision a consistent, user-requested I/O rate for DynamoDB and Elastic Block Store (EBS), the AWS Database Services team has now released Provisioned IOPS, a new high performance storage option for the Amazon Relational Database Service (Amazon RDS). Customers can provision up to 10,000 IOPS (input/output operations per second) per database instance to help ensure that their databases can run the most stringent workloads with rock solid, consistent performance.
Expanding the Cloud Provisioned IOPS for Amazon RDS
Following the huge success of being able to provision a consistent, user-requested I/O rate for DynamoDB and Elastic Block Store (EBS), the AWS Database Services team has now released Provisioned IOPS, a new high performance storage option for the Amazon Relational Database Service (Amazon RDS). Customers can provision up to 10,000 IOPS (input/output operations per second) per database instance to help ensure that their databases can run the most stringent workloads with rock solid, consistent performance. High Performance I/O Amazon RDS Provisioned IOPS is intended for applications that need predictable performance and have database workloads that generate largely random I/O. Amazon RDS Provisioned IOPS is ideal for mission-critical online transaction processing (OLTP) workloads that require a high performance storage option with consistent IOPS, within a narrow band of tolerance.
Back-to-Basics Weekend Reading - Leases
I was in Los Angeles this week for the Digital Media on AWS Summit and to visit many of the studios and production houses that are using AWS for production and post-production work. There is some real jaw dropping work being done around this town and I had the privilege to see some of these highly guarded secrets, all powered by AWS. Of the work that is already public the systems that Uplynk has built for Disney/ABC are impressive. The single source format approach has their customers very enthusiastic about how simple multi device stream is for them with Uplynk handling all the transcoding, dynamic ad-insertion, black out handling by switching from live to vod, etc.
Back-to-Basics Weekend Reading - Leases
I was in Los Angeles this week for the Digital Media on AWS Summit and to visit many of the studios and production houses that are using AWS for production and post-production work. There is some real jaw dropping work being done around this town and I had the privilege to see some of these highly guarded secrets, all powered by AWS.
Back-to-the-Future Weekend Reading - CryptDB
This weekend I am not going back in time to fundamentals, but looking forward to the future. Encryption techniques to protect sensitive information such as personal identifiable information are becoming more and more practical, and ubiquitously. Whether in-the-cloud or on-premise there is a shift to a model where individual applications need to protect themselves instead of relying on firewall-like techniques. That goes especially for the interaction between applications and storage engines, and between applications and databases. In last year's SOSP Hari Balakrishnan's group at MIT CSAIL team presented a paper on CryptDB which has a novel SQL-aware encryption approach. "
Back-to-the-Future Weekend Reading - CryptDB
This weekend I am not going back in time to fundamentals, but looking forward to the future. Encryption techniques to protect sensitive information such as personal identifiable information are becoming more and more practical, and ubiquitously. Whether in-the-cloud or on-premise there is a shift to a model where individual applications need to protect themselves instead of relying on firewall-like techniques.
Expanding Flexibility - Introducing the Reserved Instance Marketplace
Today we launched a new feature that enables you to buy and sell Amazon EC2 Reserved Instances. Reserved Instances are an important pricing option for AWS customers to drive cost down. If you are able to predict the capacity required to run your application, there is likely some combination of Reserved Instance options that will help you drive you costs down significantly (up to 71%) when compared to on-demand pricing. There are three options: heavy-, medium- and low-usage options that allow you to optimize your savings depending on how much you plan to use your Reserved Instance. However, sometimes business and architectures change so that you need to change your mix of Reserved Instances.
Expanding Flexibility - Introducing the Reserved Instance Marketplace
Today we launched a new feature that enables you to buy and sell Amazon EC2 Reserved Instances. Reserved Instances are an important pricing option for AWS customers to drive cost down. If you are able to predict the capacity required to run your application, there is likely some combination of Reserved Instance options that will help you drive you costs down significantly (up to 71%) when compared to on-demand pricing.
A Million Miles Away
I just received a note from United Airlines that I had flown 1 million miles on their airline. I didn't start flying United until I moved to Seattle, so all of these have been miles on Amazon business. The folks I was with immediately started cracking some jokes about a million miles of torture, but that was not at all what came to my mind. I thought about all the wonderful things I got to do on those million miles; all the great engineers and CxO's that I have met, all the cool startups that made time for me and shared their passion, and all the amazing enterprises going through major transformations who opened up about the inner workings of their businesses to me.
A Million Miles Away
I just received a note from United Airlines that I had flown 1 million miles on their airline. I didn’t start flying United until I moved to Seattle, so all of these have been miles on Amazon business. The folks I was with immediately started cracking some jokes about a million miles of torture, but that was not at all what came to my mind.
Back-to-Basics Weekend Reading - Load Sharing
One of the main reasons for picking some of these older papers as back-to-basics reading is that the first researchers on a topic had to develop the fundamental models and principles from scratch. As such there is much to learn from the descriptions of those first algorithms and models. One area that is tremendously important in todays distributed systems and has a long history is load balancing and load management. In this 1986 paper on load sharing Eager, Lazowska and Zahorjan layout the basics. Even though these early system lack todays scale and complexity the fundamentals are there.
Back-to-Basics Weekend Reading - Load Sharing
One of the main reasons for picking some of these older papers as back-to-basics reading is that the first researchers on a topic had to develop the fundamental models and principles from scratch. As such there is much to learn from the descriptions of those first algorithms and models. One area that is tremendously important in todays distributed systems and has a long history is load balancing and load management.
Back-to-Basics Weekend Reading - Granularity of locks
I am at funconf in Ireland. After a ride in a Delorean, a private train ride to Galway and a helicopter flight I am sitting outside a cottage on the island of Inishmore. The attendants are an amazing group of people, mostly engineers, and I am humbled to be invited along.
Back-to-Basics Weekend Reading - Granularity of locks
I am at funconf in Ireland. After a ride in a Delorean, a private train ride to Galway and a helicopter flight I am sitting outside a cottage on the island of Inishmore. The attendants are an amazing group of people, mostly engineers, and I am humbled to be invited along. Everybody is equally passionate about "doing the right" in tech, in business and in life. The discussion shift quickly form one to the other. I brought two papers with me on this part of the trip, one was the paper that Ben Black sent out earlier this week on Data-Structures for Geometric Approximation, which is a great in-depth review of that area and it will take me a while to finish it.
Back-to-Basics Weekend Reading - The 5 Minute Rule
I am in the midst of my South America tour in the beautiful but very cold Santiago, Chili. The AWS team launched this week Amazon Glacier, a cold storage archive service at the very low price point of $0.01 per GB/month. Which makes this week a good moment to read up on some of the historical work around the costs of data engineering. For this purpose I have picked work based on two papers by Jim Gray, the brilliant IBM / Tandem / Microsoft researcher, who won a Turing award for his contributions to data and transaction processing. The papers are from 1987, 1997 and 2007.
Back-to-Basics Weekend Reading - The 5 Minute Rule
I am in the midst of my South America tour in the beautiful but very cold Santiago, Chile. The AWS team launched this week Amazon Glacier, a cold storage archive service at the very low price point of $0.01 per GB/month. Which makes this week a good moment to read up on some of the historical work around the costs of data engineering.
Expanding the Cloud Managing Cold Storage with Amazon Glacier
Managing long-term digital archiving is a challenge for almost every company. With the introduction of Amazon Glacier, IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. Many organizations have to manage some form of long term archiving. Enterprises have regulatory and business requirements to retain everything from email to customers transactions, hospitals create archives of all digital assets related to patients, research and scientific organizations are creating substantial historical archives of their findings, governments want to provide long-term open data access, media companies are creating huge repositories of digital assets, and libraries and other organizations have been looking to archive everything that takes place in society.
Expanding the Cloud ? Managing Cold Storage with Amazon Glacier
Managing long-term digital archiving is a challenge for almost every company. With the introduction of Amazon Glacier, IT organizations now have a solution that removes the headaches of digital archiving and provides extremely low cost storage. Many organizations have to manage some form of long term archiving. Enterprises have regulatory and business requirements to retain everything from email to customers?
Back-to-Basics Weekend Reading - Staged Event-Driven Architecture
I am in São Paolo, Brazil for the 2012 AWS Latin America Summit and for The Next Web Latin America conference. I will also be visiting Chile and Mexico on this trip and have the great fortune to meet many of our Latin American AWS customers. Staged Event-Driven Architecture. Matt Welsh's thesis work at Berkeley was on building high-performance internet services. In a time when the debates on Threads vs Events were still rampant he came up with a practical and elegant approach of combining both, delivering excellent results. Several of the principles from this paper have made it into systems I have since built.
Back-to-Basics Weekend Reading - Staged Event-Driven Architecture
I am in São Paolo, Brazil for the 2012 AWS Latin America Summit and for The Next Web Latin America conference. I will also be visiting Chile and Mexico on this trip and have the great fortune to meet many of our Latin American AWS customers. Staged Event-Driven Architecture. Matt Welsh’s thesis work at Berkeley was on building high-performance internet services.
Total Cost of Ownership and the Return on Agility
In the many meetings with customers in which I have done a deep dive on their architecture and applications to help them create an accurate cost picture, I have observed two common patterns: 1) It is hard for customers to come to an accurate Total Cost of Ownership (TCO) calculation of an on-premise installation and 2) they struggle with how to account for the Return on Agility ; the fact that they are now able to pursue business opportunities much faster at much lower costs points than before. Both of these are important as they help customer accurately gauge the economic benefits of running their applications in the cloud.
Total Cost of Ownership and the Return on Agility
In the many meetings with customers in which I have done a deep dive on their architecture and applications to help them create an accurate cost picture, I have observed two common patterns: 1) It is hard for customers to come to an accurate Total Cost of Ownership (TCO) calculation of an on-premise installation and 2) they struggle with how to account for the ?Return on Agility?; the fact that they are now able to pursue business opportunities much faster at much lower costs points than before.
Back-to-Basics Weekend Reading - On Naming and Binding
I have just returned from two wonderful weeks in the rugged NorthWest wilderness. Every time again I am amazed of the magnificent outdoors at only two hours drive from Seattle. This weeks back to basics is a paper David Richardson reminded me of when we were discussing the merits of Saltzer's end-to-end paper. It is a note by Saltzer that in the most clear and fundamental way lays out the concepts of naming, addressing, routing and binding in distributed systems. It was republished as an IETF RFC given it importance for the design of internet systems: Saltzer, J. H., "On the Naming and Binding of Network Destinations", RFC 1498, August 1993.
Back-to-Basics Weekend Reading - On Naming and Binding
I have just returned from two wonderful weeks in the rugged NorthWest wilderness. Every time again I am amazed of the magnificent outdoors at only two hours drive from Seattle. This weeks back to basics is a paper David Richardson reminded me of when we were discussing the merits of Saltzer’s end-to-end paper.
Back-to-Basics Weekend Reading - On Systems Simplicity
This weekend's reading departs a bit from the more academic papers of the past weeks. They are two classics that deal with great observations about the tensions between simplicity and complexity in building systems. The first paper is the 1980 Turing Lecture delivered by Tony Hoare, "The Emporers Old Clothes". He had received the Turing Award for his contributions to the fundamentals of programming languages, although for me some of his most influential work, communication sequential processes, still had to happen. In his lecture prof. Hoare, in the superb way of great story tellers, has many observations about programmers, program languages and systems building.
Back-to-Basics Weekend Reading - On Systems Simplicity
This weekend’s reading departs a bit from the more academic papers of the past weeks. They are two classics that deal with great observations about the tensions between simplicity and complexity in building systems. The first paper is the 1980 Turing Lecture delivered by Tony Hoare, “The Emporers Old Clothes”. He had received the Turing Award for his contributions to the fundamentals of programming languages, although for me some of his most influential work, communication sequential processes, still had to happen.
Back-to-Basics Weekend Reading - Virtualizing Operating Systems
This weekend's back-to-basics reading is on operating system virtualization. There are two papers that deserve the "classic" tag as they both form the basis for operating system virtualization that is in production today. Stanford's Disco, the predecessor of VMWare, uses a full hardware virtualization approach, where Cambridge's Xen introduced us to paravirtualization. Disco: Running Commodity Operating Systems on Scalable Multiprocessors by Edouard Bugnion, Scott Devine, Kinshuk Govil, Mendel Rosenblum in the Proceedings of the 16th ACM Symposium on Operating Systems Principles, October 5-8, 1997, St. Malo, France. Xen and the art of virtualization by Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Timothy L.
Back-to-Basics Weekend Reading - Virtualizing Operating Systems
This weekend’s back-to-basics reading is on operating system virtualization. There are two papers that deserve the “classic” tag as they both form the basis for operating system virtualization that is in production today. Stanford’s Disco, the predecessor of VMWare, uses a full hardware virtualization approach, where Cambridge’s Xen introduced us to paravirtualization.
Expanding The Cloud High Performance I/O Instances for Amazon EC2
AWS customers are bringing their most demanding workloads onto the cloud. These include the likes of high performance computation, for which we introduced the Cluster Compute and Cluster GPU instance types. Customers are also bringing workloads on AWS that require dedicated and high performance IO for which we are now introducing a new Amazon EC2 instance type, the High I/O Quadruple Extra Large (hi1.4xlarge), to meet their needs. The hi1.4xlarge has 8 cores and 60.5GB of memory. Most importantly it has 2 SSDs of 1 TB each and a 10 Gb/s Ethernet NIC that using placement groups can be directly connected to other High I/O instances.
Expanding The Cloud ? High Performance I/O Instances for Amazon EC2
AWS customers are bringing their most demanding workloads onto the cloud. These include the likes of high performance computation, for which we introduced the Cluster Compute and Cluster GPU instance types. Customers are also bringing workloads on AWS that require dedicated and high performance IO for which we are now introducing a new Amazon EC2 instance type, the High I/O Quadruple Extra Large (hi1.
Register for AWS re: Invent
The first annual AWS user and partner conference will be held November 27-29 at The Venetian in Las Vega. It is shaping up to be a great event with many Amazonians, partners and customers presenting in well over 150 sessions. There are sessions in many different categories: Architecture, Big Data, HPC, Computer & Networking, Storage, Databases, Security, Tools & Languages, Media Sharing & Content Delivery, Managing AWS Resources, Enterprise IT, Mobile, Start-up, and more. Starting today the information about the sessions and the registration is live at http://reinvent.awsevents.com. General registration opens up on July 25, 2012, however active AWS customers will get a chance to jump the line and start registering on July 20.
Register for AWS re: Invent
The first annual AWS user and partner conference will be held November 27-29 at The Venetian in Las Vega. It is shaping up to be a great event with many Amazonians, partners and customers presenting in well over 150 sessions. There are sessions in many different categories: Architecture, Big Data, HPC, Computer & Networking, Storage, Databases, Security, Tools & Languages, Media Sharing & Content Delivery, Managing AWS Resources, Enterprise IT, Mobile, Start-up, and more.
Back-to-Basics Weekend Reading - Hints for Computer Systems Design
For a while now I have been on a track to read one influential/fundamental Computer Science paper each weekend. I find that going back to the basics of system, network and language design forces a good appreciation for keeping designs simple and focus on those fundamentals that matter most to users. Often I posted the paper on twitter and a number of times I have had requests like "what was the paper you posted three weeks ago about memory management?" . I will now post them here so going back in time will be easy. Last weeks paper was the classic End-To-End Arguments in System Design, by J.
Back-to-Basics Weekend Reading - Hints for Computer Systems Design
For a while now I have been on a track to read one influential/fundamental Computer Science paper each weekend. I find that going back to the basics of system, network and language design forces a good appreciation for keeping designs simple and focus on those fundamentals that matter most to users.
Amazon DynamoDB - From the Super Bowl to WeatherBug
Amazon DynamoDB is the fastest growing new service in the history of AWS. In the five months since it launched in January, DynamoDB, our fast and scalable NoSQL database service, has been setting AWS growth records. Customers have used DynamoDB to support Super Bowl advertising campaigns, drive Facebook applications, collect and analyze data from sensor networks, track gaming information, and more. Customers such as Electronic Arts, News International, SmugMug, Shazam, IMDb, Amazon Cloud Drive, and many others are using DynamoDB to power their applications. The number of items that customers are storing in DynamoDB is more than doubling every couple of months (an item is the basic unit of data stored in DynamoDB and is between 0-64KB).
Amazon DynamoDB - From the Super Bowl to WeatherBug
Amazon DynamoDB is the fastest growing new service in the history of AWS. In the five months since it launched in January, DynamoDB, our fast and scalable NoSQL database service, has been setting AWS growth records. Customers have used DynamoDB to support Super Bowl advertising campaigns, drive Facebook applications, collect and analyze data from sensor networks, track gaming information, and more.
Dynamic Content Support in Amazon CloudFront
In the past three and a half years, Amazon CloudFront has changed the content delivery landscape. It has demonstrated that a CDN does not have to be complex to use with expensive contracts, minimum commits, or upfront fees, such that you are forcibly locked into a single vendor for a long time. CloudFront is simple, fast and reliable with the usual pay-as-you-go model. With just one click you can enable content to be distributed to the customer with low latency and high-reliability. Today Amazon CloudFront has taken another major step forward in ease of use. It now supports delivery of entire websites containing both static objects and dynamic content.
Dynamic Content Support in Amazon CloudFront
In the past three and a half years, Amazon CloudFront has changed the content delivery landscape. It has demonstrated that a CDN does not have to be complex to use with expensive contracts, minimum commits, or upfront fees, such that you are forcibly locked into a single vendor for a long time.
AWS re: Invent
Hundreds of thousands of businesses in over 190 countries are relying on AWS to support some or all of their business and IT needs. From SAP and Oracle production installations to risk management HPC, from Internet banking to protein folding, from social gaming to mobile collaboration tools, from video distribution to hotel reservations systems, supply chain management and medical archiving, the list of what our customers achieve using the AWS cloud is fascinating. Many of our customers will tell you that although the cost savings that AWS brings them are important, more important is that they are able to be more agile, that they are able to move faster in a world with murderous competition and highly compressed time-to-markets.
AWS re: Invent
Invention comes in many forms and at many scales. The most radical and transformative of inventions are often those that empower others to unleash their creativity ? to pursue their dreams. We are creating powerful self-service platforms that allow thousands of people to boldly experiment and accomplish things that would otherwise be impossible or impractical.
Expanding the Cloud for Windows Developers
The software that powers todays world of Internet services has become incredibly diverse. Todays announcement of Amazon RDS for Microsoft SQL Server and .NET support for AWS Elastic Beanstalk marks another important step in our commitment to increase the flexibility for AWS customers to use the choice of operating system, programming language, development tools and database software that meet their application requirements. Using the AWS Toolkit for Visual Studio, you can now deploy your .NET applications to AWS Elastic Beanstalk directly from your Visual Studio environment without changing any code. You can then off load the management and scaling of your database and application stack to Amazon RDS and AWS Elastic Beanstalk, and focus on adding value to your customers.
Expanding the Cloud for Windows Developers
The software that powers today?s world of Internet services has become incredibly diverse. Today?s announcement of Amazon RDS for Microsoft SQL Server and .NET support for AWS Elastic Beanstalk marks another important step in our commitment to increase the flexibility for AWS customers to use the choice of operating system, programming language, development tools and database software that meet their application requirements.
Do You Want to Help Build the Next AWS Service?
Over the past several years I?ve spent much of my time traveling around the world speaking about distributed systems. From building infinitely scalable data stores, architectures for high performance computing, to the challenges imposed by the CAP theorem, there are wonderful, complex, fascinating problems to be solved in the area of distributed computing.
Expanding the Cloud Introducing AWS Marketplace
Today Amazon Web Services launched AWS Marketplace, an online store that makes it easy for you to find, buy, and immediately start using software and services that run on the AWS Cloud. You can use AWS Marketplaces 1-Click deployment to quickly launch pre-configured software on your own Amazon EC2 instances and pay only for what you use, by the hour or month. AWS handles billing and payments, and software charges appear on your AWS bill. Marketplace has software listings from well-known vendors including 10gen, CA, Canonical, Couchbase, Check Point Software, IBM, Microsoft, SAP, Zend, and others, as well as many widely used open source offerings including Wordpress, Drupal, and MediaWiki.
Do You Want to Help Build the Next AWS Service?
Over the past several years Ive spent much of my time traveling around the world speaking about distributed systems. From building infinitely scalable data stores, architectures for high performance computing, to the challenges imposed by the CAP theorem, there are wonderful, complex, fascinating problems to be solved in the area of distributed computing. During my travels Ive met thousands of brilliant engineers who are leveraging the cloud to deliver exciting new products and revolutionize IT as we know it. One thing thats become obvious to me is that there are innovative, inspiring developers in every corner of the planet from Australia to Iceland and from Israel to Peru.
Expanding the Cloud ? Introducing AWS Marketplace
Today Amazon Web Services launched AWS Marketplace, an online store that makes it easy for you to find, buy, and immediately start using software and services that run on the AWS Cloud. You can use AWS Marketplace?s 1-Click deployment to quickly launch pre-configured software on your own Amazon EC2 instances and pay only for what you use, by the hour or month.
Expanding the Cloud Introducing Amazon CloudSearch
Today Amazon Web Services is introducing Amazon CloudSearch, a new web service that brings the power of the Amazon.coms search technology to every developer. Amazon CloudSearch provides a fully-featured search engine that is easy to manage and scale. It offers full-text search with features like faceting and user-defined rank functions. And like most AWS services, Amazon CloudSearch scales automatically as your data and traffic grow, making it an easy choice for applications small to large. With Amazon CloudSearch, developers just create a Search Domain, upload data, and start querying. Why Search? Search is an essential part of many of today's cloud-centric applications.
Expanding the Cloud ? Introducing Amazon CloudSearch
Today Amazon Web Services is introducing Amazon CloudSearch, a new web service that brings the power of the Amazon.com?s search technology to every developer. Amazon CloudSearch provides a fully-featured search engine that is easy to manage and scale. It offers full-text search with features like faceting and user-defined rank functions. And like most AWS services, Amazon CloudSearch scales automatically as your data and traffic grow, making it an easy choice for applications small to large.
Customer Conversations - How Intuit and Edmodo Innovate using Amazon RDS
From tax preparation to safe social networks, Amazon RDS brings new and innovative applications to the cloud Empowering innovation is at the heart of everything we do at Amazon Web Services (AWS). I often get to meet, discuss, and learn from innovators how they are using AWS to deliver transformative applications to their users, customers and partners. Often we think about innovation as doing 'new things' or based on revolutionary new technologies such as DynamoDB, but it is more important to ensure that one can also innovate based on existing paradigms. One of the services that is very successful in driving innovation at our customers in this context is Amazon RDS, the Relational Database Service.
Customer Conversations - How Intuit and Edmodo Innovate using Amazon RDS
From tax preparation to safe social networks, Amazon RDS brings new and innovative applications to the cloud Empowering innovation is at the heart of everything we do at Amazon Web Services (AWS). I often get to meet, discuss, and learn from innovators how they are using AWS to deliver transformative applications to their users, customers and partners.
A Thousand Platforms ...
Todays AWS Elastic Beanstalk announcement of PHP and Git support reminded me of the post where I mentioned that we want to let a thousand platforms bloom on AWS. Some might ask why AWS would want a thousand platforms. One of the most important AWS principles is flexibility. Flexibility is in the choice of software and languages running on AWS, in the tools and interfaces available to manipulate resources and applications, and in the ability to leverage services from other providers. One of our customers I met last week was talking about his application and how it runs on AWS; He collects geo-location data, analyzes and crunches this data using Elastic Map Reduce, stores the data for quick access in DynamoDB, runs his user interface on Heroku and his web services layer for mobile devices on Elastic Beanstalk.
A Thousand Platforms ...
Today?s AWS Elastic Beanstalk announcement of PHP and Git support reminded me of the post where I mentioned that we want to let a thousand platforms bloom on AWS. Some might ask why AWS would want a thousand platforms. One of the most important AWS principles is flexibility. Flexibility is in the choice of software and languages running on AWS, in the tools and interfaces available to manipulate resources and applications, and in the ability to leverage services from other providers.
Driving Compute Cost Down for AWS Customers
AWS today announced a substantial price drop from March 1, 2012 for many of the Amazon EC2, Amazon RDS, and Amazon ElastiCache instances types around the world. For example, the popular m1.small instance type will see a price drop of 6% for EC2 On-Demand usage and 33% for EC2 Reserved Instance usage.
Driving Compute Cost Down for AWS Customers
AWS today announced a substantial price drop from March 1, 2012 for many of the Amazon EC2, Amazon RDS, and Amazon ElastiCache instances types around the world. For example, the popular m1.small instance type will see a price drop of 6% for EC2 On-Demand usage and 33% for EC2 Reserved Instance usage. Some of the other instance types have even greater savings: for example, the high memory M2 instances will see a 10% price cut for On Demand and 37% for Reserved instances. Similarly, Amazon RDS will cut its On-Demand prices by up to 10% and Reserved Instance prices by up to 42%.
Expanding the Cloud The Amazon Simple Workflow Service
Today AWS launched an exciting new service for developers: the Amazon Simple Workflow Service. Amazon SWF is an orchestration service for building scalable distributed applications. Often an application consists of several different tasks to be performed in particular sequence driven by a set of dynamic conditions. Amazon SWF makes it very easy for developers to architect and implement these tasks, run them in the cloud or on premise and coordinate their flow. Amazon SWF manages the execution flow such that the tasks are load balanced across the registered workers, that inter-task dependencies are respected, that concurrency is handled appropriately and that child workflows are executed.
Expanding the Cloud ? The Amazon Simple Workflow Service
Today AWS launched an exciting new service for developers: the Amazon Simple Workflow Service. Amazon SWF is an orchestration service for building scalable distributed applications. Often an application consists of several different tasks to be performed in particular sequence driven by a set of dynamic conditions. Amazon SWF makes it very easy for developers to architect and implement these tasks, run them in the cloud or on premise and coordinate their flow.
Driving Storage Costs Down for AWS Customers
One of the things that differentiates Amazon Web Services from other technology providers is its commitment to let customers benefits from continuous cost-cutting innovations and from the economies of scale AWS is able to achieve. As we showed last week one of the services that is growing rapidly is the Amazon Simple Storage Service (S3). AWS today announced a substantial price drop per February 1, 2012 for Amazon S3 standard storage to help customers drive their storage cost down. A customer storing 50TB will see on average a 12% drop in cost when they get their Amazon S3 bill for February.
Driving Storage Costs Down for AWS Customers
One of the things that differentiates Amazon Web Services from other technology providers is its commitment to let customers benefits from continuous cost-cutting innovations and from the economies of scale AWS is able to achieve. As we showed last week one of the services that is growing rapidly is the Amazon Simple Storage Service (S3).
Expanding the Cloud - The AWS Storage Gateway
Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customers on-premises applications. We have been working closely with our customers on their requests to bring the power of the Amazon Web Services cloud closer to their existing on-premises compute infrastructures. The Amazon Virtual Private Cloud extends on-premises compute with all the power of AWS, making it elastic, scalable and highly reliable. AWS Identity and Access Management brings together on-premises and cloud identity management. VM Import allows our customers to move virtual machine images from their datacenters to the Cloud and Amazon Direct Connect makes the network latencies and bandwidth between on-premises and AWS more predictable.
Expanding the Cloud - The AWS Storage Gateway
Today Amazon Web Services has launched the AWS Storage Gateway, making the power of secure and reliable cloud storage accessible from customers? on-premises applications. We have been working closely with our customers on their requests to bring the power of the Amazon Web Services cloud closer to their existing on-premises compute infrastructures.
Amazon DynamoDB a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications
Today is a very exciting day as we release Amazon DynamoDB, a fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications. DynamoDB is the result of 15 years of learning in the areas of large scale non-relational databases and cloud services. Several years ago we published a paper on the details of Amazons Dynamo technology, which was one of the first non-relational databases developed at Amazon. The original Dynamo design was based on a core set of strong distributed systems principles resulting in an ultra-scalable and highly reliable database system. Amazon DynamoDB, which is a new service, continues to build on these principles, and also builds on our years of experience with running non-relational databases and cloud services, such as Amazon SimpleDB and Amazon S3, at scale.
Amazon DynamoDB ? a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications
Today is a very exciting day as we release Amazon DynamoDB, a fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications. DynamoDB is the result of 15 years of learning in the areas of large scale non-relational databases and cloud services. Several years ago we published a paper on the details of Amazon?s Dynamo technology, which was one of the first non-relational databases developed at Amazon.
Countdown to What is Next in AWS
Join me at 9AM PST on Wednesday January 18, 2012 to find out what is next in the AWS Cloud. Registration required. Watch live streaming video from AWSCloudEvent at livestream.com
Countdown to What is Next in AWS
Join me at 9AM PST on Wednesday January 18, 2012 to find out what is next in the AWS Cloud. Registration required.
Expanding the Cloud Introducing the AWS South America (Sao Paulo) Region
Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS Region in Sao Paulo, Brazil. This new Region has been highly requested by companies worldwide, and it provides low-latency access to AWS services for those who target customers in South America. South America is one of the fastest growing economic regions in the world. In particular, South American IT-oriented companies are seeing very rapid growth. Case in point: over the past 10 years IT has risen to become 7% of the GDP in Brazil. With the launch of the South America (Sao Paolo) Region, AWS now provides companies large and small with infrastructure that allows them to get to market faster while reducing their costs which enables them to focus on delivering value, instead of wasting time on non-differentiating tasks.
Expanding the Cloud ? Introducing the AWS South America (Sao Paulo) Region
Today, Amazon Web Services is expanding its worldwide coverage with the launch of a new AWS Region in Sao Paulo, Brazil. This new Region has been highly requested by companies worldwide, and it provides low-latency access to AWS services for those who target customers in South America. South America is one of the fastest growing economic regions in the world.
Expanding the Cloud - Introducing Amazon ElastiCache
Today AWS has launched Amazon ElastiCache, a new service that makes it easy to add distributed in-memory caching to any application. Amazon ElastiCache handles the complexity of creating, scaling and managing an in-memory cache to free up brainpower for more differentiating activities. There are many success stories about the effectiveness of caching in many different scenarios; next to helping applications achieving fast and predictable performance, it often protects databases from requests bursts and brownouts under overload conditions. Systems that make extensive use of caching almost all report a significant reduction in the cost of their database tier. Given the widespread use of caching in many of the applications in the AWS Cloud, a caching service had been high on the request list of our customers.
Expanding the Cloud - Introducing Amazon ElastiCache
Today AWS has launched Amazon ElastiCache, a new service that makes it easy to add distributed in-memory caching to any application. Amazon ElastiCache handles the complexity of creating, scaling and managing an in-memory cache to free up brainpower for more differentiating activities. There are many success stories about the effectiveness of caching in many different scenarios; next to helping applications achieving fast and predictable performance, it often protects databases from requests bursts and brownouts under overload conditions.
Job Openings in AWS - Senior Leader in Database Services
There are some great job openings within Amazon Web Services. I will try to highlight some of those in coming weeks. This week it is an opening for senior leaders with AWS Database Services. AWS Database Services is responsible for setting the database strategy and delivering distributed structured storage services to our AWS customers. This team is constantly rethinking the assumptions behind how traditional databases were built and constantly working on building the right database architectures suited for the Cloud environment. The database services organization is looking for senior leaders who will be able to hire and lead a large software development team that is responsible for designing and running services that are at the cutting edge of distributed database technology that helps our customers to build scalable database-driven applications in the cloud and have a significant bottom-line impact on our business.
Job Openings in AWS - Senior Leader in Database Services
There are some great job openings within Amazon Web Services. I will try to highlight some of those in coming weeks. This week it is an opening for senior leaders with AWS Database Services. AWS Database Services is responsible for setting the database strategy and delivering distributed structured storage services to our AWS customers.
Driving down the cost of Big-Data analytics
The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Many of our Big-Data customers already saw a big drop in their AWS bill last month when the cost of incoming bandwidth was dropped to $0.
Driving down the cost of Big-Data analytics
The Amazon Elastic MapReduce (EMR) team announced today the ability to seamlessly use Amazon EC2 Spot Instances with their service, significantly driving down the cost of data analytics in the cloud. Many of our Big-Data customers already saw a big drop in their AWS bill last month when the cost of incoming bandwidth was dropped to $0.00. Now, given that historically customers using Spot Instances have seen cost saving up to 66% over On-Demand Instance prices, Amazon EMR customers are poised to achieve even greater cost savings. Analyzing vast amounts of data is critical for companies looking to incorporate customer insights into their business, including building recommendation engines or optimizing customer targeting.
No Server Required - Jekyll & Amazon S3
As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. If you have a largely static site you can rely on the enormous power of S3 to make serving your content highly scalable and storing it extremely durable.
No Server Required - Jekyll & Amazon S3
As some of you may remember I was pretty excited when Amazon Simple Storage Service (S3) released its website feature such that I could serve this weblog completely from S3. If you have a largely static site you can rely on the enormous power of S3 to make serving your content highly scalable and storing it extremely durable. Amazon S3 is much more than just storage; the network and distributed systems infrastructure to ensure that content can be served fast and at high rates without customers impacting each other, is amazing. Just dropping your website in an S3 bucket brings all that power to you.
Expanding the Cloud - The AWS GovCloud (US) Region
Today AWS announced the launch of the AWS GovCloud (US) Region. This new region, which is located on the West Coast of the US, helps US government agencies and contractors move more of their workloads to the cloud by implementing a number of US government-specific regulatory requirements. The concept of regions gives AWS customers control over the placement of their resources and services. Next to GovCloud (US) there are five general purpose regions; two in the US (one on the west coast and one on the east coast), one in the EU (in Ireland) and two in APAC (in Singapore and Tokyo).
Expanding the Cloud - The AWS GovCloud (US) Region
Today AWS announced the launch of the AWS GovCloud (US) Region. This new region, which is located on the West Coast of the US, helps US government agencies and contractors move more of their workloads to the cloud by implementing a number of US government-specific regulatory requirements. The concept of regions gives AWS customers control over the placement of their resources and services.
Spot Instances - Increased Control
Today we announced the launch of an exciting new feature that will significantly increase your control over your Amazon EC2 Spot instances. With this change, we will improve the granularity of pricing information you receive by introducing a Spot Instance price per Availability Zone rather than a Spot Instance price per Region. Spot Instances enable you to bid on unused Amazon EC2 capacity. Customers whose bids exceed the Spot price gain access to the available Spot Instances and run as long as the bid exceeds the Spot Price. Spot Instances are ideal for use cases like web and data crawling, financial analysis, grid computing, media transcoding, scientific research, and batch processing.
Spot Instances - Increased Control
Today we announced the launch of an exciting new feature that will significantly increase your control over your Amazon EC2 Spot instances. With this change, we will improve the granularity of pricing information you receive by introducing a Spot Instance price per Availability Zone rather than a Spot Instance price per Region.
Expanding the Cloud - AWS Import/Export Support for Amazon EBS
The AWS Import/Export team has announced today that they have expanded their functionality significantly by adding Import into Amazon EBS. AWS Import/Export transfers data off of storage devices using Amazon's high-speed internal network and bypassing the Internet. With this new functionality AWS Import/Export now supports importing data directly into Amazon EBS snapshots. Once loaded into an Amazon EBS snapshot, The customer can create a volume based on that snapshot and attach it to an Amazon EC2 instance, or they can share that snapshot with others. Amazon Import/Export is an important tool for customers to accelerate moving large amounts of data into the AWS storage systems.
Expanding the Cloud - AWS Import/Export Support for Amazon EBS
The AWS Import/Export team has announced today that they have expanded their functionality significantly by adding Import into Amazon EBS. AWS Import/Export transfers data off of storage devices using Amazon's high-speed internal network and bypassing the Internet. With this new functionality AWS Import/Export now supports importing data directly into Amazon EBS snapshots.
APAC Summer Tour
I have just landed in Tokyo for what will be a month long tour visiting our customers in the Asia Pacific Region. Next to customer visits I will take part in a number of events organized by AWS and by our partners. This week in Japan there are three public events planned: July 4 - AWS HPC Night at Fuji Soft Hall in Akihabara. Next to a presentation by me about HPC on AWS, there is a panel with Japanese HPC experts moderated by Dr Kazuyuki Shudo of Titec. July 5 - Cloud IT Leadership seminar on Business Continuity in partnership with Deloitte.
APAC Summer Tour
I have just landed in Tokyo for what will be a month long tour visiting our customers in the Asia Pacific Region. Next to customer visits I will take part in a number of events organized by AWS and by our partners. This week in Japan there are three public events planned: July 4 -AWS HPC Night at Fuji Soft Hall in Akihabara.
Driving Bandwidth Cost Down for AWS Customers.
Often we think about innovation as going after new unchartered territories, but it is also important to innovate in those existing dimensions that will remain important for customers. For Amazon retail, some of those dimensions are low pricing, large catalog, fast shipping, and convenience. Every effort we put into improving these drives a flywheel that yields benefits both immediately and for the long-term. For example, when our retail customers contributed to create larger economies of scale for Amazon.com, we used the savings to lower pricing such that our customers could also benefit. In Amazon Web Services there are similar dimensions that are forever important to our customers; scale, reliability, security, performance, ease of use, and of course pricing.
Driving Bandwidth Cost Down for AWS Customers.
Often we think about innovation as going after new unchartered territories, but it is also important to innovate in those existing dimensions that will remain important for customers. For Amazon retail, some of those dimensions are low pricing, large catalog, fast shipping, and convenience. Every effort we put into improving these drives a flywheel that yields benefits both immediately and for the long-term.
New Route 53 and ELB features: IPv6, Zone Apex, WRR and more
An important contribution to the success of the Amazon Web Services is the willingness to listen closely to our customers and to use this feedback to drive the feature roadmap of a service. I am excited that today both the Route 53, the highly available and scalable DNS service, and the Elastic Load Balancing teams are releasing new functionality that has been frequently requested by their customers: Route 53 now GA: Route 53 is now Generally Available and will provide an availability SLA of 100%. See the Route 53 detail page for the exact definitions and how SLA violations will be handled.
New Route 53 and ELB features: IPv6, Zone Apex, WRR and more
An important contribution to the success of the Amazon Web Services is the willingness to listen closely to our customers and to use this feedback to drive the feature roadmap of a service. I am excited that today both the Route 53, the highly available and scalable DNS service, and the Elastic Load Balancing teams are releasing new functionality that has been frequently requested by their customers: Route 53 now GA: Route 53 is now Generally Available and will provide an availability SLA of 100%.
The Amazon.com 2010 Shareholder Letter Focusses on Technology
In the 2010 Shareholder Letter Jeff Bezos writes about the unique technologies developed at Amazon.com over the years. Given that I have frquently written about many of these technologies on this blog I asked investor relations to be allowed to reprint it here. You can find the original here. To our shareowners: Random forests, naïve Bayesian estimators, RESTful services, gossip protocols, eventual consistency, data sharding, anti-entropy, Byzantine quorum, erasure coding, vector clocks ... walk into certain Amazon meetings, and you may momentarily think you've stumbled into a computer science lecture. Look inside a current textbook on software architecture, and you'll find few patterns that we don't apply at Amazon.
The Amazon.com 2010 Shareholder Letter Focusses on Technology
In the 2010 Shareholder Letter Jeff Bezos writes about the unique technologies developed at Amazon.com over the years. Given that I have frequently written about many of these technologies on this blog I asked investor relations to be allowed to reprint it here. You can find the original here. To our shareowners: Random forests, naïve Bayesian estimators, RESTful services, gossip protocols, eventual consistency, data sharding, anti-entropy, Byzantine quorum, erasure coding, vector clocks .
Mashing Up Science - The Mendeley API Binary Battle
Two years ago when I was first introduced to Mendeley I concluded that if they executed right they could "change the face of science". Mendeley is a free reference manager and academic social network that can help you organize your research, collaborate with others online, and discover the latest research. Turn the clock forward two years and Mendeley has now evolved into the world's largest crowdsourced research database, with 70 million documents, usage statistics and reader demographics, social tags, and related research recommendations. This database is available under a creative commons license. To see what innovation can happen when this data is freely available Mendeley has issued a challenge dubbed "The Mendeley API Binary Battle" - build an application that mashes up the Mendeley data and you can win $10,001!
Mashing Up Science - The Mendeley API Binary Battle
Two years ago when I was first introduced to Mendeley I concluded that if they executed right they could "change the face of science". Mendeley is a free reference manager and academic social network that can help you organize your research, collaborate with others online, and discover the latest research.
Music to my Ears - Introducing Amazon Cloud Drive
Today Amazon.com announced new solutions to help customers manage their digital music collections. Amazon Cloud Drive and Amazon Cloud Player enable customers to securely and reliably store music in the cloud and play it on any Android phone, tablet, Mac or PC, wherever they are. As a big music fan with well over 100Gb in digital music I am particularly excited that I now have access to all my digital music anywhere I go. Order in the Chaos The number of digital objects in our lives is growing rapidly. What used to be only available in physical formats now often has digital equivalents and this digitalization is driving great new innovations.
Music to my Ears - Introducing Amazon Cloud Drive and Amazon Cloud Player
Today Amazon.com announced new solutions to help customers manage their digital music collections. Amazon Cloud Drive and Amazon Cloud Player enable customers to securely and reliably store music in the cloud and play it on any Android phone, tablet, Mac or PC, wherever they are. As a big music fan with well over 100Gb in digital music I am particularly excited that I now have access to all my digital music anywhere I go.
Hacking with AWS at The Next Web Hackaton
Over the past years The Next Web Conference has become a premier conference on internet life and its technologies. I have been to the conference almost every year and it is getting better every time. Amsterdam is of course the ideal place for such a conference :-). The TNW team is doing a great job in getting an excellent program together that draws an audience from around the world, not just Europe, and there is an interesting mix of startups, enterprises, investors and media attending. The images from the 2008 TNW Conference have travelled around the world in my Animoto demo: This year TNW is showing that it is not just a conference for talkers but also for builders by organizing a massive Hackaton in the two days running up to the conference.
Hacking with AWS at The Next Web Hackaton
Over the past years The Next Web Conference has become a premier conference on internet life and its technologies. I have been to the conference almost every year and it is getting better every time. Amsterdam is of course the ideal place for such a conference :-). The TNW team is doing a great job in getting an excellent program together that draws an audience from around the world, not just Europe, and there is an interesting mix of startups, enterprises, investors and media attending.
Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo) Region
Today Amazon Web Services is expanding its world-wide coverage with the launch of a new AWS Region located in Tokyo, Japan. Japanese companies and consumers have become used to low latency and high-speed networking available between their businesses, residences, and mobile devices. With the launch of the Asia Pacific (Tokyo) Region, companies can now leverage the AWS suite of infrastructure web services directly connected to Japanese networks. The advanced Asia Pacific network infrastructure also makes the AWS Tokyo Region a viable option for customers from South Korea. A well know application area that makes use of the advanced network infrastructure in Japan is social gaming, and it shouldn't be surprising that some of the early customers of the new AWS Tokyo Region are large gaming companies like Gumi and Zynga Japan.
Expanding the Cloud - Introducing the AWS Asia Pacific (Tokyo) Region
Today Amazon Web Services is expanding its world-wide coverage with the launch of a new AWS Region located in Tokyo, Japan. Japanese companies and consumers have become used to low latency and high-speed networking available between their businesses, residences, and mobile devices. With the launch of the Asia Pacific (Tokyo) Region, companies can now leverage the AWS suite of infrastructure web services directly connected to Japanese networks.
From the Archives - Gapingvoid's Nobody Cares
While cleaning out the digital attic I ran into this drawing that Hugh MacLeod (aka "gapingvoid") made for me in reponse to a storm-in-a-teacup about blogging Amazon. As usual Hugh came straight to the heart of the matter :-) BTW Hugh's new book "Evil Plans, having Fun on the Road to World Domination"
From the Archives - Gapingvoid's Nobody Cares
While cleaning out the digital attic I ran into this drawing that Hugh MacLeod (aka "gapingvoid") made for me in reponse to a storm-in-a-teacup about blogging Amazon. As usual Hugh came straight to the heart of the matter :-) BTW Hugh's new book "Evil Plans, having Fun on the Road to World Domination" was released last week. You can order it here
Simplifying IT - Create Your Application with AWS CloudFormation
With the launch of AWS CloudFormation today another important step has been taken in making it easier for customers to deploy applications to the cloud. Often an application requires several infrastructure resources to be created and AWS CloudFormation helps customers create and manage these collections of AWS resources in a simple and predictable way. Using declarative Templates customers can create Stacks of resources ensuring that all resources have been created, in the right sequence and with the correct confirmation. Earlier this year I met with an ISV partner who transformed his on-premise ERP software into a software-as-a-service offering. They had taken the approach that they would not only be offering their software as a scalable multi-tenant product but also as a single tenant environment for customers that want to have their own isolated environment.
Simplifying IT - Create Your Application with AWS CloudFormation
With the launch of AWS CloudFormation today another important step has been taken in making it easier for customers to deploy applications to the cloud. Often an application requires several infrastructure resources to be created and AWS CloudFormation helps customers create and manage these collections of AWS resources in a simple and predictable way.
Free at Last - A Fully Self-Sustained Blog Running in Amazon S3
In a follow up to the last blog post I have removed the last two dependencies this blog had on running a server somewhere: comments are now served by Disqus and search is now handled by Bing. I should have moved to Disqus long time again as it is such rich discussion environment. It imported the commented from my Moveable Type server without a hitch. The choice for the search box from Bing was driven by that it was very easy to setup and it was free, where Google Site Search asked for $100/year. I'll evaluate search again in a few weeks, but for the moment this is good enough.
Free at Last - A Fully Self-Sustained Blog Running in Amazon S3
In a follow up to the last blog post I have removed the last two dependencies this blog had on running a server somewhere: comments are now served by Disqus and search is now handled by Bing. I should have moved to Disqus long time again as it is such rich discussion environment.
New AWS feature: Run your website from Amazon S3
Since a few days ago this weblog serves 100% of its content directly out of the Amazon Simple Storage Service (S3) without the need for a web server to be involved. Because my blog is almost completely static content I wanted to run in this very simple configuration since the launch of Amazon S3. It would allow the blog to be powered by the incredible scale and reliability of Amazon S3 with a minimum of effort from my side. I know of several other customers who had asked for this greatly simplifying feature as well. I had held out implementing an alternative to my simple blog server that had been running at a traditional hosting site for many years until this preferred simple solution became available: today marks that day and I couldn't be happier about it.
New AWS feature: Run your website from Amazon S3
Since a few days ago this weblog serves 100% of its content directly out of the Amazon Simple Storage Service (S3) without the need for a web server to be involved. Because my blog is almost completely static content I wanted to run in this very simple configuration since the launch of Amazon S3.
It is not the critic who counts ...
When Steve Case was asked "how do you turn defeat in to failure" he gracefully quoted Teddy Roosevelt. It is a great speech that will probably inspire people forever. I believe the sentiment of that speech is even more important today. We are enjoying a tremendous rise in entrepreneurial activities and the many startups in technology and in the consumer internet are driving hard to innovate and get their product in the hands of customers at neck breaking speeds. At AWS we are proud to power many of these companies and in my personal interactions I see a whole new generation of technology leaders arising who will have impact for a long time to come.
It is not the critic who counts ...
When Steve Case was asked "how do you turn defeat in to victory" he gracefully quoted Teddy Roosevelt. It is a great speech that will probably inspire people forever. I believe the sentiment of that speech is even more important today. We are enjoying a tremendous rise in entrepreneurial activities and the many startups in technology and in the consumer internet are driving hard to innovate and get their product in the hands of customers at neck breaking speeds.
AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud
Flexibility is one of the key principles of Amazon Web Services - developers can select any programming language and software package, any operating system, any middleware and any database to build systems and applications that meet their requirements. Additionally customers are not restricted to AWS services; they can mix-and-match services from other providers to best meet their needs. A whole range of innovative new services, ranging from media conversion to geo-location-context services have been developed by our customers using this flexibility and are available in the AWS ecosystem. To enable this broad choice, the core of AWS is composed of building blocks which customers and partners can use to build any system or application in the way they see fit.
AWS Elastic Beanstalk: A Quick and Simple Way into the Cloud
Flexibility is one of the key principles of Amazon Web Services - developers can select any programming language and software package, any operating system, any middleware and any database to build systems and applications that meet their requirements. Additionally customers are not restricted to AWS services; they can mix-and-match services from other providers to best meet their needs.
DROAM - Dreaming about Cheap Data Roaming
I frequently travel outside of the US. Often to Europe and increasingly to the Middle & Far East and Australia. The one thing that I have always struggled with during my travels are the data plans of the cell phone companies. They are complex and ridiculously expensive. For an internet road warrior they are a complete nightmare. One wireless company for example has an international plan that will charge you $25 per month for 50MB after which they will charge you $20 per MB. I frequently do more than 30MB a day, so this plan has no advantage for me. After many investigations I settled for T-Mobile which has an international Blackberry supplement that is $20/month for all BB mail traffic.
DROAM - Dreaming about Cheap Data Roaming
I frequently travel outside of the US. Often to Europe and increasingly to the Middle & Far East and Australia. The one thing that I have always struggled with during my travels are the data plans of the cell phone companies. They are complex and ridiculously expensive. For an internet road warrior they are a complete nightmare.
Big Just Got Bigger - 5 Terabyte Object Support in Amazon S3
Today, Amazon S3 announced a new breakthrough in supporting customers with large files by increasing the maximum supported object size from 5 gigabytes to 5 terabytes. This allows customers to store and reference a large file as a single object instead of smaller 'chunks'. When combined with the Amazon S3 Multipart Upload release, this dramatically improves how customers upload, store and share large files on Amazon S3. Who has files larger than 5GB? Amazon S3 has always been a scalable, durable and available data repository for almost any customer workload. However, as use of the cloud as grown, so have the file sizes customers want to store in Amazon S3 as objects.
Big Just Got Bigger - 5 Terabyte Object Support in Amazon S3
Today, Amazon S3 announced a new breakthrough in supporting customers with large files by increasing the maximum supported object size from 5 gigabytes to 5 terabytes. This allows customers to store and reference a large file as a single object instead of smaller 'chunks'. When combined with the Amazon S3 Multipart Upload release, this dramatically improves how customers upload, store and share large files on Amazon S3.
Expanding the Cloud with DNS - Introducing Amazon Route 53
I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. DNS is one of the fundamental building blocks of internet applications and was high on the wish list of our customers for some time already. Route 53 has the business properties that you have come to expect from an AWS service: fully self-service and programmable, with transparent pay-as-you-go pricing and no minimum usage commitments. Some fundamentals on Naming Naming is one of the fundamental concepts in Distributed Systems. Entities in a system are identified through their name, which is separate from the way that you would choose to access that entity, the address that the access point resides at and what route to take to get to that address.
Expanding the Cloud with DNS - Introducing Amazon Route 53
I am very excited that today we have launched Amazon Route 53, a high-performance and highly-available Domain Name System (DNS) service. DNS is one of the fundamental building blocks of internet applications and was high on the wish list of our customers for some time already. Route 53 has the business properties that you have come to expect from an AWS service: fully self-service and programmable, with transparent pay-as-you-go pricing and no minimum usage commitments.
This week in review: GPUs, Zombies, Biomimicry and Tom Waits
Here are some the links I shared this week on twitter and facebook: Cloud Computing Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. There were blog posts by Jeff Barr The Cluster GPU Instance and James Hamilton HPC in the Cloud with GPGPUs, as well as my background posting: Expanding the Cloud - Adding the Incredible Power of the Amazon EC2 Cluster GPU Instances The Cluster Compute and the Cluster GPU Instances are now supported by Amazon Elastic Map Reduce. There was an excellent first benchmarking report of the Cluster GPU Instances by the folks at Cycle Computing - "A Couple More Nails in the Coffin of the Private Compute Cluster" The Top500 supercomputer list for November 2010 was released and an Amazon EC2 Cluster Compute Instance based cluster came in at #231 The Amazon Web Services received ISO 27001 certification ...
This week in review: GPUs, Zombies, Biomimicry and Tom Waits
Here are some the links I shared this week on twitter and facebook: Cloud Computing Big news this week was of course the launch of Cluster GPU instances for Amazon EC2. There were blog posts by Jeff Barr The Cluster GPU Instance and James Hamilton HPC in the Cloud with GPGPUs, as well as my background posting: Expanding the Cloud - Adding the Incredible Power of the Amazon EC2 Cluster GPU Instances
Expanding the Cloud - Adding the Incredible Power of the Amazon EC2 Cluster GPU Instances
Today Amazon Web Services takes another step on the continuous innovation path by announcing a new Amazon EC2 instance type: The Cluster GPU Instance. Based on the Cluster Compute instance type, the Cluster GPU instance adds two NVIDIA Telsa M2050 GPUs offering GPU-based computational power of over one TeraFLOPS per instance. This incredible power is available for anyone to use in the usual pay-as-you-go model, removing the investment barrier that has kept many organizations from adopting GPUs for their workloads even though they knew there would be significant performance benefit. From financial processing and traditional oil & gas exploration HPC applications to integrating complex 3D graphics into online and mobile applications, the applications of GPU processing appear to be limitless.
Expanding the Cloud - Adding the Incredible Power of the Amazon EC2 Cluster GPU Instances
Today Amazon Web Services takes another step on the continuous innovation path by announcing a new Amazon EC2 instance type: The Cluster GPU Instance. Based on the Cluster Compute instance type, the Cluster GPU instance adds two NVIDIA Telsa M2050 GPUs offering GPU-based computational power of over one TeraFLOPS per instance.
Around the World in 28 Days
On Monday I will leave Seattle for 4 weeks of meeting existing and future customers of the Amazon Web Services. With existing customers I get a change to dive deep on their AWS usage and understand what works well and where we can do better. Visiting future customers is equally exiting as you get a change to understand their current architecture, if it is a migration, and how they plan to exploit cloud services in their new setup. There is huge variety in exiting architectures and I am often impressed about the ingenuity of the engineers in how to best transform the application if "Lift & Shift" is not an option.
Around the World in 28 Days
On Monday I will leave Seattle for 4 weeks of meeting existing and future customers of the Amazon Web Services. With existing customers I get a change to dive deep on their AWS usage and understand what works well and where we can do better. Visiting future customers is equally exiting as you get a change to understand their current architecture, if it is a migration, and how they plan to exploit cloud services in their new setup.
Reboot
Like many folks who started down the path of using real-time micro-blogging services (read: twitter) the convenience of those platforms has made that sharing happens there instead of on the blogs we (used to) keep. I fully intended to stay committed to using this blog for high-quality, relevant sharing of thoughts but the reality has turned out to be different. I did write for those occasions where I thought additional background on AWS service and feature launches was relevant, but beyond that not much materialized. That was a mistake which I will try to correct in the coming months as I'll get back into blogging mode.
Reboot
Like many folks who started down the path of using real-time micro-blogging services (read: twitter) the convenience of those platforms has made that sharing happens there instead of on the blogs we (used to) keep. I fully intended to stay committed to using this blog for high-quality, relevant sharing of thoughts but the reality has turned out to be different.
Expanding the Cloud - Cluster Compute Instances for Amazon EC2
Today, Amazon Web Services took very an important step in unlocking the advantages of cloud computing for a very important application area. Cluster Computer Instances for Amazon EC2 are a new instance type specifically designed for High Performance Computing applications. Customers with complex computational workloads such as tightly coupled, parallel processes, or with applications that are very sensitive to network performance, can now achieve the same high compute and networking performance provided by custom-built infrastructure while benefiting from the elasticity, flexibility and cost advantages of Amazon EC2 During my academic career, I spent many years working on HPC technologies such as user-level networking interfaces, large scale high-speed interconnects, HPC software stacks, etc.
Expanding the Cloud - Cluster Compute Instances for Amazon EC2
Today, Amazon Web Services took very an important step in unlocking the advantages of cloud computing for a very important application area. Cluster Computer Instances for Amazon EC2 are a new instance type specifically designed for High Performance Computing applications.
Expanding the Cloud - Amazon S3 Reduced Redundancy Storage
Today a new storage option for Amazon S3 has been launched: Amazon S3 Reduced Redundancy Storage (RRS). This new storage option enables customers to reduce their costs by storing non-critical, reproducible data at lower levels of redundancy. This has been an option that customers have been asking us about for some time so we are really pleased to be able to offer this alternative storage option now. Durability in Amazon S3 Amazon Simple Storage Service (S3) was launched in 2006 as "Storage for the Internet" with the promise to make web-scale computing easier for developers. Four years later it stores over 100 billion objects and routinely performs well over 120,000 storage operations per second.
Expanding the Cloud - Amazon S3 Reduced Redundancy Storage
Today a new storage option for Amazon S3 has been launched: Amazon S3 Reduced Redundancy Storage (RRS). This new storage option enables customers to reduce their costs by storing non-critical, reproducible data at lower levels of redundancy. This has been an option that customers have been asking us about for some time so we are really pleased to be able to offer this alternative storage option now.
Expanding the Cloud - Opening the AWS Asia Pacific (Singapore) Region
Today Amazon Web Services has taken another important step in serving customers worldwide: the AWS Asia Pacific (Singapore) Region is now launched. Customers can now store their data and run their applications from our Singapore location in the same way they do from our other U.S. and European Regions. The importance of Regions Quite often "The Cloud" is portrayed as something magically transparent that lives somewhere in the internet. This portrayal can be a desirable and useful abstraction when discussing cloud services at the application and end-user level. However, when speaking about cloud services in terms of Infrastructure-as-a-Service, it is very important to make the geographic locations of services more explicit.
Expanding the Cloud - Opening the AWS Asia Pacific (Singapore) Region
Today Amazon Web Services has taken another important step in serving customers worldwide: the AWS Asia Pacific (Singapore) Region is now launched. Customers can now store their data and run their applications from our Singapore location in the same way they do from our other U.S. and European Regions. The importance of Regions Quite often "
I am looking for new application and platform services
The ecosystem of new application and platform services in the cloud is the future of application development. It will drive rapid innovation and we'll see a wealth of mobile, web and desktop applications arrive that we couldn't dream about a few years ago, and these building blocks are the enablers of that. These services will be delivered not only by new startups but also by enterprises looking to capitalize on their IP. As examples of such services I always use Twillio (voice &sms) and Simplegeo (location), but it is time to start building out my knowledge of all the different services that are in the ecosystem.
I am looking for new application and platform services
The ecosystem of new application and platform services in the cloud is the future of application development. It will drive rapid innovation and we'll see a wealth of mobile, web and desktop applications arrive that we couldn't dream about a few years ago, and these building blocks are the enablers of that.
Amazon Cloudfront is Streaming Media 2010 Editor's pick
I am excited that Amazon Cloudfront has been selected as one of the 10 Editor's pick of 2010 by Streaming Media. Amazon Cloudfront is the Content Delivery Network (CDN) that is dead simple to use both from a technology and a business point of view. From a technology point it literally takes a single button click in the console or a single API call to have your content CDN enabled. You can find several websites marveling about how simple it is to CDN enable your content with amazon Cloudfront. On the business side Amazon Cloudfront revolutionized the CDN business by providing a true pay-as-you-go service, no longer requiring upfront commitments that are commonplace in the CDN business and which opens up CDN for everyone to use.
Amazon Cloudfront is Streaming Media 2010 Editor's pick
I am excited that Amazon Cloudfront has been selected as one of the 10 Editor's pick of 2010 by Streaming Media. Amazon Cloudfront is the Content Delivery Network (CDN) that is dead simple to use both from a technology and a business point of view. From a technology point it literally takes a single button click in the console or a single API call to have your content CDN enabled.
AWS Import/Export launches support for Legacy Storage Systems
Today Amazon Web Services takes another big step in making it easier to migrate legacy storage systems to the cloud through AWS Import/Export support for ingesting Punch Cards. AWS Import/Export accelerates moving large amounts of data into and out of AWS using portable storage media for transport. Punch cards are paper-based storage media that represent data using the presence or absence of holes in specific positions. With AWS Import/Export for Punch Cards, enterprises can begin using the service to preserve and unlock the large volumes of data that have accumulated over the last century on what was the first broadly adopted digital storage medium. "
AWS Import/Export launches support for Legacy Storage Systems
Today Amazon Web Services takes another big step in making it easier to migrate legacy storage systems to the cloud through AWS Import/Export support for ingesting Punch Cards. AWS Import/Export accelerates moving large amounts of data into and out of AWS using portable storage media for transport. Punch cards are paper-based storage media that represent data using the presence or absence of holes in specific positions.
Choosing Consistency
Amazon SimpleDB has launched today with a new set of features giving the customer more control over which consistency and concurrency models to use in their database operations. There are now conditional put and delete operations as well a new "consistent read" option. These new features will make it easier to transition those applications to SimpleDB that are designed with traditional database tools in mind. Revisiting the Consistency Challenges Architecting distributed systems that need to reliably operate at world-wide scale is not a simple task. There are many factors that come into play when you need to meet stringent availability and performance requirements under ultra-scalable conditions.
Choosing Consistency
Amazon SimpleDB has launched today with a new set of features giving the customer more control over which consistency and concurrency models to use in their database operations. There are now conditional put and delete operations as well a new "consistent read" option. These new features will make it easier to transition those applications to SimpleDB that are designed with traditional database tools in mind.
Expanding the Cloud - Amazon EC2 Spot Instances
Today we launched a new option for acquiring Amazon EC2 Compute resources: Spot Instances. Using this option, customers bid any price they like on unused Amazon EC2 capacity and run those instances for as long their bid exceeds the current "Spot Price." Spot Instances are ideal for tasks that can be flexible as to when they start and stop. This gives our customers an exciting new approach to IT cost management. The central concept in this new option is that of the Spot Price, which we determine based on current supply and demand and will fluctuate periodically. If the maximum price a customer has bid exceeds the current Spot Price then their instances will be run, priced at the current Spot Price.
Expanding the Cloud - Amazon EC2 Spot Instances
Today we launched a new option for acquiring Amazon EC2 Compute resources: Spot Instances. Using this option, customers bid any price they like on unused Amazon EC2 capacity and run those instances for as long their bid exceeds the current "Spot Price." Spot Instances are ideal for tasks that can be flexible as to when they start and stop.
Powerful New Amazon EC2 Boot Features
Today a powerful new feature is available for our Amazon EC2 customers: the ability to boot their instances from Amazon EBS (Elastic Block Store). Customers like the simplicity of the AMI (Amazon Machine Image) model where they either choose a preconfigured AMI or upload their own AMI into Amazon S3. A wide variety of operating systems and software configurations is available for use. But customers have also asked us for more flexibility and control in the way that Amazon EC2 instances are booted such that they have finer grained control over for example what software configurations and data sets are available to the instance at boot time.
Expanding the Cloud - New AWS Region: US-West (Northern California)
We have expanded the AWS footprint in the US and starting today a new AWS Region is available for use: US-West (Northern California). This new Region consists of multiple Availability Zones and provides low-latency access to the AWS services from for example the Bay Area. In the US, AWS customers now can choose between the US-East (Northern Virginia) Region and the new US-West (Northern California) Region. In addition, the EU (Ireland) Region is available to customers who want local access to services from Europe to address their performance or jurisdiction requirements. As we announced earlier this month a Region with multiple Availability Zones will come online in Singapore in the first half of 2010, with other regions in Asia to follow later in 2010.
Powerful New Amazon EC2 Boot Features
Today a powerful new feature is available for our Amazon EC2 customers: the ability to boot their instances from Amazon EBS (Elastic Block Store). Customers like the simplicity of the AMI (Amazon Machine Image) model where they either choose a preconfigured AMI or upload their own AMI into Amazon S3.
Expanding the Cloud - New AWS Region: US-West (Northern California)
We have expanded the AWS footprint in the US and starting today a new AWS Region is available for use: US-West (Northern California). This new Region consists of multiple Availability Zones and provides low-latency access to the AWS services from for example the Bay Area. In the US, AWS customers now can choose between the US-East (Northern Virginia) Region and the new US-West (Northern California) Region.
82 Billion Objects in Amazon S3
At the end of Q3 2009 we counted over 82 billion objects in Amazon S3. Congrats to the team for providing such a rock solid service! When looking at the graph keep in mind that the first 4 markers are a year apart, but the last one only 6 months.
82 Billion Objects in Amazon S3
At the end of Q3 2009 we counted over 82 billion objects in Amazon S3. Congrats to the team for providing such a rock solid service! When looking at the graph keep in mind that the first 4 markers are a year apart, but the last one only 6 months.
Expanding the Cloud: The Amazon Relational Database Service (RDS)
Today marks the launch of Amazon RDS - the Amazon Relational Database Service. Amazon RDS is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. Amazon RDS handles all the "muck" of relational database management freeing up its users to focus on their applications and business. Fine Tuning Data Management At Amazon we have a long history of fine tuning our data management solutions to make sure that our systems can be reliable and cost-effective as we continue to scale. Almost from the beginning of operating the Amazon ecommerce platform it was clear that its scalability, reliability, performance, and cost-effectiveness were all dependent on the way that data was managed.
Expanding the Cloud: The Amazon Relational Database Service (RDS)
Today marks the launch of Amazon RDS - the Amazon Relational Database Service. Amazon RDS is a web service that makes it easy to set up, operate, and scale a relational database in the cloud. Amazon RDS handles all the "muck" of relational database management freeing up its users to focus on their applications and business.
Expanding the Cloud: Amazon Web Services to support the Federal Government
In the past week both Vivek Kundra, the U.S. CIO, and Casey Coleman, the CIO of the GSA, have made very strong statements in supporting the use of cloud computing to power Federal programs. A good example is today's announcement about apps.gov. In conversations with Vivek and Casey, I am struck every time by how much their observations that Federal CIOs are focused too much on infrastructure issues are similar to the observations within Amazon a number of years ago that motivated us to develop the AWS Infrastructure services. At that time, Amazon engineering teams focused more than 70% of their work effort on keeping their infrastructure efficient, scalable and reliable, which were important, but non-differentiating tasks.
Expanding the Cloud: Amazon Web Services to support the Federal Government
In the past week both Vivek Kundra, the U.S. CIO, and Casey Coleman, the CIO of the GSA, have made very strong statements in supporting the use of cloud computing to power Federal programs. A good example is today's announcement about apps.gov. In conversations with Vivek and Casey, I am struck every time by how much their observations that Federal CIOs are focused too much on infrastructure issues are similar to the observations within Amazon a number of years ago that motivated us to develop the AWS Infrastructure services.
Seamlessly Extending the Data Center - Introducing Amazon Virtual Private Cloud
At this 3rd anniversary of the launch of Amazon Elastic Compute Cloud (Amazon EC2), it is amazing to see the impact this service has had on the industry. It is truly disruptive technology and its impact has reached far beyond a pure technology offering as the benefits of the cloud have changed the way we view IT Infrastructure. As one of the CIOs at the ACM Cloud Computing Roundtable summarized it: "IT used to be the blocker in anything we did, but with our shift to the cloud IT is now the enabler." From young businesses and established enterprises to hospitals and governments agencies, all are equally enthusiastic cloud customers for whom IT infrastructure has changed forever.
Seamlessly Extending the Data Center - Introducing Amazon Virtual Private Cloud
At this 3rd anniversary of the launch of Amazon Elastic Compute Cloud (Amazon EC2), it is amazing to see the impact this service has had on the industry. It is truly disruptive technology and its impact has reached far beyond a pure technology offering as the benefits of the cloud have changed the way we view IT Infrastructure.
Feedback for Amazon Web Services
Ingrained in the DNA of Amazon Technologist is a single-minded focus on the needs of our customers. The Amazon development process is even called "Working from the customer backwards". Essential in this process is a good understanding of what the customers need in terms of new services, new features for existing services, or different approaches to things that we are already doing. We collect this feedback continuously from various sources: the AWS forums, the AWS Premium Support Team, Amazonians on the road talking to customers, solution architects helping to define customer architectures, ISV partners building on our services, system integration partners who relay customer needs, advisory boards, and of course the Amazon ecommerce engineers building on the AWS platform.
Feedback for Amazon Web Services
Ingrained in the DNA of the Amazon Technologist is a single-minded focus on the needs of our customers. The Amazon development process is even called "Working from the customer backwards". Essential in this process is a good understanding of what the customers need in terms of new services, new features for existing services, or different approaches to things that we are already doing.
Amazon is in-world and hiring!
Join recruiters and hiring managers from several of Amazon's global offices on July 14, 2009. We'll be in-world from 6am through midnight (Pacific/Seattle time) for the first ever Amazon Second Life Job Fair. This free event is a unique opportunity for candidates to have direct access to hiring managers and recruiters from around the world! We are looking across all levels of technical and non-technical profiles - from hands-on engineers to program managers and game-changing principal architects. Visit our U.S. career site at www.amazon.com/careers for open U.S. positions and links to our global careers pages, then join us in-world at www.bit.ly/AmazonJobFair on July 14.
Amazon is in-world and hiring!
Join recruiters and hiring managers from several of Amazon's global offices on July 14, 2009. We'll be in-world from 6am through midnight (Pacific/Seattle time) for the first ever Amazon Second Life Job Fair. This free event is a unique opportunity for candidates to have direct access to hiring managers and recruiters from around the world!
Expanding the Cloud: Moving large data sets into Amazon S3 with AWS Import/Export.
Before networks were everywhere the easiest way to transport information from one computer in your machine room was to write the data to a floppy disk, run to the computer and load the data there from that floppy. This form of data transport was jokingly called "sneaker net". It was efficient because networks only had limited bandwidth and you wanted to reserve that for essential tasks. In some ways the computing world has changed dramatically; networks have become ubiquitous and the latency and bandwidth capabilities have improved immensely. Next to this growth in network capabilities we have been able to grow something else to even bigger proportions, namely our datasets.
Expanding the Cloud: Moving large data sets into Amazon S3 with AWS Import/Export.
Before networks were everywhere, the easiest way to transport information from one computer in your machine room was to write the data to a floppy disk, run to the computer and load the data there from that floppy. This form of data transport was jokingly called "sneaker net". It was efficient because networks only had limited bandwidth and you wanted to reserve that for essential tasks.
Automating the management of Amazon EC2 using Amazon CloudWatch, Auto Scaling and Elastic Load Balancing
The Amazon Elastic Compute Cloud (Amazon EC2) embodies much of what makes infrastructure as a service such a powerful technology; it enables our customers to build secure, fault-tolerant applications that can scale up and down with demand, at low cost. Core in achieving these levels of efficiency and fault-tolerance is the ability to acquire and release compute resources in a matter of minutes, and in different Availability Zones. Of course the best way to achieve efficiency and fault-tolerance while maintaining good performance is to fully automate the management of the Amazon EC2 Instances, such that you can optimize the use of the compute resources in different scenarios.
Automating the management of Amazon EC2 using Amazon CloudWatch, Auto Scaling and Elastic Load Balancing
The Amazon Elastic Compute Cloud (Amazon EC2) embodies much of what makes infrastructure as a service such a powerful technology; it enables our customers to build secure, fault-tolerant applications that can scale up and down with demand, at low cost. Core in achieving these levels of efficiency and fault-tolerance is the ability to acquire and release compute resources in a matter of minutes, and in different Availability Zones.
Making A Dramatic Difference
As some of you may know both my daughters are studying Drama in London. Last time when I visited them I met two friends of Kim, Georgia Munnion and Lauren Hopkins. They are all classmates and they are graduating this year. Georgia and Lauren impressed me with a plan they have for spending the two months after their graduation in Nepal providing educational Drama Workshops to Himalayan children. We will use Drama as a basis for a process of individual social development, and to improve individual, group and team-building skills, through a range of individual and ensemble exercises. Using our acquired theatre knowledge and experience, we will provide education and enjoyment to underprivileged children, who are mostly unable to return home during their holidays, due to snow-bound and monsoon-bound trails and the recent outbreak of war.
Making A Dramatic Difference
As some of you may know both my daughters are studying Drama in London. Last time when I visited them I met two friends of Kim, Georgia Munnion and Lauren Hopkins. They are all classmates and they are graduating this year. Georgia and Lauren impressed me with a plan they have for spending the two months after their graduation in Nepal providing educational Drama Workshops to Himalayan children.
Good Advice on Keeping Your Database Simple and Fast.
Keeping your database simple and fast is often difficult if you use higher level frameworks such as ActiveRecords in Ruby or Java object persistence technologies such as Hibernate. There is a lot of magic that is happening out of sight that you have no control over. If you then have to scale your application it is often the relational database that these technologies require that becomes the performance and scaling bottleneck. Often requiring complex custom implementations of partitioning and sharding to make it work. The AWS services Amazon S3 and Amazon SimpleDB were designed to handle the dominant storage usage patterns within Amazon and they greatly reduced our need to rely on relational storage for scaling our systems.
Good Advice on Keeping Your Database Simple and Fast.
Keeping your database simple and fast is often difficult if you use higher level frameworks such as ActiveRecords in Ruby or Java object persistence technologies such as Hibernate. There is a lot of magic that is happening out of sight that you have no control over. If you then have to scale your application it is often the relational database that these technologies require that becomes the performance and scaling bottleneck.
Introducing Amazon EC2 Reserved Instances - A way to further reduce IT costs.
Flexibility is a key advantage of using Amazon Web Services; you can obtain resources instantaneously without the headache of owning them. If you no longer need the resource, you release it and only pay for what you have used. This is a very powerful model that has helped many of our customers drive capital expense out of their IT operation. It has helped both enterprises and startups reduce the risk that comes with developing new products and businesses. While this on-demand flexibility is ideal for a whole range of scenarios, some Amazon EC2 customers who have more predictable workloads have asked us for even greater flexibility in the cost model through the ability to reserve capacity.
Introducing Amazon EC2 Reserved Instances - A way to further reduce IT costs.
Flexibility is a key advantage of using Amazon Web Services; you can obtain resources instantaneously without the headache of owning them. If you no longer need the resource, you release it and only pay for what you have used. This is a very powerful model that has helped many of our customers drive capital expense out of their IT operation.
Expanding the Cloud: Expanding Amazon EC2 for Windows
Today we have some important news for our Amazon EC2 customers who are running Windows Server and Windows SQLServer instances and who have been looking to extend their coverage for fault-tolerance and locality reasons. Starting today Windows instances can be launched in an additional Availability Zone is the US and they can also be launched in two Availability Zones in Europe. This allows developers who use our Windows instances to build solutions that can tolerate various failure and recovery scenarios. It also puts Windows Server into the hands of developers who want low latency for their European customers. We have also integrated these features into the Amazon AWS console, such that you now can use the console to launch instances in any of our regions, regardless whether it is Europe or the US.
Expanding the Cloud: Expanding Amazon EC2 for Windows
Today we have some important news for our Amazon EC2 customers who are running Windows Server and Windows SQLServer instances and who have been looking to extend their coverage for fault-tolerance and locality reasons. Starting today Windows instances can be launched in an additional Availability Zone is the US and they can also be launched in two Availability Zones in Europe.
Eventually Consistent - Revisited
I wrote a first version of this posting on consistency models about a year ago, but I was never happy with it as it was written in haste and the topic is important enough to receive a more thorough treatment. ACM Queue asked me to revise it for use in their magazine and I took the opportunity to improve the article. This is that new version. Eventually Consistent - Building reliable distributed systems at a worldwide scale demands trade-offs between consistency and availability. At the foundation of Amazon's cloud computing are infrastructure services such as Amazon's S3 (Simple Storage Service), SimpleDB, and EC2 (Elastic Compute Cloud) that provide the resources for constructing Internet-scale computing platforms and a great variety of applications.
Eventually Consistent - Revisited
I wrote a first version of this posting on consistency models about a year ago, but I was never happy with it as it was written in haste and the topic is important enough to receive a more thorough treatment. ACM Queue asked me to revise it for use in their magazine and I took the opportunity to improve the article.
Teamwork
A question I get asked frequently is how working in industry is different from working in academia. My answer from the beginning has been that the main difference is teamwork. While in academia there are collaborations among faculty and there are student teams working together, the work is still rather individual, as is the reward structure. In industry you cannot get anything done without teamwork. Products do not get build by individuals but by teams; definition, implementation, delivery and operation are all collaborative processes that have many people from many different disciplines working together. As such the Information Week's Chief of the Year award cannot be my award.
Teamwork
A question I get asked frequently is how working in industry is different from working in academia. My answer from the beginning has been that the main difference is teamwork. While in academia there are collaborations among faculty and there are student teams working together, the work is still rather individual, as is the reward structure.
Expanding the Cloud: Amazon EC2 in Europe
Starting today the Amazon Elastic Computing Cloud (EC2) supports the ability to launch instances in multiple geographically distinct regions. The new EU region enables users to launch instances in Europe. This addresses the requests from many our European customers and from companies that want to run instances closer to European customers. Over the past year I have visited with many of our European customers and frequently they remarked "if only we had EC2 in Europe". We heard their requests load and clear and have worked very hard to roll out the European Region. This is a very important milestone on the road to local access to all our services.
Expanding the Cloud: Amazon EC2 in Europe
Starting today the Amazon Elastic Computing Cloud (EC2) supports the ability to launch instances in multiple geographically distinct regions. The new EU region enables users to launch instances in Europe. This addresses the requests from many our European customers and from companies that want to run instances closer to European customers.
Expanding the Cloud: Amazon CloudFront
Today marks the launch of Amazon CloudFront, the new Amazon Web Service for content delivery. It integrates seamlessly with Amazon S3 to provide low-latency distribution of content with high data transfer speeds through a world-wide network of edge locations. It requires no upfront commitments and is a pay-as-you-go service in the same style as the other Amazon Web Services. Amazon CloudFront has been designed to be fast; the service will cache copies of the content in edge locations close to the end-user's location, significantly lowering the access latency to the content. High sustainable data transfer rates can be achieved with the service especially when distributing larger objects.
Expanding the Cloud: Amazon CloudFront
Today marks the launch of Amazon CloudFront, the new Amazon Web Service for content delivery. It integrates seamlessly with Amazon S3 to provide low-latency distribution of content with high data transfer speeds through a world-wide network of edge locations. It requires no upfront commitments and is a pay-as-you-go service in the same style as the other Amazon Web Services.
Using the Cloud to build highly-efficient systems
These are times where many companies are focusing on the basics of their IT operations and are asking themselves how they can operate more efficiently to make sure that every dollar is spent wisely. This is not the first time that we have gone through this cycle, but this time there are tools available to CIOs and CTOs that help them to manage their IT budgets very differently. By using infrastructure as a service, basic IT costs are moved from a capital expense to a variable cost, building clearer relationships between expenditures and revenue generating activities. CFOs are especially excited about the premise of this shift.
Amazon EC2 in Full Production
Congratulations to the Amazon EC2 team for the hard work to get to the point where the beta tag is removed from the service and it is now in full production. Not only that, but there now is an SLA, and Microsoft Windows and SQL Server are available as of today. More details on the Amazon EC2 product page and on the Amazon Web Services weblog.
Using the Cloud to build highly-efficient systems
These are times where many companies are focusing on the basics of their IT operations and are asking themselves how they can operate more efficiently to make sure that every dollar is spent wisely. This is not the first time that we have gone through this cycle, but this time there are tools available to CIOs and CTOs that help them to manage their IT budgets very differently.
Amazon EC2 in Full Production
Congratulations to the Amazon EC2 team for the hard work to get to the point where the beta tag is removed from the service and it is now in full production. Not only that, but there now is an SLA, and Microsoft Windows and SQL Server are available as of today.
Expanding the Cloud: Microsoft Windows Server on Amazon EC2
The backend servers that power the world of Internet Services have become increasingly diverse. With today's announcement that Microsoft Windows Server is available on Amazon EC2 we can now run the majority of popular software systems in the cloud. Windows Server ranked very high on the list of requests by customers so we are happy that we will be able to provide this. One particular area that customers have been asking for Amazon EC2 with Windows Server was for Windows Media transcoding and streaming. There is a range of excellent codecs available for Windows Media and there is a large amount of legacy content in those formats.
Expanding the Cloud: Microsoft Windows Server on Amazon EC2
The backend servers that power the world of Internet Services have become increasingly diverse. With today's announcement that Microsoft Windows Server is available on Amazon EC2 we can now run the majority of popular software systems in the cloud. Windows Server ranked very high on the list of requests by customers so we are happy that we will be able to provide this.
AWS Startup Challenge 2008
The last week for submitting the applications for the AWS Startup Challenge has started. Looking at the proposals that are being submitted it looks like this will be another very inspiring challenge. These proposals are reviewed by a panel and five finalists will be selected. The finalists will come to Seattle to compete for $50K in cash, $50K in AWS credits, 2 years of Premium Support and more. All finalists will receive Rightscale Premium for 6 months and there will be a number of promotional events that includes all the finalists. Last year there were 900 applications which made for very intense proposal reading sessions.
AWS Startup Challenge 2008
The last week for submitting the applications for the AWS Startup Challenge has started. Looking at the proposals that are being submitted it looks like this will be another very inspiring challenge. These proposals are reviewed by a panel and five finalists will be selected. The finalists will come to Seattle to compete for $50K in cash, $50K in AWS credits, 2 years of Premium Support and more.
Expanding the Cloud
For many the "Cloud" in Cloud Computing signifies the notion of location independence; that somewhere in the internet services are provided and that to access them you do not need any specific knowledge of where they are located. Many applications have already been built using cloud services and they indeed achieve this location transparency; their customers do not have to worry about where and how the application is being served. However for developers to do their job properly the cloud cannot be fully transparent. As much as we would like to make it easy and simple for everyone, building high-performance and highly reliable applications in the cloud requires that the developers have more control.
Expanding the Cloud
For many the "Cloud" in Cloud Computing signifies the notion of location independence; that somewhere in the internet services are provided and that to access them you do not need any specific knowledge of where they are located. Many applications have already been built using cloud services and they indeed achieve this location transparency; their customers do not have to worry about where and how the application is being served.
Amazon EBS - Elastic Block Store has launched
Today marks the launch of Amazon EBS (Elastic Block Store), the long awaited persistent storage service for EC2. Details can be found on the EC2 detail page, the press release and Jeff Barr's posting over on the AWS evangelists blog. Also the folks at Rightscale have two detailed postings: why Amazon EBS matters and Amazon EBS explained. With the launch of the Elastic Block Store we complete an important milestone in offering a complete suite of storage solutions as part of the Amazon Infrastructure Services. Back in the days when we made the architectural decision to virtualize the internal Amazon infrastructure one of the first steps we took was a deep analysis of the way that storage was used by the internal Amazon services.
Amazon EBS - Elastic Block Store has launched
Today marks the launch of Amazon EBS (Elastic Block Store), the long awaited persistent storage service for EC2. Details can be found on the EC2 detail page, the press release and Jeff Barr's posting over on the AWS evangelists blog. Also the folks at Rightscale have two detailed postings: why Amazon EBS matters and Amazon EBS explained.
Werner Vogels - Some Basic Information
This page has some of the official material that can be used for conferences, interviews, etc. This is the official bio. Dr. Werner Vogels is Vice President & Chief Technology Officer at Amazon.com where he is responsible for driving the company?s technology vision, which is to continuously enhance the innovation on behalf of Amazon?s customers at a global scale.
Root Cause
For those of you interested in the details of last Sunday's Amazon S3 Availability issue you should read the detailed explanation posted at the AWS Status Dashboard. Root cause was single bit corruption of internal state messages that are distributed via Gossip techniques.
Root Cause
For those of you interested in the details of last Sunday's Amazon S3 Availability issue you should read the detailed explanation posted at the AWS Status Dashboard. Root cause was single bit corruption of internal state messages that are distributed via Gossip techniques.
An Album for Each Year
What better way to pick up posting again than with following a meme. Nick Carr in Albums Going Steady describes the challenge to list "a favorite album for every year of your life." I actually do not have the problem described by Nick and others to really start with my birth year. The challenge has two restrictions: only one album per year and there can be no repeats of artists. I have added for myself the restriction that I should actually own the album, which restricts the set to choose from significantly and also makes for some peculiar choices. Here is my list 1958: Jerry Lee Lewis, Great Balls of Fire 1959: Ray Charles, What I'd Say 1960: Miles Davis, Sketches of Spain 1961: Robert Johnson, King of the Delta Blues Singers 1962: Booker T & MG, Green Onions 1963: James Brown, Live at the Apollo 1964: John Coltrane, ...
An Album for Each Year
What better way to pick up posting again than with following a meme. Nick Carr in Albums Going Steady describes the challenge to list "a favorite album for every year of your life." I actually do not have the problem described by Nick and others to really start with my birth year.
The Perfect Laptop - Unboxing the X300
The laptop that appeared on the cover of business week as part of the story “Building the Perfect Laptop” is the Thinkpad X300. It arrived at my doorstep this afternoon. It is everything it promised to be and more; superlight, rugged, SSD, full ports, wifi, lan & cell networks, dvd, replaceable batteries and 13.3" screen with 1440x900 graphics. And all of this weighs in at 1420 grams. It is amazing how light it is for a full featured laptop. See the unboxing pictures on flickr.
The Perfect Laptop - Unboxing the X300
The laptop that appeared on the cover of business week as part of the story ?Building the Perfect Laptop? is the Thinkpad X300. It arrived at my doorstep this afternoon. It is everything it promised to be and more; superlight, rugged, SSD, full ports, wifi, lan & cell networks, dvd, replaceable batteries and 13.
Ahead in the Cloud
My opening slide for tomorrow's keynote at the MySQL Conference has this feel of speed and excitement to it that represents the current progress towards Cloud Computing. Persistent Storage for EC2 will be an important part of the presentation, but I'll mainly focus on general non-functional lesson from building large-scale services.
Ahead in the Cloud
My opening slide for tomorrow's keynote at the MySQL Conference has this feel of speed and excitement to it that represents the current progress towards Cloud Computing. Persistent Storage for EC2 will be an important part of the presentation, but I'll mainly focus on general non-functional lesson from building large-scale services.
Persistent Storage for Amazon EC2
I would like to introduce to you the newest feature of Amazon EC2: Persistent local storage. This has been very high on the request list of EC2 customers and I believe that combined with the Availability Zones and Elastic IP Address features released earlier this month this makes EC2 the ideal environment for building highly scalable and reliable applications. Significant innovation has gone into this feature: Instead of restricting developers to the use of a particular (distributed) file-system we once again decided to look at what is the most fundamental building block and how we could offer that in the most scalable and reliable manner.
Persistent Storage for Amazon EC2
I would like to introduce to you the newest feature of Amazon EC2: Persistent local storage. This has been very high on the request list of EC2 customers and I believe that combined with the Availability Zones and Elastic IP Address features released earlier this month this makes EC2 the ideal environment for building highly scalable and reliable applications.
On the Road to Highly Available EC2 Applications
Today Amazon Web Services launched two new features in Amazon EC2 that are essential tools in building highly resilient applications: Elastic IP addresses and Availability Zones. In summary: Elastic IP addresses are associated with a customer account and allow the customer to do its own dynamic mapping of IP address to instance. Using this dynamic mapping applications can remain reachable even in the presence of failures. This is an area where for example DNS reconfiguration is too slow a technique. Availability Zones allow the customer to specify in which location to launch a new EC2 instance. The world is divided up into Regions and a Region can hold multiple Availability Zones.
On the Road to Highly Available EC2 Applications
Today Amazon Web Services launched two new features in Amazon EC2 that are essential tools in building highly resilient applications: Elastic IP addresses and Availability Zones. In summary: Elastic IP addresses are associated with a customer account and allow the customer to do its own dynamic mapping of IP address to instance.
The Next Web Event
While the past months have been relatively quiet there is now a period coming up with public events that will take me across a few continents. The period already started two weeks with a Distinguished Lecture at the School of Computer Science of CMU. I had a wonderful day meeting many academics to discuss the relevance of particular research subjects for companies such as Amazon. I believe that Peter Lee, the Head of Computer Science, is the only Head/Chair/Dean of a Computer Science department who maintains a weblog. It is worth reading for a different perspective. Last week there was a very different event: A fireside chat at the close of the Under the Radar conference.
The Next Web Event
While the past months have been relatively quiet there is now a period coming up with public events that will take me across a few continents. The period already started two weeks with a Distinguished Lecture at the School of Computer Science of CMU. I had a wonderful day meeting many academics to discuss the relevance of particular research subjects for companies such as Amazon.
Happy Birthday, Amazon S3!
A few days ago, on March 14, Amazon S3 quietly celebrated its 2nd birthday. I think congratulations are in order and I certainly wish the service "many happy returns of this day". That S3 is growing up fast is obvious from the number of objects the customers trust us with.
Happy Birthday, Amazon S3!
A few days ago, on March 14, Amazon S3 quietly celebrated its 2nd birthday. I think congratulations are in order and I certainly wish the service "many happy returns of this day". That S3 is growing up fast is obvious from the number of objects the customers trust us with.
Got Questions?
On Thursday I'll be on stage at the Under the Radar conference for a fireside chat with Robert Scoble. The Under the Radar folks have asked for input into what questions Robert should ask me. The chat will be focused on cloud computing and related topics, so if you have questions, post them on their blog.
HDR - Take 1
Today I finished processing my first set of High Dynamic Range images. HDR is where you take 3 shots of an object using automatic exposure bracketing, in this case at [-2, 0, +2]. Each of the images will have details of areas (highlight, shadows) that the others do not have.
Usenix Open Access
I was going to pick up posting again and what better way to do that than to point to today's announcement by Usenix to open up the access to all of their conference proceedings. Compared to the IEEE Computer Society and the ACM, who still hide the published material behind the walls of their digital libraries, Usenix already had a very liberal one-year-members-only policy.
?????? ???? ?????????????
For those of you who always wanted an Amazon robot you can now buy this special Amazon version of the Danboard character from the Yotsuba & ! series.
Eventually Consistent
I wrote a first version of this posting on consistency models in December 2007, but I was never happy with it as it was written in haste and the topic is important enough to receive a more thorough treatment. ACM Queue asked me to revise it for use in their magazine and I took the opportunity to improve the article.
Marc
Sadness. Marc Orchant, the ulimate optimist, is no longer with us.
And the Winner is ...
Today we had a wonderful day with the finalists of the 900 contestants in the AWS Startup Challenge, all seven proved themselves worthy of winning the ultimate price. The finalists were: Brainscape Commerce360 Justin.tv Milemeter Ooyala WeoGeo Go to the startup challenge vote page to see videos of each of these companies and to see the voting results.
Breaking through Physical Boundaries
The emotions about reading books in digital form and the Amazon Kindle are running high, already before the device was released. For me there are two features that sold me on the device: the networked content push and the content interaction. I have had many pda's, phones and tablets over the years that I setup to automatically pull in feeds such I could read them on the bus or plane.
Tribute to Honor Jim Gray
On May 31 2008 a tribute will be held at UC Berkeley to honor Jim Gray, who went missing during a solo sailing trip in January of this year. Although Jim is listed as missing, and will be until 2011, a Tribute be held to honor him before too much time has passed.
Amazon S3 in Europe
I am about to go on stage at Web 2.0 Expo in Berlin where we will announce the ability for Amazon S3 customers to store their objects in Amazon?s European storage cloud. This has been a frequently requested feature by our European customers for various reasons, better latency control being the most important one.
Max, Min & Fair
If we would just forget about discriminatory traffic management (e.g. based on deep packet inspection) life on the network would be pretty simple, even under overload conditions. A lot has been written about this already but Wes really nails it in his summary. Maximized profit, minimized frustration, everybody?s happy?
GMR
The 2007 Nobel prize in Physics has been awarded to Albert Fert and Peter Grünberg for independently discovering Giant Magnetoresistance in 1988. Their work had a tremendous impact on the computer industry as it revolutionized the way we could store and retrieve information on hard disks. It was the first major application of nanotechnology and allowed hard disks to shrink from the size of a large washing machine to the device that fits in an mp3 player.
HBR - The Institutional Yes
The Institutional Yes is a Harvard Business Review interview with Jeff Bezos about the way strategies are developed at Amazon. I have written before about how the relentless customer focus translates into driving architecture and design using the ?working backwards? approach. The interview with Jeff gives you more insight on the impact of customer focus on overall strategy and how it drives a culture of experimentation.
Clarifying Internal-only
There is a lot of positive feedback about the Dynamo paper but I noticed that something I wrote in introducing the paper is being misunderstood. This was my fault, I wrote it too strongly. What I meant by internal-only is that Dynamo is not directly exposed externally. However, Dynamo and similar Amazon technologies are used to power parts of our Amazon Web Services, such as S3.
Amazon's Dynamo
In two weeks we?ll present a paper on the Dynamo technology at SOSP, the prestigious biannual Operating Systems conference. Dynamo is internal technology developed at Amazon to address the need for an incrementally scalable, highly-available key-value storage system. The technology is designed to give its users the ability to trade-off cost, consistency, durability and performance, while maintaining high-availability.
Steve's Back
My old friend Steve Vinoski is back online. Steve was with Iona for many, many years, working as the main architect on many of their Middleware technologies. Steve left Iona this spring to work for a start-up in the Boston area. But Steve became really famous for this drawing, which he used to express his opinion about my keynote at Middleware 2004.
Chat
As a follow-up to the keynote I gave at the Next Generation Datacenter Conference the folks at Network World organized an online chat, to happen for an hour tomorrow (October 1) from 11-12 PST. I have never done something like this before so it will be fun to see how good my typing skills are, but at least I get to sneak out of a very long meeting for an hour.
The Final Blade Runner
I just finished reading the Wired interview with Ridley Scott about Blade Runner, the ultimate film noir. The movie is certainly one of my favorites and I own several editions of it. Of course I now also want this ?final cut version?, hoping that it would be available in HD. It turns out it being produced in 7 different packagings, all to be released on December 18 (which is pretty late for online holiday shoppers in my eyes)
50 X
In the recent years Mike Stonebraker has been advocating that the current commercial databases have become one-size-fits-all tools that are so general and heavy-weight that they do not excel in any particular area. Mike has written some papers on this topic of comparing general databases with specialized storage engines in areas of data-warehousing, text search, stream processing and scientific processing.
Amazon MP3
The beta of the long awaited DRM free Amazon music store is ready for use. Go download!
Amazon Widgets
A popular request has been to make more of the Amazon product data available as easy widgets for reuse on blogging, social network and associates websites. You can now find a nice collection of different widgets at http://widgets.amazon.com. Some of these were already available for associates, but there are some really nice new ones.
Mechanical Turk and the Search for Steve Fossett
Since Saturday morning you can help search for Steve Fossett through Amazon's Mechanical Turk. Click here to get to the start page and then press the "Work on HITs" button to start reviewing satellite images. As with the search for Jim Gray, this is only possible because of the collaboration of several organizations.
No QA. No Backups. No Sleep
Midnight in Seattle. Amazon Hackday 2007 in progress. World-wide
The Amazon Flexible Payments Service (Amazon FPS)
Today Amazon AWS launched a limited public beta of Amazon FPS, the Amazon Flexible Payment Service. Amazon FPS is a payment service that is 100% focused on the needs of developers. Traditionally, developers have been limited in how they can manage payments. If they need to charge with a certain frequency, execute a transaction at a specific time, combine many smaller payments into one single transaction or want to charge a commission on a transaction between two of their customers, they need to create their own payment infrastructure and processor relationships.
Wired on Jim Gray
The august edition of Wired has a long article on the disappearance of Jim Gray, earlier this year. Steve Silberman worked diligently on this article and interviewed many people including Donna Carnes, Jim?s wife. I like the article because it focuses more on Jim as the technologist, friend and mentor than on the technicalities of the search after his disappearance.
Potter Delivered
2.223 million pre-orders on our sites world-wide. 1.4 million on Amazom.com alone. These orders trickled in over the period of 5.5 months, but from a distributed systems perspective today is the day as these orders go en-masse from pre-orders to orders, being charged and delivered. It is one smooth operation. The planning for single day delivery is truly impressive, especially on the supply-chain, transportation and fulfillment side where we need to do this without impacting the regular delivery flow.
Job Opening for a Senior Research Engineer
When I was building one of my first teams at Amazon, one that had to work on some really advanced distributed systems technology I put up a job description on this weblog. I was certainly pleased with the responses. Last year at a conference I heard from some of my former academic colleagues that they were using this description to educate their students abput where they were lacking in knowledge or experience.
The Different CTO Roles
I was putting together a short panel presentation on the role of a Chief Technology Officer in corporate innovation and I once again realized that there is quite a bit of confusion around the role of the CTO. The first thing that always comes up when you want to discuss the role of a CTO is that there is no well established definition of what a CTO actually does.
Dynamo Reference
I omitted the Dynamo reference from the previous collection, but now that the SOSP program is live: Guiseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swami Sivasubramanian, Peter Vosshall and Werner Vogels, ?Dynamo: Amazon's Highly Available Key-Value Store?, to appear in the Proceedings of the 21st ACM Symposium on Operating Systems Principles, Stevenson, WA, October 2007.
Reading References
I recently gave a few talks in which I gave some reading advice to the audience and I promised to follow-up with posting the links here. The first article is the interview of Michael Stonebraker by Margo Seltzer in the May/June edition of ACM Queue (unfortunately the article is online yet, but this link seems to work thanks to Peter O?Kelly).
New in May
Tori Amos - American Doll Posse ? Each of the 5 persona on the album have their personal blogs, more details at the wikipedia page for the album (May 1). Groove Armada - Soundboy Rock ? Unpredictable (May 10 US release).
Dr. Swami
Today at noon Swami made the transition to Dr. Swaminathan Sivasubramanian at the Vrije Universiteit in Amsterdam. In most European countries the doctoral defense (promotie in Dutch) goes with a lot of protocol and pomp & circumstance. A very specific dress code, with two secundants who need to be able to take over in case you faint during the defense.
Myths of Innovation
Congrats to Scott Berkun for sending his "Myths of Innovation"book to the printer. Scott has collected an excellent set of recommendations for his book; among others JSB, Guy and me. It is well deserved; I read a beta version of the book and I was impressed. It is very different from Scott?s first book "
On the Reliability of Hard Disks
Today in the opening session of FAST there are two papers on the studies of hard disk reliability. Both these papers present very interesting results that blow away some of the common assumptions in failure modelling of systems. Bianca Schroeder and Garth Gibson from CMU in their paper Disk Failures in the Real World: What Does an MTTF of 1,000,000 Hours Mean to You?
The Conference Season is Opening Up Again
I have had the luxury of almost 4 months with any real conferences. Don?t get me wrong it is not that I do not enjoy the public speaking side of my job; it is just that I experience it always as rather disruptive. It is great to focus for a while and get things done.
The Search for Jim Gray Continues
As Mike Olsen just wrote on the Tenacious Search weblog, today was the first day that action could be taken on the boats found in the satellite and ER-2 streams. Bad weather has kept any aerial search parties on the ground until now. This morning two planes were dispatched to locations derived from images coordinates combined with drift models.
Half a Million Assignments Completed.
Over 530,000 Mechanical Turk assignments have been completed by more than 12,000 volunteers in the search for Jim Gray. We need a little more of a push and then all the images will have been processed. A team of experts lead by Alex Szalay of John Hopkins University has been working through the thousands of images marked for further investigation.
Turkers Working Hard on the Search for Jim Gray
It is now 3 PM on Sunday afternoon and the group of volunteers in thethe search for Jim Gray has worked their way through almost 100,000 assignments since Friday 5 PM. Since then we have seen over 6000 individual workers completing anywhere from 1 to almost a 1000 assignments. And there are still more to go.
High Altitude Search for Jim Gray
We have now added the data captured by the NASA ER-2 plane yesterday over the ocean area outside of San Francisco. We were very fortunate that this flight was scheduled for yesterday and that the NASA folks were interested in having it capture these images. We have been able to split them just like yesterday?s satellite images and create HITs (Human Interface Tasks) from them.
Help Find Jim Gray
Computer science icon Jim Gray mysteriously disappeared after a solo trip with his sail boat outside San Francisco Bay. The coast guard has been searching for 4 days but has not been able to locate anything, not even debris. On Thursday 3 private planes searched through the coastal areas and they also returned unsuccessful.
Jim Gray Missing at Sea
My long time friend and mentor Jim Gray took his sail boat out for a trip to the Farallon Island yesterday and has not been heard from since. The coast guard has been searching since last night. Jim is an extremely experienced sailor with more than 40 years experience. This is very worrisome.
SL -> WoW
Each year I try to immerse myself in a particular technology, either as a user or as a developer. This technology has preferably nothing to do with my day-job, or at least not directly. For the past year this was Second life. Today I sold all the SL real-estate that I acquired in the past year and parked my avatar permanently in The Blarney Stone Irish Bar in Dublin.
Out of the Office, Town and Country
Thank you for being loyal readers of this chaos, I hope you will all have great holidays, I know I certainly will. See you in the new year!
Sagan's Billions
Today is the 10th anniversary of Carl Sagan?s death. There is a world-wide event going on today to remember this master of science. I like to believe every one has stories of how Sagan impacted their lives, at least those in my generation. I remember this guy fascinated me, because he spoke so passionately and caring and full of amazement about science, much more than any teacher I had ever had.
The odds of getting a PS3 or Wii for Chrismas
As you have probably have heard by now Amazon is making Sony Playstation 3 (20 GB & 60 GB models) and Nintendo Wii available to existing Amazon customers who sign up via a special edition of the Amazon Customers Vote. A random draw will then decide who will get the change to buy these ridiculously scarce game machines.
The King of Disk Drives
There will be lots of stories remembering Al Shugart as ?Uncle Al?; these stories will be filled with love and fun, and lots of good laughter. They will portrait Al as a good natured loving mentor and an unconventional industry maverick, but there was so much more to this man.
Working Backwards
In the fine grained services approach that we use at Amazon, services do not only represent a software structure but also the organizational structure. The services have a strong ownership model, which combined with the small team size is intended to make it very easy to innovate. In some sense you can see these services as small startups within the walls of a bigger company.
New in November
J.J. Cale and Eric Clapton - The Road to Econdido - November 7 The Stranglers - Suite XVI - November 7 (I got it from the UK last month) Tenacious D - The Pick of Destiny - November 14 (Check out the NSFW video by Kricfalusi) Jay-Z - Reasonable Doubt - November 14 (Rerelease from 1996) Tom Waits - Orphans - November 21 (3 Discs: Brawlers, Bawlers and Bastards, including covers from the Ramones and songs from Brecht and Weill) The Casino Royal soundtrack will be released on November 14.
October Stuff
Zune. I finally got to play with a Zune media player. I am very impressed, I own all the generations of iPod, but this device is clearly a step up in terms of innovation. I can confirm that the enthusiasm in some of the earlier reports have good grounds. I have had many gadgets, but in the past two years except for the PSP nothing could really excite me, the Zune however is different at so many levels.
Carbonado
As of today one of the technologies developed internally at Amazon is available on sourceforge as an open-source project. From the Carbonado docs: Carbonado is an extensible, high performance persistence abstraction layer for Java applications, providing a relational view to the underlying persistence technology. Persistence can be provided by a JDBC accessible SQL relational database, or it can be a BDB.
Home to S3
Jeremy Zawodny has some interesting calculations on the absolute cost advantage of using S3 for your home backup server. For me however the most compelling argument is later in his posting when he mentions the administrative cost of running a highly available home server. I recently lost 2 disks in my home raid setup rendering the 1 TB data store useless.
Fulfilling
Amazon continues on the path of opening up all of its services for customers and partners to use. One of the services customers repeatedly asked us for was to open up our fulfillment network for them to use. Currently there are over 1 million small and large active sellers on the Amazon platform and most need to do their own fulfillment and shipping.
The 2006 Young Innovators
Every year the MIT Technology Review publishes a list of technologists and scientist under the age of 35 to honor their ground breaking inventions and research. In the past I have written a number of recommendations in support of these innovators and last year I was particularly happy to see George Candea recognized for his work on recovery oriented computing in general and for micro-reboots in particular.
Unbox Remote Control
One of the coolest features in Unbox, Amazon?s Video Download Service, is the remote delivery. A video you have bought will show up in Your Media Library and from there you can control which PC the movie should be downloaded to. I just bought a Margaret Cho show while at work and asked it to be downloaded to a laptop at home.
OpenSearch.org
Congrats to DeWitt for launching the OpenSearch.org community website
Amazon in Scotland
I have been easing my way back into Amazon life by visiting the Amazon Development Center Scotland (ADCS), located in the outskirts of Edinburgh. Before joining Amazon I was always skeptical whether distributing development could actually work. The overhead in communication, lack of quality personal contact and handling time-zones always seemed like barriers that were too high to overcome, and I have seen many cases fail.
OOTO August 2006
I have been off the grid in the old world and will remain so until the end of the month. If you are waiting for responses to emails, postings, approvals or comments; have patience.
Life is not a State-Machine
Last week I gave a keynote at the ACM Principles of Distributed Computing Conference (PODC) on the topic of technology transfer. My choice of topic was triggered by recent presentations by a number of other research luminaries, who had remarked that the distributed computing research community had failed to make its mark; lots of good ideas, little impact.
Amazon 2 Second Life
I promised at Supernova I would give an update on the integration of Amazon.com (through Amazon.com E-Commerce Web Services) into Second Life: Jeff Barr has the scoop and demonstrates it on the Amazon Web Services Weblog. As expected it was a complete grassroots effort with no official Amazon involvement. Go visit the life2life store now (yes that is second life url, you need the program installed to use this as a locator).
Facts & Sources
I really enjoy spending time at comics.com and gocomics.com. On comics.com there is a category list of all comics and editorial cartoons available on their site. On gocomics there is a separate list for comics and editorial cartoons. Both sites offer interesting subscription models. Candorville by Darrin Bell on 7/15/2006:
Graphs as Art
If done well a graph representation of data can reveal a lot of information about the structure of the data. And with particular care graphs can even become art. Sala has achieved this by building a very nice graph representation applet for web pages. If you visit this posting you?ll see some really cool graph representations.
Can you carry this for me?
Wow! 20.1 inch screen & 20.8 pounds; Introducing the Dell XPS M2010. I have been looking for a new laptop but for some reason I don?t see myself walking around with this one... It will be an interesting challenge to open it up in your economy airplane seat. Base price $3800.
Your Queues are Ready
Yesterday the Amazon Simple Queue Service moved from beta to production. SQS provides persistent messaging with the scalability and reliability of Amazon?s infrastructure. As a developer you can create an unlimited number of Queues and store unlimited messages in a Queue. As usual we manage your data in highly reliable fashion, capable of surviving complete data center failures.
Divine Feeds
The Atom and RSS Gods complained about my feeds. I made some offerings to the sacred Validator and I now no longer have to fear their wrath. I aplogize to those readers who had to suffer through a series of confusing feed updates and I welcome those who can now suddenly actually consume the feeds
A New Family Member
Today was a good news day: Geoff Arnold has decided to join the Amazon engineering family. As Geoff mentions in the announcement on his weblog Amazon is all about scale. In recent presentations I have been demonstrating how Amazon Engineers are scalability experts who can take any concept idea and turn it into a service that can serve hundreds of customers and then grow it seamlessly to support hundreds of millions of customers.
The Final
Zidane is a brilliant player, so it is sad to see his international career end this way. It was a very bizarre moment in an otherwise rather open match. Italy ruled during the first half, but the French had control over the second half and the extra time. Again Henry showed that he is not the cold killer he should be.
Gnomedex Topic â Net Neutrality
The phone and cable companies will fundamentally alter the Internet in America unless Congress acts to stop them. They have the market power, and regulatory permission to restrict American consumers? access to broadband Internet content, including music and movies, and have announced their plans to do so. There is a lot of freedom at Gnomedex for the discussion leaders to address the topics that are on their minds.
Some Observations on Conferences
The coffee breaks are the best parts of the conference I stole this remark from Harrison Owen. His observation was that the traditional way people come together to discuss issues was not very effective, but that they all thought the coffee breaks were the place where all the action happened.
Sports Book
Whenever I end up in Vegas I am continuously amazed by the magnitude of the way things are implemented here and at the same time the superficiality of it all. Normally I take a stroll north or south of the strip to ground me and get back some sense of reality.
What Are The Odds
I am on the road this week (I am speaking at Oracle GMM, Garner Enterpise Architecture & Supernova). The first stop is in Las Vegas. This is on the sports board in Vegas: Personally I think the Argentinians look a lot more on fire than the Brasileiros. Of course I am not rooting for either team.
Get Your Priorities Straight
it is good to now and then be reminded of what the real important things in live are. From the "gents"of a London pub:
You Guard it with Your Life
The important part about the laptop with 243,000 customer records including credit cards stolen is not the theft itself. It is what the heck these credit card numbers were doing on the laptop in the first place. They should never, ever have been there. There is no reason they should have left the ultra secure location they were kept in.
Growing (up) is hard
Building systems that can guarantee performance and availability while scaling up to handle exponential growth in datasets and user requests is still very much a Dark Art. It is an art we master quite well by now at Amazon but it took a lot of growing pains to get to this level of sophistication.
The Grocery
Another great new store launched today: Amazon.com Grocery An Amazon store filled non-perishable grocery items all eligible for Prime and Super Saver Shipping. An important new feature was launched alongside of it: Shopping Lists. This is like a wish list but you can use it for repeat purchases, different quantities and more.
Industrial & Scientific
All you Make junkies, go visit Amazon Industrial & Scientifc (beta)!
The Amazon Technology Platform
Earlier this year my good friend and mentor Jim Gray dropped by at Amazon to interview me for ACM Queue magazine. I had real fun with Jim?s questions and I like to believe that the resulting interview is interesting. It appeared in this month?s edition and runs about 8 pages in the print edition.
May Countdown
One more day: Pearl Jam ? Pearl Jam(5/2) One more week: Neil Young ? Living with War (Free Streaming of the Full Album) (5/8) One more week: Red Hot Chili Peppers ? Stadium Arcadium (DVD Edition) (5/9) Two more weeks: Hoobastank ? Every Man For Himself (5/16) Two more weeks: John Entwistle ? Bass Guitar Master Class (5/16) Three weeks: Ridley Scott ? Kingdom of Heaven - Directors Cut (5/23) One more month: Soccer World Cup (5/9 - 6/9)
Performance and Scalability
In the A Word On Scalability posting I tried to write down a more precise definition of scalability than is commeonly used. There were good comments about the definition at the posting as well as in a discussion at The ServerSide. To recap in a less precise manner I stated that
On the Wire
I don't want to turn this into a corporate announcement weblog, but I believe some of my new readers may be interested in this. Go check out Amazon Wire, a podcast about books, music, movies and those who create them.
Amazon Gets IM - Part II
It is nice that people thought yesterday?s posting was funny. There is however a very serious aspect to it. From the reactions I understand that people think that posting IM handles is too far-out to even consider seriously, which made it a good April?s Fools joke. For me, I do not believe it is ridiculous for a company to engage in real-time interaction.
Amazon Gets IM
In order to get closer to their customers, humanize Amazon, increase sales, and stay modern, Amazon.com has decided to make all Instant Messenger (IM) handles of its employees public. This way Amazon.com customers will get unprecedented access to the talented engineers at Amazon to answer all their questions, or just to have an interesting conversation about a new book or that old sci-fi movie.
Reactive Innovation?
Dennis Howlett reminds me that the last paragraph of my Naked Answers post suggests that I think the only way innovation happens is in reaction to customer demand. Customer feedback is an important ingredient, but not of course not the only component. Henri Fords famous quote ?If I had asked people what they wanted, they would have said, "Faster horses.
A Word on Scalability
Scalability is frequently used as a magic incantation to indicate that something is badly designed or broken. Often you hear in a discussion ?but that doesn?t scale? as the magical word to end an argument. This is often an indication that developers are running into situations where the architecture of their system limits their ability to grow their service.
Naked Answers
Today Shel Israel and Robert Scoble stopped by at Amazon to present their book Naked Conversations in our Fishbowl series. As you can read in Shel's observations and Robert?s they appear shocked that we used a critical voice to address their work. Welcome to life at Amazon, we set a very high bar for our own works and we expect anyone that comes to sell us an approach to actually be prepared to really defend their ideas.
S3 - The Amazon Simple Storage Service
Go check out the Amazon Simple Storage Service (S3). This is Amazon.com?s Internet scale storage service available through a web service interface. S3 is another example of the Amazon Web Services mission: to expose all of the atomic-level pieces of the Amazon.com platform. Providing scalable, reliable, secure and fast storage is something that Amazon developers have already enjoyed for some time and now it is available to the developer community outside of Amazon.
Not Really A Top Ten
Halley Suitt, now CEO of Top Ten Sources, has a way of talking me into doing things where I actually have no time for. This time it was giving her a list of my top ten weblog feeds. This is an impossible task as I don?t think there is such an absolute list, but try and explain that to someone who runs a company called Top Ten Sources.
Generating SiteInfo with MT
On the A9 Developer weblog there is an article today on he SiteInfo feature that was until now only available when you used the A9 Toolbar. They now have developed a Firefox extension that will give you the same functionality. Normally the siteinfo tab gives you the Alexa information about a site: traffic ranking, reviews, etc.
Associating
I have to admit that one of the reasons for my ?January Recommendations? posting was to experiment with the Amazon.com associates program. Associates earn up to 10% of the purchase if they drive traffic to Amazon.com and the customer actually places an order. Not that I was expecting to make any money of this, but I was interested in seeing what the experience is for an associate if s/he needs to create a collection of links, and how good the tools are for tracking the results.
Opportunities in Modeling Complex Distributed Systems
Modeling systems has always been part of the toolkit of the computer scientist. We often try to bring systems back to simple queuing models to understand throughput and latency questions, and then use those results to predict resource usage and drive allocation. Can one actually be confident that such a simple model can accurately reflect reality?
A Neutral Net
Today the Senate hearings on Net Neutrality took place. The details of the testimonies are online. I believe that everyone who has a vested interest in seeing business and service innovation continue to flourish on the internet should be concerned about the proposal for access-tiering introduced by the last-mile broadband vendors.
Another Reality
Because of my absentmindedness last year I forgot to point to another writer that has joined the public Amazon/A9 weblog family: In december Claire Giordano of opensolaris fame joined A9 as director of product management. Please visit Claire's Alternate Version of Reality.
Spring Systems Conferences
The programs for 2 conferences with mainly operating and distributed systems contributions are online now: 1st EuroSys Conference, April 18-21, 2006, Leuven, Belgium 3rd Symposium on Networked Systems Design & Implementation (NSDI?06), May 8-10, 2006, San Jose, CA With 58 papers between the two conferences the systems research field appears to be very healthy.
Links Below Sea Level Needed
In case you didn't know; I am Dutch. If you would have heard me speak you probably would have guessed that tidbit of useless information. In the past months I have received a number of criticizing comments that I am not keeping up with the all the developments in the Netherlands in terms of computing and ecommerce.
Non-Linear Podcasts
I have trouble understanding when people actually listen to podcasts. I don?t mean the funny and musical ones, you could listen to those while doing the dishes, but I mean the deep technical talks or interviews. The ones you actually need to pay attention to, to understand and learn. I am not good at multitasking while listening intently and I don?t ride the bus or get stuck in traffic.
January Recommendations
The first month of this year has been cold and rainy in Seattle but it did bring a number of surprises in books, music and theater. I am sure many of you have seen the announcement last week that Amazon.com will be producing a weekly internet-only Bill Maher show called ?Amazon Fishbowl with Bill Maher?.
Feeds Redirected
I have redirected the rss & atom feeds at weblogs.cs.cornell.edu to the their new home at http://www.allthingsdistributed.com. I used a permanent redirect so I am hoping that bloglines and other aggregators will eventually figure out that they should start polling those feeds directly instead of continue to be redirected by the old site.
The Return & The Move
A lot of things happened in that past months that were worthwhile writing about. New distributed systems and architectural insights, great conversations, new gadgets, good books, interesting articles and conferences, and some very cool new Amazon.com technologies (e.g. Mechanical Turk and the public access to Alexa). However I thought that it was appropriate to first move the weblog from the Cornell servers to a personal place.
Two Recommended Essayists
There are two online technology essay writers who I truly enjoy reading: Scott Berkun and Paul Graham. Paul is probably the more well known of the two given his Hacker and Painters book, but I think that both have a unique insights in the software industry and the development process.
Articles
Dr. Werner Vogels
Official Bio DR. WERNER VOGELS, CTO, AMAZON.COM Dr. Werner Vogels is Chief Technology Officer at Amazon.com where he is responsible for driving the company’s customer-centric technology vision. As one of the forces behind Amazon’s approach to cloud computing, he is passionate about helping young businesses reach global scale, and transforming enterprises into fast-moving digital organizations.