Out in the Open: How to Run an Entire Data Center as Easily as a Cellphone

Flynn wants to make managing applications in your data center as easy as managing the apps on your smart phone. Photo: Ariel Zambelich/WIRED

Think about how easily you can install an app on your phone. You download a tiny software package from an app store. You press the install button. And you’re done.

Now think about the massive applications that a company like Google runs on its worldwide network of data centers — applications like Gmail or Google Search. Inside Google, an application like Gmail doesn’t run on a single machine. It’s distributed across the thousands servers spread across multiple computing centers. Companies must ensure that each individual server has all the stuff it needs to run its particular part of the application — and that’s no small task.

But companies like Google are striving to make this process much easier, to make it more like installing a tiny little app on your smartphone. With massive software tools that span their data centers, Google and others, such as Twitter, can fairly quickly distribute new code across an enormous number of machines. The problem is that not everyone has the technical expertise of a Google or a Twitter.

“The Google infrastructure was very expensive to build, and it was very specific to their needs,” says developer Daniel Siders. “They had so much difficulty scaling that no one should have to go through that again. The resources to scale should be free for everyone.” That’s why Siders started Flynn, an open source project that aims to make it easy to build and deploy apps across many servers — whether they’re virtual machines on public cloud services like the Amazon Elastic Compute Cloud, or physical machines in your own data center.

In the Beginning

Like many open source projects, Flynn was born to scratch a particular itch for a developer. Siders is a developer at Cupcake, a company that provides the computing infrastructure for a communications tool called Tent.io, and at Cupcake, setting up servers turned out to be more work than he expected. “It was often taking longer to deploy the service than to write the application,” he says. So Siders and company wrote a proposal for Flynn, and they found huge demand for the idea. Several tech companies — including CenturyLink, Nebula, and Shopify — jumped at the chance to help fund it.

There are many other services designed to make it easy to build and deploy cloud applications. But unlike “platform clouds” such as Heroku or Google App Engine, Flynn is designed so that it can be run both in your own data center or in public clouds such as Amazon Web Services — or both at the same time.

Flynn is based in part on two other tools: One is Google’s Omega, a system that lets Google developers treat the company’s thousands of servers much like a single machine. More specifically, it distributes all the smaller pieces of an application across many different machines, ensuring that free resources don’t go to waste. Although Google has never released the source code for Omega, it did publish a paper explaining how the system worked, allowing the Flynn team to build their own clone.

The other is an open source tool called Docker. Docker bundles applications into self-sufficient units called “containers.” These can be easily moved from server to server, and they include everything the application needs to run. Unlike a virtual machine — which recreates the entire operating system — Docker containers are can take advantage of the host server’s operating system and other software, even though the containers are separated from each other. Basically, it’s another way of improving the efficiency of your infrastructure.

The Competition

There are other software tools that seek to do much the same thing, including Pivotal’s Cloud Foundry and Red Hat’s OpenShift. But for many developers, those tools are overkill, explains Lucas Carlson, the founder of a cloud company called AppFog, which was acquired by CenturyLink last year. Carlson says CenturyLink and AppFog will remain committed to CloudFoundry for the foreseeable future, but he indicates that Flynn represents the future of cloud development.

It’s clear that the Flynn team’s ambitions extend far beyond being merely a lightweight alternative to the older players in the field, and Docker is a big part of those ambitions. Other companies, such as Deis, offer a means of juggling Docker containers. But, ultimately, Flynn aims to be more comprehensive than Deis, which is focused solely on Docker , leaving resource optimization to other Omega clones, such as Mesos.

The particulars are complicated. But what just about everyone agrees on is that regardless of which open source platforms end up winning the most adoption, Docker will change the way developers think about deploying apps. “I don’t think the business side of Docker has been well explained yet,” says Carlson. “Even though it’s a tech trend that is just unstoppable.”