My Other Computer Is A Datacenter

Managing a datacenter doesn't require a hyperwall, just the right operating system.

ersi hyperwall photo by Kris Krüg on Flicker

Today, your personal computer is local. You own the hardware. The data is in a box under your desk. The processes run on a local operating system you installed.

Even those simple statements, which used to be absolute assertions, are getting fuzzier and fuzzier. You probably have data in "the cloud". You probably have used Google Docs, an application in the browser, running in Google's datacenters. Your email hasn't lived on your machine for a long time.

The next generation of computing, however, runs on multiple machines. Businesses are already doing this, running large datacenters, hosting applications distributed across multiple machines. Increasingly this looks like a new form factor. Increasingly abstractions hide the distribution and present a unified view. Increasingly the big players in the field are talking about their software as presenting a distributed kernel or distributed operating system for applications and services to run on.

What does this look like?

Lets take a look at the most mature of these: Mesos.

Kernel

Mesos is a generic distributed resource manager. Applications written for Mesos can schedule their processes on any number of distributed machines. Those processes might be short lived, like batch jobs, or they might be long lived, like web application servers. Those processes are also isolated, wrapped in containers, unable to negatively impact each other, enabling multiple applications and multiple users to live in harmony in the same shared environment.

If Mesos is a distributed kernel, what are the other operating system services?

Init System

How do you start up long lived applications? How do you restart them if they crash? What's the parent process that hosts other applications? On Mesos, the answer is Marathon: a distributed init system that runs applications in containers. It's simple; it accepts docker containers; and it's low level enough to run other applications, even ones that want to spawn other processes (Mesos tasks).

Storage

Compute resources (cpu, memory, processes) are core to an operating system, but you can't have a full OS without device drivers that talk to hard drives or other storage abstractions. In the distributed Mesos datacenter storage takes many forms. The most familiar of those is HDFS. It looks and acts very much like a traditional file system. It's easy to migrate to, and maintains multiple copies of stored data to survive individual drive failures. But it's not the only player. Need a document store? Hadoop works on Mesos. Need robust key/value storage? Cassandra works on Mesos. Need a SQL database? MySQL (Mysos) runs on Mesos.

Jobs & Time-Based Jobs

Not all applications are long lived. Some are ephemeral. For chaining those jobs together, use Spark. Need to run your jobs at a certain time, regularly? On a linux OS, cron handles those jobs. On a distributed Mesos datacenter, Chronos handles fault-tolerant time-dependent job scheduling. Got MapReduce workloads? YARN (Myriad) works on Mesos too.

Logging

Every app logs differently. You can easily log to one of the storage services, or you can use one of several logging or metrics services. Need a publish/subscribe data stream? Kafka runs on Mesos. Need a data stream backbone? Run Flume in Marathon and pipe the data to Kafka. Need real-time distributed search and analytics engine? ElasticSearch works on Mesos. 

User Space

Not all applications are run by the OS init system. User space applications are often more complicated, run on-demand, and have multiple components and service dependencies. For that, use Kubernetes.

Process Dashboard

The penultimate layer of this new ecosystem is the management of this new form factor. How do you observe something that's bigger than you? You have to step back and simplify. You have to take all the resource consumption data and distributed application process metrics and display them in a readable, browsable, navigable format. You need DCOS, which wraps Mesos to provide just that.

Package Management

Want to install all these newfangled datacenter applications and services? DCOS again comes to the rescue, with its command line package manager and app-store-like Mesosphere Universe.

The datacenter is more powerful than ever, and now you have the tools to take advantage of it.

Discuss on Hacker News