Sunday, July 7, 2013

Introduction to Apache Mesos

Update : Interested in Mesos? You might find my book "Apache Mesos Essentials" useful.

As promised I am going to give a short introduction of mesos, sort of Mesos-101.

Mesos is a cluster management framework, which was started at UCB in 2009. It is in production use at many companies including Twitter. Since 2011, it is in incubation at apache and very recently graduated as a top level project. The official announcement of the graduation is yet to be done once formalities are over.

Mesos is inspired by the idea of sharing resources in a cluster between multiple frameworks while providing resource isolation. In modern clusters, the computing requirement of different frameworks are radically different and there is no single framework to rule them all. Instead, large organization needs ability to share data and resources between multiple frameworks. Both mesos and YARN are aiming to solve this problem in different ways. Following picture from Matei's talk describes it visually.

Currently, mesos can run, Hadoop, Spark, MPI and Hypertable as frameworks. Most notable features of Mesos are fault-tolerance and scalability. One important design decision is the use of Linux Containers for resource isolation which provides very strong isolation.

Scheduling being my area of research, I want to highlight the choice of the resource request model as one of the main difference between mesos and yarn. Mesos traded simplicity in implementing a framework for more flexible resource management. It uses two-level or decentralized scheduling model, where mesos delegates the task of scheduling to the frameworks. Mesos decides which framework gets how much resources and makes resource offers to frameworks. Frameworks can choose to accept or decline the offer. This model is great as frameworks can optimize on how their subtasks given resources. For example, framework like hadoop can use it to accept offers based on data locality. On the downside it burdens framework with more work as they are now complex to implement.

Mesos has many use-cases. It is a great tool for researchers as you can run multiple instances of same framework. This allows development and production clusters to be running on the same set of nodes.

There are lot of things to cover especially about the architecture and design choices made by mesos. But this being introductory post, I don't want to make it too complex (what ? It already is ? ;) If you are interested in more details the original paper published at NSDI'11 is a great way to start. (or you can wait for some time when I write next post :D )


Post a Comment

Copyright © 2014 Dharmesh Kakadia. Powered by Blogger.