is a cluster manager that provides efficient resource
isolation and sharing across distributed applications,
It can run Hadoop,
Spark (a new
framework for low-latency interactive and
iterative jobs), and other applications.
Mesos is open source in the Apache Incubator.
You can use Mesos to:
- Run Hadoop, MPI, Spark, and other frameworks on a dynamically shared pool of nodes.
- Run multiple instances of Hadoop on the same cluster to
isolate production and experimental jobs,
or even multiple versions of Hadoop.
- Run long-lived services (e.g. Hypertable and HBase) on the same nodes
as batch applications and share resources between them.
- Build new cluster computing frameworks without reinventing low-level
facilities for farming out tasks, and have them coexist with
- Fault-tolerant replicated master using
- Scalability to 10,000s of nodes using fast, event-driven C++ implementation.
- Isolation between tasks with
- Multi-resource scheduling (memory and CPU aware).
- Efficient application-controlled scheduling mechanism (resource offers) that lets frameworks achieve their own placement goals (e.g. data locality).
- Java, Python and C++ APIs for developing new parallel applications.
- Web UI for viewing cluster state.
Mesos is being used to manage clusters at
UC Berkeley, and
UC San Francisco.
Mesos started out as a research project in the
UC Berkeley AMP Lab before
being open sourced and joining the Apache Incubator.
You can find more about the research behind Mesos, including our papers,
on our research page.
Apache Incubator Notice
Apache Mesos is an effort undergoing incubation at The Apache Software
Foundation (ASF), sponsored by the name of Apache Incubator. Incubation is
required of all newly accepted projects until a further review indicates that
the infrastructure, communications, and decision making process have stabilized
in a manner consistent with other successful ASF projects. While incubation
status is not necessarily a reflection of the completeness or stability of the
code, it does indicate that the project has yet to be fully endorsed by the