Abstract

Aurora is a service scheduler used to schedule jobs onto Apache Mesos.

Proposal

Aurora is a scheduler that provides all of the primitives necessary to quickly deploy and scale stateless and fault tolerant services in a datacenter.

Aurora builds on top of Apache Mesos and provides common features that allow any site to run large scale production applications. While the project is currently used in production at Twitter, we wish to develop a community to increase contributions and see it thrive in the future.

Background

The initial development of Aurora was done at Twitter, and its codebase was recently open sourced. This proposal is for Aurora to join the Apache Incubator.

Rationale

While the Apache Mesos core focuses on distributing individual tasks across nodes in a cluster, typical services consist of dozens or hundreds of replicas of tasks. As a service scheduler, Aurora provides the abstraction of a "job" to bundle and manage these tasks. Aurora provides many key functionalities centered around a job, including: definition, the concept of an instance and the serverset, deployment and scheduling, health checking, and introspection. It also allows cross-cutting concerns to be handled like observability and log collection.

Current Status

Meritocracy

By submitting this incubator proposal, we’re expressing our intent to build a diverse developer community around Aurora that will conduct itself according to The Apache Way and use meritocratic means of accepting contributions. Several members of the Aurora team overlap with Apache Mesos, which successfully graduated from the Incubator and has embraced a meritocratic model of governance; we plan to follow a similar path forward with Aurora and believe that a synergy between both projects will make this even easier.

Community

Aurora is currently being used internally at Twitter. By open sourcing the project, we hope to extend our contributor base significantly and create a vibrant community around the project.

Core Developers

Aurora is currently being developed by a team of seven engineers at Twitter.

Alignment

The ASF is a natural choice to host the Aurora project, given the goal of open sourcing the project and fostering a community to grow and support the software. Additionally, Aurora integrates with Apache Mesos, and Apache ZooKeeper for service discovery.

We believe that inclusion within Apache will build stronger ties between these projects, and create further alignment between their goals and communities.

Known Risks

Orphaned Products

The core developers plan to continue working full time on the project, and there is very little risk of Aurora being abandoned since it is running hundreds of services as part of Twitter’s infrastructure. Additionally, members of the Mesos community beyond Twitter have expressed interest in an advanced scheduler like Aurora (see “Interested Parties” section); we believe that need will drive some of the community involvement necessary for the project to incubate successfully.

Inexperience with Open Source

Initial Aurora committers have varying levels of experience using and contributing to Open Source projects, however by working with our mentors and the Apache community we believe we will be able to conduct ourselves in accordance with Apache Incubator guidelines. The close relationship between the Aurora team and Apache Mesos means there is an awareness of the incubation process and a willingness to embrace The Apache Way.

Homogenous Developers

The initial set of committers are from a single organization, however we expect that once approved for incubation the project will attract contributors from more organizations. We have already had conversations with other companies who have expressed an interest in Aurora.

Reliance on Salaried Developers

Initial Aurora committers are salaried developers at Twitter, however shortly after open sourcing the code we plan to diversify the project’s core committers and contributors.

Relationships with Other Apache Products

Initially, Aurora has been developed as a scheduler for Apache Mesos. Additionally, it relies on ZooKeeper for service discovery, allowing servers to register at a location and clients to subsequently discover the servers.

An Excessive Fascination with the Apache Brand

While we respect the reputation of the Apache brand and have no doubts that it will attract contributors and users, our interest is primarily to give Aurora a solid home as an open source project following an established development model. We have also given reasons in the Rationale and Alignment sections.

Documentation

This proposal exists online as http://wiki.apache.org/incubator/AuroraProposal. Basic build instructions are included in the existing github repository, and the source code has thorough documentation. User documentation exists internally to Twitter, and as part of incubation will be adapted to share and improve user documentation overall.

Initial Source

JIRA Aurora (AURORA)

Source and Intellectual Property Submission Plan

A snapshot of the Aurora scheduler has been posted on GitHub for review. During incubation, the codebase will be available at https://github.com/apache/incubator-aurora/

External Dependencies

All Aurora dependencies have Apache compatible licenses.

Cryptography

Not applicable.

Required Resources

Mailing Lists

aurora-private for private PMC discussions
aurora-dev
aurora-commits

Subversion Directory

We prefer to use Git as our source control system: git://git.apache.org/aurora

Issue Tracking