Isolating YARN Applications in Docker Containers

The Docker executor for YARN involves work on YARN along with its counterpart in Docker to forge the necessary API end points. The purpose of this page is to collect related tickets across both projects in one location.

Motivation

The advantages Containers and Docker offer to Hadoop YARN are well understood. Here is a partial list.

Work items

Realizing these benefits requires changes to both Docker and YARN. Several of the necessary Docker features for the above such as excluding intermediate data directory from copy-on-write file system and adding data node Unix socket from host into the container for short-circuit IO are already available. The following new pieces of work needs to be done.