MR-279 branch is merged into mapreduce trunk and this changes things a bit for developing on mapreduce.

You can get all the help that is needed from (1) the README at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/README and (2) the INSTALL file at http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/INSTALL. Reproducing some of those contents here for the short-term lookup.


Checking out source code


svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce


Directory structure


trunk/

trunk/hadoop-mapreduce - Classic code. JT/TT reside here

trunk/hadoop-mapreduce/ - New code related to yarn reside here.

server libraries and tests.


Building


Building yarn code and install into the local maven cache.

Building classic code once yarn code is built.


Eclipse


"mvn eclipse:eclipse" and then import the checked out source root as a maven project.

importing as java project should continue to work.


FAQ


1) Build fails with "[ERROR] Failed to execute goal org.codehaus.mojo:make-maven-plugin:1.0-beta-1:autoreconf (autoreconf) on project hadoop-yarn-server-nodemanager: autoreconf command returned an exit value != 0. Aborting build; see debug output for more information. -> [Help 1]"

This means that you don't have the autotool chain necessary for building the native code. You will need to build the native code to build LinuxContainerExecutor needed for running the cluster with security enabled.

If you are not interested in running it with security enabled, you can skip building the native code by passing "-P-cbuild".

2) Build fails with "[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on project hadoop-yarn-api: Command execution failed. Process exited with an error: 1(Exit value: 1) -> [Help 1]"

This means that you don't have protoc installed on your machine. Installing protoc (and adding a non-standard installation directory to your LD_LIBRARY_PATH) should get it working.

3) Getting errors while compiling protobuf 2.4 For compiling YARN, You need to have protobuf 2.4.0a or higher (Download from http://code.google.com/p/protobuf/downloads/list). Install protobuf 2.4.0a or higher (Download from http://code.google.com/p/protobuf/downloads/list)

Installing protoc requires gcc 4.1.x or higher. If the make step fails with (Valid until a fix is released for protobuf 2.4.0a)

Hope that helps. If you run into issues, please send an email or create a JIRA issue.

Building on Linux

Linux distributions may install protocol buffers via their repostories. This can save all the installation problems, or it can cause extra problems

Problem: Protoc compiler fails on Linux if you see an error like

protoc: error while loading shared libraries: libprotobuf.so.7: cannot open shared object file: No such file or directory

It may be that the installation already has an older copy of protocol buffers installed, and this is getting in the way of the newly installed version

Test: run protoc --version to see what version is picked up. Here is the example of a valid version $ protoc --version libprotoc 2.4.1 }}}

If a version older than 2.4.0 appears, you need to uninstall protoc and possibly libprotoc using your platform's package management tools.