Apache Hadoop Hackathon, May 18, 2011

Hosted at Cloudera's San Francisco and Palo Alto offices.

This page is aliased at: http://bit.ly/hadoop-hack-may18

Useful resources

• HowToContribute

• EclipseEnvironment

• Previous hackathon notes: http://bit.ly/hadoop-hack-may11

• Eli's build scripts: https://github.com/elicollins/hadoop-dev

Quick Start

Checking out Hadoop: Git:

mkdir hadoop-git ; cd hadoop-git

git clone https://github.com/apache/hadoop-common.git
git clone https://github.com/apache/hadoop-hdfs.git
git clone https://github.com/apache/hadoop-mapreduce.git

(or if we fix ssh: #git clone git://git.apache.org/hadoop-common.git
#git clone git://git.apache.org/hadoop-mapreduce.git
#git clone git://git.apache.org/hadoop-hdfs.git
)

svn:

mkdir hadoop-svn ; cd hadoop-svn
svn co https://svn.apache.org/repos/asf/hadoop/common/trunk
svn co https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk
svn co https://svn.apache.org/repos/asf/hadoop/hdfs/trunk
(for trunk -- for branches, use /repos/asf/hadoop/common/branches/branch-0.22 )

Running tests

ant test-core -Dtest.output=yes -Dtestcase=TestEditLog

test.output will print output to console, useful for hanging tests

Eclipse: see EclipseEnvironment

Submitting a patch

Open a jira
Make change
Run tests
git diff --no-prefix > /tmp/HADOOP-1234.txt

Review queues

• Common: https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12311124&mode=hide

• HDFS: https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12313301&mode=hide

• MapReduce: https://issues.apache.org/jira/secure/IssueNavigator.jspa?requestId=12313302&mode=hide

Suggestions for what to work on

Infrastructure improvements

• Create a Hudson job that produces a release tarball: https://builds.apache.org/hudson/view/G-L/view/Hadoop/job/Hadoop-22-Build/

• Include 32-bit and 64-bit native libraries in Jenkins tarball builds: https://issues.apache.org/jira/browse/HADOOP-7283

Make it easier for others to contribute

• Improve documentation at HowToContribute, EclipseEnvironment

• Write instructions for other IDEs
• What's the most confusing thing you found about the contribution process? How can we improve it?

Help get 0.22 out the door

• Close out 0.22 blockers. Perhaps more appropriate for people with context.

• If those are too hard check out the other jiras for 0.22 Common, HDFS, MapReduce

Try to use the release (or build from trunk)

• Work on the documentation
  • Try out the current documentation
  • File jiras and submit fixes for bugs and improvements.
  • Eg config options that should be in the docs but are not..
  • Or have been deprecated and should be removed or updated.
  • Write new documentation that's needed (eg on FS config)

• Setup a small cluster on your laptop or in VMs or using Apache Whirr and bang on it.

Help get trunk in shape

• Help out with the SVN unsplit: https://issues.apache.org/jira/browse/HADOOP-7106. Git expertise is welcome!

• Review/commit patches in the review queues:

  • Common

  • HDFS

  • MapReduce

• Work out the kinks of HBase trunk on HDFS trunk
  • Eg HDFS-1103, HDFS-1152, HDFS-1139, HDFS-1056, HDFS-1060.
• Improve error and log messages
• Improve command line usability (eg error messages)

• Newbie JIRAs