FrontPage

Hadoop is a framework for running applications on large clusters built of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named Map/Reduce, where the application is divided into many small fragments of work, each of which may be executed or reexecuted on any node in the cluster. In addition, it provides a distributed file system (HDFS) that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both Map/Reduce and the distributed file system are designed so that node failures are automatically handled by the framework.

General Information

• Overview of Hadoop

• FAQ

• Official Hadoop Core Website : download, bug-tracking, mailing-lists, etc.

• Presentations and articles about Hadoop

• PoweredBy, a list of sites and applications powered by Hadoop

• Support

  • Getting help from the hadoop community.

  • People and companies for hire.

User Documentation

• ImportantConcepts

• GettingStartedWithHadoop (lots of details and explanation)

• QuickStart (for those who just want it to work now)

• Running Hadoop On Ubuntu Linux (Single-Node Cluster) (tutorial on installing, configuring and running Hadoop on a single machine)

• HowToConfigure Hadoop software

• WebApps for monitoring your system

• Command Line Options for hadoop shell script.

• Hadoop Code Overview

• How to handle name node failure

• How to read from and write to HDFS

• [TroubleShooting] What do when things go wrong

• How to get metrics into ganglia

• Map/Reduce

  • HadoopMapReduce

  • HadoopMapRedClasses

  • HowManyMapsAndReduces

  • TaskExecutionEnvironment

  • HowToDebugMapReducePrograms

• Examples

  • WordCount

  • Python Word Count

  • C/C++ Word Count

  • Grep

  • Sort

  • RandomWriter

• Amazon

  • Running Hadoop on AmazonEC2

  • Running Hadoop with AmazonS3

• Benchmarks

  • Hardware benchmarks

  • Data processing benchmarks

• Contrib

  • HadoopStreaming (Useful for using Hadoop with other programming languages)

  • Hbase, a Bigtable-like structured storage system for Hadoop HDFS

  • DistributedLucene, a Proposal for a distributed Lucene index in Hadoop

  • MountableHDFS, Fuse-DFS & other Tools to mount HDFS as a standard filesystem on Linux (and some other Unix OSs)

Developer Documentation

• Roadmap, listing release plans.

• HowToContribute

• HowToCodeReview

• Jira usage guidelines

• HowToCommit

• HowToRelease

• HudsonBuildServer

• DevelopmentHints

• ProjectSuggestions

• Building/Testing under IntelliJ IDEA

Related Resources

• Nutch Hadoop Tutorial (Useful for understanding Hadoop in an application context)

• IBM MapReduce Tools for Eclipse (An Eclipse plug-in that simplifies the creation and deployment of MapReduce programs)

• Hadoop IRC channel is #hadoop at irc.freenode.net.

• Using Spring and Hadoop (Discussion of possibilities to use Hadoop and Dependency Injection with Spring)

• Heart, a Planet-Scale RDF Data Store and a Distributed Processing Engine

• Hama, a Parallel Matrix Computational Package based on Hadoop Map/Reduce