Architecture

This effort is still a "work in progress". Please feel free to add comments.

Introduction

HAMA is a distributed framework on Hadoop for massive matrix and graph computations, currently being incubated as one of the incubator project by the Apache Software Foundation.

Goal

The Hama project goal is to provide easy matrix/graph computing programming environment on the Hadoop (distributed system). We are focusing on are as follows:

Compatibility
Scalability
Flexibility
Usability and Applicability

The overall architecture of HAMA

Below diagram is illustrates the overall architecture of HAMA.

               +--------------------------------------+
               |    Matrix/Graph Computing Program    |  User Applications
               +--------------------------------------+
           +------------------------------------------+
           |       HAMA : BSP, ............, etc      |  Computing Engines
           +------------------------------------------+ 
 +----------------------------------------------------+
 |                     ZooKeeper                      |  Distributed Locking Service
 +----------------------------------------------------+
 +----------------------------------------------------+
 |           Hadoop : HDFS, HBase, ..., etc           |  Distributed Storage Systems
 +----------------------------------------------------+

BSP framework

TODO: explain what is the BSP? and Hama BSP?

BSP Architecture/Design

One BSP cluster consists of one BSPMaster, multiple GroomServer and one or more zookeeper servers in a network environment.

work in progress HAMA-251,HAMA-258

BSPMaster

BSPMaster is responsible to do the following:

Maintaining groom server status. 2. Controlling super steps in a cluster. 3. Maintaining job progress information. 4. Scheduling Jobs and Assigning tasks to groom servers 5. Disseminating execution class across groom servers. 6. Controlling fault. 7. Providing users with the cluster control interface.

A BSP Master and multiple grooms are started by the script. Then, the bsp master starts up with a RPC server for groom servers. Groom servers starts up with a BSPPeer instance - later, BSPPeer needs to be integrated with GroomServer - and a RPC proxy to contact the bsp master. After started, each groom periodically sends a heartbeat message that encloses its groom server status, including maximum task capacity, unused memory, and so on.

Each time the bsp master receives a heartbeat message, it brings up-to-date groom server status - the bsp master makes use of groom servers' status in order to effectively assign tasks to idle groom servers - and returns a heartbeat response that contains assigned tasks and others actions that a groom server has to do. For now, we have a FIFO job scheduler and very simple task assignment algorithms.

GroomServer

A Groom Server (shortly referred to as groom) is a process that performs bsp tasks assigned by BSPMaster. Each groom contacts the BSPMaster, and it takes assigned tasks and reports its status by means of periodical piggybacks with BSPMaster. Each groom is designed to run with HDFS or other distributed storages. Basically, a groom server and a data node should be run on one physical node.

BSP Example

TODO: explain how to programming BSP program.
work in progress HAMA-244

    // BSP job configuration 
    BSPJob bsp = new BSPJob(); 
    // Set the job name 
    bsp.setJobName("BSP test job"); 

    // Set in/output path and formatter 
    bsp.setInputPath(conf, new Path("input path")); 
    bsp.setOutputPath(conf, new Path("output path")); 
    bsp.setInputFormat(MyInputFormat.class); 
    bsp.setOutputFormat(MyOutputFormat.class); 
     
    // Set the BSP code 
    bsp.setBSPCode(BSPProgram.class); 
    BSPJobClient.runJob(bsp);

---- Edit conflict - other version: ----

Hama Examples
- Matrix Inversion
- Pi Estimator
- PageRank

---- Edit conflict - your version: ----

---- End of edit conflict ----