Running Hama Over InfiniBand

Create an SDP configuration file

Each BSP peer (task) will have its own port number for communicating with others. For example, if you set set bsp.peer.port to 61000 and set bsp.tasks.maximum to 5, BSP peers will use the port range from 61000 to 61005.

To enable SDP[1] support for BSP peers, create a configuration file like the following sample:

[edward@192.168.10.1 ~]# cat {$HAMA_HOME}/conf/sdp.conf
# Use SDP when binding to
bind 192.168.10.1 61000-61010

# Use SDP when connecting to
connect 192.168.10.0/24 61000-61010

NOTE: This configuration file must reside on every node.

Set the system property that specifies the location of the configuration file

And then, set the location of the configuration file by adding -Dcom.sun.sdp.conf to the bsp.child.java.opts property as following:

  <property>
    <name>bsp.child.java.opts</name>
    <value>-Dcom.sun.sdp.conf={$HAMA_HOME}/conf/sdp.conf -Xmx1024m</value>
    <description>Java opts for the groom server child processes.  
    The following symbol, if present, will be interpolated: @taskid@ is replaced 
    by current TaskID. Any other occurrences of '@' will go unchanged.
    For example, to enable verbose gc logging to a file named for the taskid in
    /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
          -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc
    The configuration variable bsp.child.ulimit can be used to control the
    maximum virtual memory of the child processes. 
    </description>
  </property>

If everything is OK, you will see the following logs.

CONNECT to 192.168.10.2:61005 (socket converted to SDP protocol)
CONNECT to 192.168.10.5:61002 (socket converted to SDP protocol)
CONNECT to 192.168.10.4:61003 (socket converted to SDP protocol)
CONNECT to 192.168.10.1:61006 (socket converted to SDP protocol)
...

1. http://en.wikipedia.org/wiki/Sockets_Direct_Protocol

RunningHamaOverInfiniBand (last edited 2012-10-23 11:21:22 by edwardyoon)