Running Hama Over InfiniBand
- Java 7+ required.
Create an SDP configuration file
Each BSP peer (task) will have its own port number for communicating with others. For example, if you set set bsp.peer.port to 61000 and set bsp.tasks.maximum to 5, BSP peers will use the port range from 61000 to 61005.
To enable SDP^[1]^ support for BSP peers, create a configuration file like the following sample:
[edward@192.168.10.1 ~]# cat {$HAMA_HOME}/conf/sdp.conf
# Use SDP when binding to
bind 192.168.10.1 61000-61010
# Use SDP when connecting to
connect 192.168.10.0/24 61000-61010
NOTE: This configuration file must reside on every node.
Set the system property that specifies the location of the configuration file
And then, set the location of the configuration file by adding -Dcom.sun.sdp.conf to the bsp.child.java.opts property as following:
<property>
<name>bsp.child.java.opts</name>
<value>-Dcom.sun.sdp.conf={$HAMA_HOME}/conf/sdp.conf -Xmx1024m</value>
<description>Java opts for the groom server child processes.
The following symbol, if present, will be interpolated: <at:var at:name="taskid" /> is replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/<at:var at:name="taskid" />.gc
The configuration variable bsp.child.ulimit can be used to control the
maximum virtual memory of the child processes.
</description>
</property>
If everything is OK, you will see the following logs.
CONNECT to 192.168.10.2:61005 (socket converted to SDP protocol) CONNECT to 192.168.10.5:61002 (socket converted to SDP protocol) CONNECT to 192.168.10.4:61003 (socket converted to SDP protocol) CONNECT to 192.168.10.1:61006 (socket converted to SDP protocol) ...