FAQ for HBase Operations and Troubleshooting

Questions

  1. Can I fix OutOfMemoryExceptions in hbase?
  2. How do I enable hbase DEBUG-level logging?
  3. Why do I see "java.io.IOException...(Too many open files)" in my logs?
  4. What ports does HBase use?
  5. Why is HBase ignoring HDFS client configuration such as dfs.replication?
  6. Can I safely move the master from node A to node B?
  7. Can I safely move the hbase rootdir in hdfs?
  8. How do I add/remove a node?
  9. Why do servers have start codes?
  10. How do I monitor my HBase Cluster?

Answers

1. Can I fix OutOfMemoryExceptions in hbase?

Out-of-the-box, hbase uses a default of 1G heap size. Set the HBASE_HEAPSIZE environment variable in ${HBASE_HOME}/conf/hbase-env.sh if your install needs to run with a larger heap. HBASE_HEAPSIZE is like HADOOP_HEAPSIZE in that its value is the desired heap size in MB. The surrounding '-Xmx' and 'm' needed to make up the maximum heap size java option are added by the hbase start script (See how HBASE_HEAPSIZE is used in the ${HBASE_HOME}/bin/hbase script for clarification).

2. How do I enable hbase DEBUG-level logging?

Either add the following line to your log4j.properties file – log4j.logger.org.apache.hadoop.hbase=DEBUG – and restart your cluster or, if running a post-0.15.x version, you can set DEBUG via the UI by clicking on the 'Log Level' link (but you need set 'org.apache.hadoop.hbase' to DEBUG without the 'log4j.logger' prefix).

3. Why do I see "java.io.IOException...(Too many open files)" in my logs?

See the Troubleshooting section in the HBase Book http://hbase.apache.org/book.html#trouble

4. What ports does HBase use?

Not counting the ports used by hadoop – hdfs and mapreduce – by default, hbase runs the master and its informational http server at 60000 and 60010 respectively and regionservers at 60020 and their informational http server at 60030. ${HBASE_HOME}/conf/hbase-default.xml lists the default values of all ports used. Also check ${HBASE_HOME}/conf/hbase-site.xml for site-specific overrides.

5. Why is HBase ignoring HDFS client configuration such as dfs.replication?

If you have made HDFS client configuration on your hadoop cluster, HBase will not see this configuration unless you do one of the following:

  • Add a pointer to your HADOOP_CONF_DIR to CLASSPATH in hbase-env.sh or symlink your hadoop-site.xml from the hbase conf directory.
  • Add a copy of hadoop-site.xml to ${HBASE_HOME}/conf, or
  • If only a small set of HDFS client configurations, add them to hbase-site.xml

The first option is the better of the three since it avoids duplication.

6. Can I safely move the master from node A to node B?
Yes. HBase must be shutdown. Edit your hbase-site.xml configuration across the cluster setting hbase.master to point at the new location.

7. Can I safely move the hbase rootdir in hdfs?
Yes. HBase must be down for the move. After the move, update the hbase-site.xml across the cluster and restart.

8. How do I add/remove a node?

For removing nodes, see the section on decommissioning nodes in the HBase Book http://hbase.apache.org/book.html#decommission

Adding and removing nodes works the same way in HBase and Hadoop. To add a new node, do the following steps:

  1. Edit $HBASE_HOME/conf/regionservers on the Master node and add the new address. 2. Setup the new node with needed software, permissions. 3. On that node run $HBASE_HOME/bin/hbase-daemon.sh start regionserver 4. Confirm it worked by looking at the Master's web UI or in that region server's log.

Removing a node is as easy, first issue "stop" instead of start then remove the address from the regionservers file.

For Hadoop, use the same kind of script (starts with hadoop-*), their process names (datanode, tasktracker), and edit the slaves file. Removing datanodes is tricky, please review the dfsadmin command before doing it.

9. Why do servers have start codes?

If a region server crashes and recovers, it cannot be given work until its lease times out. If the lease is identified only by an IP address and port number, then that server can't do any progress until the lease times out. A start code is added so that the restarted server can begin doing work immediately upon recovery. For more, see https://issues.apache.org/jira/browse/HBASE-1156.

10. How do I monitor my HBase Cluster?

HBase emits performance metrics (see http://hbase.apache.org/metrics.html) that you can monitor with Ganglia. Alternatively, you could use SPM for HBase (see http://sematext.com/spm/hbase-performance-monitoring/index.html).

  • No labels