Questions
Can someone give an example of basic API-usage going against hbase?
Why do I see "java.io.IOException...(Too many open files)" in my logs?
How do I access HBase from my Ruby/Python/Perl/PHP/etc. application?
How do I create a table with a column family named "count" (or some other HQL reserved word)?
Why is HBase ignoring HDFS client configuration such as dfs.replication?
Answers
See ![[WWW]](/wiki/modern/img/moin-www.png){kind=small}
Bryan Duxbury's post on this topic.
2. Can someone give an example of basic API-usage going against hbase?
The two main client-side entry points are HBaseAdmin and HTable. Use HBaseAdmin to create, drop, list, enable and disable tables. Use it also to add and drop table column families. For adding, updating and deleting data, use HTable. Here is some pseudo code absent error checking, imports, etc., that creates a table, adds data, does a fetch of just-added data and then deletes the table.
// First get a conf object. This will read in the configuration
// that is out in your hbase-*.xml files such as location of the
// hbase master node.
HBaseConfiguration conf = new HBaseConfiguration();
// Create a table named 'test' that has two column families,
// one named 'content, and the other 'anchor'. The colons
// are required for column family names.
HTableDescriptor desc = new HTableDescriptor("test");
desc.addFamily(new HColumnDescriptor("content:"));
desc.addFamily(new HColumnDescriptor("anchor:"));
HBaseAdmin admin = new HBaseAdmin(conf);
admin.createTable(desc);
HTableDescriptor[] tables = admin.listTables();
// New table should be in list of returned tables.
// Or you could call admin.exists();
HTable table = new HTable(conf, "test");
// Add content to 'column:' on a row named 'row_x'
String row = "row_x";
BatchUpdate update = new BatchUpdate(row);
update.put("content:", Bytes.toBytes("some content");
table.commit(update);
// Now fetch the content just added
byte data[] = table.get(row, "content:");
// Delete the table.
admin.deleteTable(desc.getName());
For further examples, check out the hbase unit tests. These are probably your best source for sample code. Start with the code in org.apache.hadoop.hbase.TestHBaseCluster. It does a general table setup and then performs various client operations on the created table: loading, scanning, deleting, etc.
Don't forget your client will need a running hbase instance to connect to (See the Getting Started section toward the end of this Hbase Package Summary page).
See Hbase/Jython for the above example code done in Jython
3. What other hbase-like applications are there out there?
Apart from Google's bigtable, here are ones we know of:
PNUTS, a Platform for Nimble Universal Table Storage, being developed internally at Yahoo!
- Amazon SimpleDB is a web service for running queries on structured data in real time.
"
Hypertable is an open source project based on published best practices and our own experience in solving large-scale data-intensive tasks" "
Cassandra is a distributed storage system for managing structured data while providing reliability at a massive scale."
4. Can I fix OutOfMemoryExceptions in hbase?
Out-of-the-box, hbase uses the default JVM heap size. Set the HBASE_HEAPSIZE environment variable in ${HBASE_HOME}/conf/hbase-env.sh if your install needs to run with a larger heap. HBASE_HEAPSIZE is like HADOOP_HEAPSIZE in that its value is the desired heap size in MB. The surrounding '-Xmx' and 'm' needed to make up the maximum heap size java option are added by the hbase start script (See how HBASE_HEAPSIZE is used in the ${HBASE_HOME}/bin/hbase script for clarification).
5. How do I enable hbase DEBUG-level logging?
Either add the following line to your log4j.properties file -- log4j.logger.org.apache.hadoop.hbase=DEBUG -- and restart your cluster or, if running a post-0.15.x version, you can set DEBUG via the UI by clicking on the 'Log Level' link (but you need set 'org.apache.hadoop.hbase' to DEBUG without the 'log4j.logger' prefix).
6. Why do I see "java.io.IOException...(Too many open files)" in my logs?
Currently Hbase is a file handle glutton. Running an Hbase loaded w/ more than a few regions, its possible to blow past the common 1024 default file handle limit for the user running the process. Running out of file handles is like an OOME, things start to fail in strange ways. To up the users' file handles, edit /etc/security/limits.conf on all nodes and restart your cluster.
The math runs roughly as follows: Per column family, there is at least one mapfile and possibly up to 5 or 6 if a region is under load (lets say 3 per column family on average). Multiply by the number of regions per region server. So, for example, say you have a schema of 3 column familes per region and that you have 100 regions per regionserver, the JVM will open 3 * 3 * 100 mapfiles -- 900 file descriptors not counting open jar files, conf files, etc (Run 'lsof -p REGIONSERVER_PID' to see for sure).
7. What can I do to improve hbase performance?
A configuration that can help with random reads at some cost in memory is making the hbase.io.index.interval smaller. By default when hbase writes store files, it adds an entry to the mapfile index on every 32nd addition (For hadoop, default is every 128th addition). Adding entries more frequently -- every 16th or every 8th -- will make it so there is less seeking around looking for the wanted entry but at the cost of a hbase carrying a larger index (Indices are read into memory on mapfile open; by default there are one to five or so mapfiles per column family per region loaded into a regionserver).
Some basic tests making the io.bytes.per.checksum larger -- changing it from checksum-checking every 4096 bytes instead of every 512 bytes -- seem to have no discernible effect on performance.
8. How do I access Hbase from my Ruby/Python/Perl/PHP/etc. application?
- Description of how to launch a thrift service, client bindings and examples in ruby and C++ for connecting to Hbase
- REST Interface to Hbase
Hbase/Jython An example showing how to access HBase from Jython
9. How do I create a table with a column family named "count" (or some other HQL reserved word)?
Enclose the reserved word in single or double quotes and it should work. If you find an instance where this fails, please let us know.
Some example reserved words: count, table, insert select, delete, drop, truncate, where, row, into
To delete an explicit cell, add a delete record of the exact same timestamp (Use the commit that takes a timestamp when committing the BatchUpdate that contains your delete). Entering a delete record that is newer than the cell you would delete will also work when scanning and getting with timestamps that are equal or newer to the delete entry but there is nothing to stop you going behind the delete cell entry by specifying a timestamp that is older retrieving old entries.
If you want to delete all cell entries whenever they were written, use the HTable.deleteAll method. It will go find all cells and for each enter a delete record with a matching timestamp.
There is nothing to stop you adding deletes or puts with timestamps that are from the far future or of the distant past but doing so is likely to get you into trouble; its a known issue that hbase currently does not do the necessary work checking all stores to see if an old store has an entry that should override additions made recently.
11. What ports does HBase use?
Not counting the ports used by hadoop -- hdfs and mapreduce -- by default, hbase runs the master and its informational http server at 60000 and 60010 respectively and regionservers at 60020 and their informational http server at 60030. ${HBASE_HOME}/conf/hbase-default.xml lists the default values of all ports used. Also check ${HBASE_HOME}/conf/hbase-site.xml for site-specific overrides.
12. Why is HBase ignoring HDFS client configuration such as dfs.replication?
If you have made HDFS client configuration on your hadoop cluster, HBase will not see this configuration unless you do one of the following:
Add a pointer to your HADOOP_CONF_DIR to CLASSPATH in hbase-env.sh
Add a copy of hadoop-site.xml to ${HBASE_HOME}/conf, or
If only a small set of HDFS client configurations, add them to hbase-site.xml
The first option is the better of the three since it avoids duplication.
13. Any advice for smaller clusters in write-heavy environments?
14. Can I change the regionserver behavior so it, for example, orders keys other than lexicographically, etc.?
Yes, by subclassing HRegionServer. For example that orders the row return by column values, see
HBASE-605
15. Can I safely move the master from node A to node B?
Yes. HBase must be shutdown. Edit your hbase-site.xml configuration across the cluster setting hbase.master to point at the new location.
16. Can I safely move the hbase rootdir in hdfs?
Yes. HBase must be down for the move. After the move, update the hbase-site.xml across the cluster.
17 Can HBase development be done on windows?
See the quickstart page for Hadoop. The requirements for developing HBase on Windows is the same as for Hadoop.
18 What version of Hadoop do I need to run HBase?
Different versions of HBase require different versions of Hadoop. Consult the table below to find which version of Hadoop you will need:
|
HBase Release Number |
Hadoop Release Number |
|
0.1.x |
0.16.x |
|
0.2.x |
0.17.x |
|
0.18.x |
0.18.x |
|
0.19.x |
0.19.x |
Releases of Hadoop can be found here. We recommend using the most recent version of Hadoop possible, as it will contain the most bug fixes.
Note that HBase-0.2.x can be made to work on Hadoop-0.18.x. HBase-0.2.x ships with Hadoop-0.17.x, so to use Hadoop-0.18.x you must recompile Hadoop-0.18.x, remove the Hadoop-0.17.x jars from HBase, and replace them with the jars from Hadoop-0.18.x.
Also note that after HBase-0.2.x, the HBase release numbering schema will change to align with the Hadoop release number on which it depends.