Differences between revisions 5 and 6
⇤ ← Revision 5 as of 2012-06-20 10:55:20 →
Size: 2346
Editor: JulienNioche
Comment:
← Revision 6 as of 2012-06-20 13:02:08 → ⇥
Size: 2709
Editor: FerdyGalema
Comment: add NB about HBase test lib issue
Deletions are marked like this. Additions are marked like this.
Line 45: Line 45:
'''N.B.''' It's possible to encounter the following exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration; this is caused by the fact that sometimes the hbase TEST jar is deployed in the lib dir. To resolve this just copy the lib over from your installed HBase dir into the build lib dir. (This issue is currently in progress).

Nutch 2.0 Tutorial

http://www.interadvertising.co.uk/files/nutch_logo_medium.gif http://gora.apache.org/images/gora-logo.png http://hbase.apache.org/images/hbase_logo.png

This document describes how to get Nutch 2.0 to use HBase as a storage backend for Gora.

  • Grab a distribution of Nutch 2.X from here

  • Install and configure HBase. You can get it here (N.B. Gora 0.2 uses HBase 0.90.4, however the setup is known to work with more recent versions of the HBase 0.90.x branch)

  • Specify the GORA backend in nutch-site.xml 

<property>
 <name>storage.data.store.class</name>
 <value>org.apache.gora.hbase.store.HBaseStore</value>
 <description>Default class for storing data</description>
</property>
  • Ensure the HBase gora-hbase dependency is available in ivy/ivy.xml 

    <!-- Uncomment this to use HBase as Gora backend. -->
    
    <dependency org="org.apache.gora" name="gora-hbase" rev="0.2" conf="*->default" />
  • Ensure that HBaseStore is set as the default datastore in gora.properties 

    gora.datastore.default=org.apache.gora.hbase.store.HBaseStore

  • N.B. It's probably worth setting all your usual configuration settings within nutch-site.xml etc. before progressing.

  • Compile Nutch -> ant runtime

  • Make sure HBase is started and working properly as per the quick start tutorial here

You should then be able to use it. Try going to $NUTCH_HOME/runtime/local/bin and do : 

  nutch inject /someseedDir
  nutch readdb

You should find more details in the logs on $NUTCH_HOME/runtime/local/logs/hadoop.log.

N.B. It's possible to encounter the following exception: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration; this is caused by the fact that sometimes the hbase TEST jar is deployed in the lib dir. To resolve this just copy the lib over from your installed HBase dir into the build lib dir. (This issue is currently in progress).

N.B. The process of using the other datastore implementations offered within Gora e.g. Apache Cassandra, Accumulo and Sql, can be achieved simply by tweaking the above settings prior to compiling the Nutch code.

For more details of the command line interface options, please see here, or of course run ./bin/nutch which will print usage to std out. Finally, for a more detailed Nutch (1.X) tutorial, please see here

back to FrontPage

Nutch2Tutorial (last edited 2012-06-20 13:02:08 by FerdyGalema)