Differences between revisions 301 and 302
Revision 301 as of 2015-09-26 09:19:10
Size: 9909
Editor: JulienNioche
Comment: Added links to tutorial on CloudSearch and video EMR
Revision 302 as of 2016-02-19 20:46:09
Size: 9926
Comment: update Javadoc links to valid URLs
Deletions are marked like this. Additions are marked like this.
Line 33: Line 33:
 * [[https://nutch.apache.org/apidocs-1.9/index.html|JavaDocs]] -- The !JavaDocs for the most recent Nutch-1.X release.
 * [[https://nutch.apache.org/apidocs-2.2.1/index.html|JavaDocs]] -- The !JavaDocs for the most recent Nutch-2.X release.
 * [[https://nutch.apache.org/apidocs/apidocs-1.11/index.html|JavaDocs]] -- The !JavaDocs for the most recent Nutch-1.X release.
 * [[https://nutch.apache.org/apidocs/apidocs-2.3.1/index.html|JavaDocs]] -- The !JavaDocs for the most recent Nutch-2.X release.

Welcome to the Apache Nutch Wiki


Please contribute your knowledge about Nutch here!

If you would like to update any content, would like to add your own content or would like to see something added then please

What is Apache Nutch?

Apache Nutch is a highly extensible and scalable open source web crawler software project. Stemming from Apache Lucene, the project has diversified and now comprises two codebases, namely:

  • Nutch 1.x: A well matured, production ready crawler. 1.x enables fine grained configuration, relying on Apache Hadoop data structures, which are great for batch processing.

  • Nutch 2.x: An emerging alternative taking direct inspiration from 1.x, but which differs in one key area; storage is abstracted away from any specific underlying data store by using Apache Gora for handling object to persistent mappings. This means we can implement an extremely flexibile model/stack for storing everything (fetch time, status, content, parsed text, outlinks, inlinks, etc.) into a number of NoSQL storage solutions.

Being pluggable and modular of course has it's benefits, Nutch provides extensible interfaces such as Parse, Index and ScoringFilter's for custom implementations e.g. Apache Tika for parsing. Additionally, pluggable indexing exists for Apache Solr, Elastic Search, etc.

Nutch can run on a single machine, but gains a lot of its strength from running in a Hadoop cluster

You can download Nutch here.

For more information about Apache Nutch, please see the Nutch wiki.

Nutch is a project of the Apache Software Foundation and is part of the larger Apache community of developers and users.

Nutch Version Administration


Nutch 1.X tutorial(s)

Nutch 2.X tutorial(s)

Other Tutorial(s)


  • OverviewDeploymentConfigs /!\ :This full page requires a complete update to reflect recent Nutch releases: /!\

  • NutchConfigurationFiles: An overview from Nutch developers.

  • NutchPropertiesCompleteList: A fine grained account of all Nutch property configuration.

  • HttpAuthenticationSchemes - How to enable Nutch to authenticate itself using NTLM, Basic or Digest authentication schemes.

  • NonDefaultIntranetCrawlingOptions - Desirable options to add to your Nutch intranet crawling configuration.

  • OptimizingCrawls - How to optimise your crawling/fetching speed with Nutch.

  • ErrorMessages -- What they mean and suggestions for getting rid of them. /!\ :This requires extensive updating to reflect recent Nutch releases. In addition the legacy indexing and searching material should be archived. /!\

  • IndexStructure /!\ :This page needs a slight update to provide more information on plugins and the data they send to Solr for indexing: /!\

General Information

Nutch Development

Nutch 2.x

Pre Nutch 1.3 and Archive

How to edit this Wiki

This Wiki is a collaborative site, anyone can contribute and share:

  • Create an account by clicking the "Login" link at the top of any page, and picking a username and password.
  • Edit any page by pressing Edit at the top or the bottom of the page

There are some conventions used on the Nutch wiki:

  • /!\ :TODO: /!\ (/!\ :TODO: /!\ ) is used to denote sections that definitely need to be cleaned up.

Some general info on using this Wiki Software:

FrontPage (last edited 2018-09-27 15:44:39 by RoannelFernandez)