Welcome to the Apache Nutch Wiki

Please contribute your knowledge about Nutch here!

Contents

Nutch Version Administration

NutchTutorial - How to configure Nutch to crawl in local mode and post to Apache Solr for search/index.
Hadoop Tutorial Nutch being based Hadoop, it helps to have a better understanding of Hadoop.
Nutch Hadoop Tutorial - How to setup and run Nutch in deploy mode over a Hadoop cluster.
RunNutchInEclipse - How to configure, build, crawl and debug Nutch within Eclipse
Intranet Document Search - Index and search Microsoft Office, PDF etc. documents in a file system hierarchy with a Solr backend.

OverviewDeploymentConfigs $/!\$ :This full page requires a complete update to reflect recent Nutch releases: $/!\$
NutchConfigurationFiles
HttpAuthenticationSchemes - How to enable Nutch to authenticate itself using NTLM, Basic or Digest authentication schemes.
NonDefaultIntranetCrawlingOptions - Desirable options to add to your Nutch intranet crawling configuration.
OptimizingCrawls - How to optimise your crawling/fetching speed with Nutch.
ErrorMessages -- What they mean and suggestions for getting rid of them. $/!\$ :This requires extensive updating to reflect recent Nutch releases. In addition the legacy indexing and searching material should be archived. $/!\$
SetupProxyForNutch - using Tinyproxy on Ubuntu
IndexStructure $/!\$ :This page needs a slight update to provide more information on plugins and the data they send to Solr for indexing: $/!\$

Nutch Website
Features $/!\$ :TODO:This needs to be completely overhauled to reflect recent Nutch features. $/!\$
Current Nutch Gotchas
PublicServers running Nutch
Presentations on Nutch
Press Articles
Evaluations of Search Quality
Commercial Support and developers for hire
Mailing Lists
AcademicArticles that deal with Nutch
FAQ
HardwareRequirements
NutchResources

Becoming a Nutch Developer - Start developing and contributing to Nutch.
PluginCentral -- How to write your own plugins and use other people's.
InternalDocumentation -- How Nutch works.
Nutch Version Control
FixingOpicScoring - In planning.
HowToContribute
TaskList -- Tasks for Nutch developers. $/!\$ :Severe update required: $/!\$
Committer's_Rules -- Committers should follow these guidelines when deciding, which branch to use for committing the patches and when to commit.
Release_HOWTO
Website_Update_HOWTO
Image_Search_Design
StrategicGoals
Getting_Started
ApacheConUs2009MeetUp - List of topics for MeetUp at ApacheCon US 2009 in Oakland (Nov 2-6)
Using Nutch as a Maven dependency

Nutch2Roadmap -- Discussions on the architecture and features of Nutch 2.0
NewScoring -- New stable pagerank like webgraph and link-analysis jobs.
NewScoringIndexingExample -- Two full fetch cycles of commands using new scoring and indexing systems.
GORA_HBase -- Configuring Nutch 2.0 with GORA and HBASE
Build Nutch 2.0 in Eclipse -- How to setup your IDE environment comfortably.
ErrorMessagesInNutch2 -- What they mean and suggestions for getting rid of them.

This Wiki is a collaborative site, anyone can contribute and share:

Create an account by clicking the "Login" link at the top of any page, and picking a username and password.
Edit any page by pressing Edit at the top or the bottom of the page

There are some conventions used on the Nutch wiki:

$/!\$ :TODO: $/!\$ (/!\ :TODO: /!\ ) is used to denote sections that definitely need to be cleaned up.

Some general info on using this Wiki Software:

Create a link to another page with joined capitalized words (like WikiSandBox) or with ["quoted words in brackets"]
See HelpForBeginners to get you going, HelpContents for all help pages.
HelpOnMoinWikiSyntax: quick access to wiki syntax