Differences between revisions 2 and 3
Revision 2 as of 2008-01-11 01:08:03
Size: 1221
Editor: 216
Revision 3 as of 2009-09-20 23:09:36
Size: 1221
Editor: localhost
Comment: converted to 1.6 markup
No differences found!

Options for intranet crawling that are not enabled by default

Here are some options you might want to add to your conf/nutch-site.xml configuration file if you plan on crawling your local network intranet that are not enabled by default.

Enable additional parser plugins


This will enable the parser plugins for text, html, javascript, pdf, excel, powerpoint, word, pdf, rss and zip. There are additional parsers you can enable which are listed in conf/parse-plugins.xml. If you have additional document types you wish to parse and they are listed in the parse-plugins file, just add them to the list.

Increase the file size fetch limit

                <name>http.content.limit</name> <value>2097152</value>

This will increase the default file size fetching limit to 2 megabytes. If your documents are larger (such as PDFs) then increase the number appropriately.

NonDefaultIntranetCrawlingOptions (last edited 2011-07-27 21:45:45 by LewisJohnMcgibbney)