"mergelinkdb" is an alias for "org.apache.nutch.crawl.LinkDbMerger"

Merges several LinkDb(s) together. URLFilters can be optionaly used to filter out specific content.

This tool can be useful if you built partial LinkDb(s) from groups of segments, and then you need to integrate them into one (e.g. for indexing or for searching). Or you can use it with a single LinkDb, just to filter out unwanted URLs and links.

It's possible to use this tool just for filtering - in that case only one LinkDb should be specified in arguments.

If more than one LinkDb contains information about the same URL, all inlinks are accumulated, but only at most db.max.inlinks inlinks will ever be added.

If activated, URLFilters will be applied to both the target URLs and to any incoming link URL. If a target URL is prohibited, all inlinks to that target will be removed, including the target URL. If some of incoming links are prohibited, only they will be removed, and they won't count when checking the above-mentioned maximum limit.

Usage

Configuration Files

Other Files

Caveats and Notes

DevelopmentCommandLineOptions

nutch-0.8-dev/bin/nutch_mergelinkdb (last edited 2009-09-20 23:09:53 by localhost)