DEPRECATED

dedup is an alias for org.apache.nutch.indexer.DeleteDuplicates

Deletes duplicate documents in a set of Lucene indexes. Duplicates have either the same contents (via MD5 hash) or the same URL.

Usage: bin/nutch org.apache.nutch.indexer.DeleteDuplicates (-local | -ndfs <namenode:port>) [-workingdir <workingdir>] <segmentsDir>

CommandLineOptions

bin/nutch_dedup (last edited 2014-05-16 11:02:53 by JulienNioche)