SpideringDemo.java is an example of how to use httpunit http://httpunit.sourceforge.net/ to index web pages. There is no guarante on how well this works. It's document parsing is limited to html pages and is inspired by the demo (see lucene-demos.jar).