Updating a Solr Index with Rich Documents such as PDF and MS Office

Solr has an extensible

Solr accepts index updates in [http://en.wikipedia.org/wiki/Comma-separated_values CSV] (Comma Separated Values) format. Different separators are configurable, and multi-valued fields are supported.

TableOfContents

Requirements

Solr 1.2 is the first version with CSV support for updates. The CSV request handler needs to be configured in solrconfig.xml This should already be present in the example solrconfig.xml

  <!-- CSV update handler, loaded on demand -->
  <requestHandler name="/update/csv" class="solr.CSVRequestHandler" startup="lazy">
  </requestHandler>

How to Install

1) You need a couple patch files and zips of source and testcode that are attached to the JIRA issue at https://issues.apache.org/jira/browse/SOLR-284.

2) Download the libs.zip, rich.patch, test-files.zip, source.zip, and test.zip files.

3) Unzip the libs.zip into SOLR_HOME/lib. These are the jar's required for parsing the rich documents, using PDFBox and POI.

4) Unzip the test-files.zip into SOLR_HOME/test/test-files/. These are various test files for running the included unit tests.

5) Apply the rich.patch to your source. Rich.patch has tweaks that add the solr.RichDocumentRequestHandler to your solrconfig.xml.

6) Copy the contents of source.zip into SOLR_HOME/src/java/org/apache/solr/handler

7) Copy the contents of test.zip into SOLR_HOME/src/test/org/apache/solr/handler

8) Run ant test to verify everything is working!

Methods of uploading Binary records

Binary records may be uploaded to Solr by sending the data to the /solr/update/rich URL. All of the normal methods for [SolrContentStreams uploading content] are supported.

Example

There is a sample PDF file at src/test/test-files/simple.pdf that may be used to add a PDF to the solr example server.

Example of using HTTP-POST to send the CSV data over the network to the Solr server:

cd src/test/test-files/simple.pdf
curl http://localhost:8983/solr/update/rich --data-binary @simple.pdf -H 'Content-type:text/plain; charset=utf-8'

Uploading a binary file can be more efficient than sending it over the network via HTTP. Remote streaming must be enabled for this method to work. See the following line in solrconfig.xml, change it to enableRemoteStreaming="true", and restart Solr.

  <requestParsers enableRemoteStreaming="false" multipartUploadLimitInKB="2048" />

The following request will cause Solr to directly read the input file:

curl http://localhost:8983/solr/update/rich?stream.file=src/test/test-files/simple.pdf
#NOTE: The full path, or a path relative to the CWD of the running solr server must be used.

Parameters

Some parameters may be specified on a per field basis via f.<fieldname>.param=value

fieldnames

Specifies a comma separated list of field names to use when adding documents to the Solr index. If the CSV input already has a header, the names specified by this parameter will override them.

Example: fieldnames=id,name,category

overwrite

If true (the default), overwrite documents based on the uniqueKey field declared in the solr schema.

commit

Commit changes after all records in this request have been indexed. The default is commit=false to avoid the potential performance impact of frequent commits.

Disadvantages

There is no way to provide document or field index-time boosts with the CSV format, however many indicies do not utilize that feature.