What is Distributed Search?
When an index becomes too large to fit on a single system, or when a single query takes too long to execute, an index can be split into multiple shards, and Solr can query and merge results across those shards.
If single queries are currently fast enough and one simply wishes to expand the capacity (queries/sec) of the search system, then standard whole index replication should be used.
Distributed Searching
The presence of the shards parameter in a request will cause that request to be distributed across all shards in the list. The syntax of shards is host:port/base_url[,host:port/base_url]*
Currently, only query requests will be distributed. This includes requests to the standard request handler (and subclasses such as the dismax request handler), and any other handler (org.apache.solr.handler.component.SearchHandler) using standard components that support distributed search.
The current components that support distributed search are
The Query component that returns documents matching a query
The Facet component, for facet.query and facet.field requests where facets are sorted by count (the default). Solr 1.4 and later also support sorting by name.
The Highlighting component
The Stats component
the Debug component
See also WritingDistributedSearchComponents
Distributed Searching Limitations
Documents must have a unique key
When duplicate doc IDs are received, Solr chooses the first doc and discards subsequent ones
No distributed idf (see
![[WWW]](/wiki/modern/img/moin-www.png)
http://wunderwood.org/most_casual_observer/2007/04/progressive_reranking.html ) Doesn't support QueryElevationComponent
The index could change between stages, e.g. a document that matched a query and was subsequently changed may no longer match but will still be retrieved.
Doesn't currently support date faceting
Currently only supports sorted field facets (Solr 1.4+ supports both)
Number of shards is limited by number of characters allowed for GET method's URI; most web servers generally support at least 4000 characters, but limit still exists to prevent denial-of-service attacks.
Distributed Deadlock
Each shard may also serve top-level query requests and then make sub-requests to all of the other shards. In this configuration, care should be taken to ensure that the max number of threads serving HTTP requests in the servlet container is greater than the possible number of requests from both top-level clients and other shards. If this is not the case, a distributed deadlock is possible.
Consider the simplest case of two shards, each with just a single thread to service HTTP requests. Both threads could receive a top-level request concurrently, and make sub-requests to each other. Because there are no more remaining threads to service requests, the servlet containers will block the incoming requests until the other pending requests are finished (but they won't finish since they are waiting for the sub-requests).
Distributed Indexing
It's up to the user to distribute documents across shards. The easiest method to determine what server a document should be indexed at is to use something like uniqueId.hashCode() % numServers.
Example
For simple functionality testing, it's easiest to just set up two local Solr servers on different ports.
#make a copy cd solr cp -r example example7574 #change the port number perl -pi -e s/8983/7574/g example7574/etc/jetty.xml example7574/exampledocs/post.sh #in window 1, start up the server on port 8983 cd example java -server -jar start.jar #in window 2, start up the server on port 7574 cd example7574 java -server -jar start.jar #in window 3, index some example documents to each server cd example/exampledocs ./post.sh [a-m]*.xml cd ../../example7574/exampledocs ./post.sh [n-z]*.xml #now do a distributed search across both servers with your browser or curl curl 'http://localhost:8983/solr/select?shards=localhost:8983/solr,localhost:7574/solr&indent=true&q=ipod+solr'