Hi list,

Does anybody know if the Suggester component is designed to work with shards?

I'm asking because the documentation implies that it should (since ...Suggester reuses much of the SpellCheckComponent infrastructure…, and the SpellCheckComponent is documented as supporting a distributed setup).

But when I make a request, I get an exception:

java.lang.NullPointerException
at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:493)
at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:390)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:289)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Looking at the QueryComponent.java:493 code, I see:

        SolrDocumentList docs = (SolrDocumentList)srsp.getSolrResponse().getResponse().get("response");

        // calculate global maxScore and numDocsFound
        if (docs.getMaxScore() != null) {             <<<<  This is line 493

So I'm assuming the "docs" variable is null, which would happen if there is no "response" element in the Solr response.

If I make a direct request to the request handler in one core (e.g. http://hostname:8080/solr/core0/select?qt=suggest-core&q=rad), the query works.

But I see that there's no element named "response", unlike a regular query.

<response>
  <lst name="responseHeader">
    <int name="status">0</int>
    <int name="QTime">1</int>
  </lst>
  <lst name="spellcheck">
    <lst name="suggestions">
      <lst name="rad">
        <int name="numFound">10</int>
        <int name="startOffset">0</int>
        <int name="endOffset">3</int>
        <arr name="suggestion">
          <str>radair</str>
          <str>radar</str>
        </arr>
      </lst>
    </lst>
  </lst>
</response>

So I'm wondering if my configuration is just borked and this should work, or the fact that the Suggester doesn't return a response field means that it just doesn't work with shards.
Thanks,
-- Ken
--------------------------------------------
http://about.me/kkrugler
+1 530-210-6378
--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

I should have also included one more bit of information.

If I configure the top-level (sharding) request handler to use just the suggest component as such:

  <requestHandler name="/suggest" class="org.apache.solr.handler.component.SearchHandler">
    <!-- default values for query parameters -->
    <lst name="defaults">
      <str name="echoParams">explicit</str>
      <str name="shards.qt">suggest-core</str>
      <str name="shards">localhost:8080/solr/core0/,localhost:8080/solr/core1/</str>
    </lst>
    
    <arr name="components">
      <str>suggest</str>
    </arr>
  </requestHandler>

Then I don't get a NPE, but I also get a response with no results.

<response>
  <lst name="responseHeader">
    <int name="status">0</int>
    <int name="QTime">0</int>
    <lst name="params">
      <str name="q">r</str>
    </lst>
  </lst>
</response>

For completeness, here are the other pieces to the solrconfig.xml puzzle:

  <requestHandler class="org.apache.solr.handler.component.SearchHandler" name="suggest-core">
    <lst name="defaults">
      <str name="spellcheck">true</str>
      <str name="spellcheck.dictionary">suggest-one</str>
      <str name="spellcheck.count">10</str>
    </lst>
    
    <arr name="components">
      <str>suggest</str>
    </arr>
  </requestHandler>
  
  <searchComponent class="solr.SpellCheckComponent" name="suggest">
    <lst name="spellchecker">
      <str name="name">suggest-one</str>
      <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
      <str name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
      <str name="field">name</str>  <!-- the indexed field to derive suggestions from -->
      <float name="threshold">0.05</float>
      <str name="buildOnCommit">true</str>
    </lst>
    <lst name="spellchecker">
      <str name="name">suggest-two</str>
      <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
      <str name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup</str>
      <str name="field">content</str>  <!-- the indexed field to derive suggestions from -->
      <float name="threshold">0.0</float>
      <str name="buildOnCommit">true</str>
    </lst>
  </searchComponent>

Thanks,

-- Ken

On May 1, 2012, at 3:48pm, Ken Krugler wrote:

> Hi list,
>
> Does anybody know if the Suggester component is designed to work with shards?
>
> I'm asking because the documentation implies that it should (since ...Suggester reuses much of the SpellCheckComponent infrastructure…, and the SpellCheckComponent is documented as supporting a distributed setup).
>
> But when I make a request, I get an exception:
>
> java.lang.NullPointerException
> at org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:493)
> at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:390)
> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:289)
> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
> at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
> at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
> at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)

Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

On Tue, May 1, 2012 at 6:48 PM, Ken Krugler <[EMAIL PROTECTED]> wrote:
> Hi list,
>
> Does anybody know if the Suggester component is designed to work with shards?
>

I'm not really sure it is? They would probably have to override the
default merge implementation specified by SpellChecker.

But, all of the current suggesters pump out over 100,000 QPS on my
machine, so I'm wondering what the usefulness of this is?

And if it was useful, merging results from different machines is
pretty inefficient, for suggest you would shard by term instead so
that you need only contact a single host?
--
lucidimagination.com

Hi Robert,

On May 1, 2012, at 7:07pm, Robert Muir wrote:

> On Tue, May 1, 2012 at 6:48 PM, Ken Krugler <[EMAIL PROTECTED]> wrote:
>> Hi list,
>>
>> Does anybody know if the Suggester component is designed to work with shards?
>
> I'm not really sure it is? They would probably have to override the
> default merge implementation specified by SpellChecker.

What confuses me is that Suggester says it's based on SpellChecker, which supposedly does work with shards.

> But, all of the current suggesters pump out over 100,000 QPS on my
> machine, so I'm wondering what the usefulness of this is?
>
> And if it was useful, merging results from different machines is
> pretty inefficient, for suggest you would shard by term instead so
> that you need only contact a single host?

The issue is that I've got a configuration with 8 shards already that I'm trying to leverage for auto-complete.

My quick & dirty work-around would be to add a custom response handler that wraps the suggester, and returns results with the fields that the SearchHandler needs to do the merge.

-- Ken

--------------------------
Ken Krugler
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Mahout & Solr

On Wed, May 2, 2012 at 12:16 PM, Ken Krugler
<[EMAIL PROTECTED]> wrote:

> What confuses me is that Suggester says it's based on SpellChecker, which supposedly does work with shards.
>

It is based on spellchecker apis, but spellchecker's ranking is based
on simple comparators like string similarity, whereas suggesters use
weights.

when spellchecker merges from shards, it just merges all their top-N
into one set and recomputes this same distance stuff over again.

so, suggester can't possibly work like this correctly (forget about
any technical details), as how can it make assumptions about these
weights you provided. if they were e.g. log() weights from your query
logs then it needs to do log-summation across the shards, etc for the
final combined weight to be correct. This is specific to how you
originally computed the weights you gave it. it certainly cannot be
recomputing anything like spellchecker does :)

Anyways, if you really want to do it, maybe
https://issues.apache.org/jira/browse/SOLR-2848 is helpful. The
background is in 3.x there is really only one spellchecker impl
(AbstractLucene or something like that). I don't think distributed
spellcheck works with any other SpellChecker subclasses in 3.x, i
think its "wired" to only work with the Abstract-Lucene ones.

When we added another subclass to 4.0, DirectSpellChecker, he saw that
it was broken here and cleaned up the APIs so that spellcheckers can
override this merge() operation. Unfortunately I forgot to commit
those refactorings James did (which lets any spellchecker override
merge()ing) to the 3.x branch, but the ideas might be useful.

--
lucidimagination.com