Hi,
I am wondering: will SOLR or Lucene use caches for fuzzy searches? I mean
per-term caching or something, internal to Lucene, or may be SOLR (SOLR may
use own query parser)...

Anyway, I implemented BK-Tree and playing with it right now, I altered
FuzzyTermEnum class of Lucene...
http://en.wikipedia.org/wiki/BK-tree

- it seems performance of fuzzy searches boosted at least hundred times, but
I need to do more tests... repeated similar (slightly different) queries run
with better performance, probably because of OS-level file caching... but it
could be that of BK-Tree distance! (although I need to use classic int
instead of float distance by Lucene/Levenstein etc.)

Thanks,
Fuad Efendi
+1 416-993-2060
http://www.tokenizer.ca/
Data Mining, Vertical Search

http://issues.apache.org/jira/browse/LUCENE-2230
Enjoy!
> -----Original Message-----
> From: Fuad Efendi [mailto:[EMAIL PROTECTED]]
> Sent: January-19-10 11:32 PM
> To: [EMAIL PROTECTED]
> Subject: SOLR Performance Tuning: Fuzzy Searches, Distance, BK-Tree
>
> Hi,
>
>
> I am wondering: will SOLR or Lucene use caches for fuzzy searches? I
> mean
> per-term caching or something, internal to Lucene, or may be SOLR (SOLR
> may
> use own query parser)...
>
> Anyway, I implemented BK-Tree and playing with it right now, I altered
> FuzzyTermEnum class of Lucene...
> http://en.wikipedia.org/wiki/BK-tree
>
> - it seems performance of fuzzy searches boosted at least hundred times,
> but
> I need to do more tests... repeated similar (slightly different) queries
> run
> with better performance, probably because of OS-level file caching...
> but it
> could be that of BK-Tree distance! (although I need to use classic int
> instead of float distance by Lucene/Levenstein etc.)
>
> Thanks,
> Fuad Efendi
> +1 416-993-2060
> http://www.tokenizer.ca/
> Data Mining, Vertical Search
>
>
>

I was lucky to contribute an excellent solution:
http://issues.apache.org/jira/browse/LUCENE-2230

Even 2nd edition of Lucene in Action advocates to use fuzzy search only in
exceptional cases.
Another solution would be 2-step indexing (it may work for many use cases),
but it is not "spellchecker"

1. Create a regular index
2. Create a dictionary of terms
3. For each term, find nearest terms (for instance, stick with distance=2)
4. Use "copyField" in SOLR, or smth similar to synonym dictionary; or, for
instance, generate specific Query Parser...
5. Of course, custom request handler
and etc.

It may work well (but only if query contains term from dictionary; it can't
work as a spellchecker)

Combination 2 algos can boost performance extremely...
Fuad Efendi
+1 416-993-2060
http://www.linkedin.com/in/liferay

Tokenizer Inc.
http://www.tokenizer.ca/
Data Mining, Vertical Search