Hello All,
            I know that in most of the cases there is no need to edit
the ranking formula and  I hope for the same in our case. So to make
sure that there is no need, I have following queries.
1) Does Solr (Lucene) consider exact match to be something more
important ? I mean if the query is  "description:organisation", then
which one of the following would be returned?
       Document A, consiting  just "description:organisation" , where
as Document B consisting "description:bla bla ... organisation bla
bla.. ". Does it consider length of the field-text while ranking ?

2) Let us assume that our query is "value0 field1:value1" . So here,
if we use OR as the default operator its obvious that we may get
results in which we might find dominating "value0" and no
"field1:value1" at all. We need some kind of mixture of "OR" and
"AND", which gives more importance also for the "number of keywords"
found. So I would like to find out whether we can edit some kind of
boosting (or something relevant) to achieve this.

Thanks.

> 1) Does Solr (Lucene) consider exact match to be something
> more important ? I mean if the query is 
> "description:organisation", then
> which one of the following would be returned?
>        Document A, consiting 
> just "description:organisation" , where
> as Document B consisting "description:bla bla ...
> organisation bla
> bla.. ". Does it consider length of the field-text while
> ranking ?

It is called length normalization which is done by default. It favors short documents. It punishes long documents.

> 2) Let us assume that our query is "value0 field1:value1" .
> So here,
> if we use OR as the default operator its obvious that we
> may get
> results in which we might find dominating "value0" and no
> "field1:value1" at all. We need some kind of mixture of
> "OR" and
> "AND", which gives more importance also for the "number of
> keywords"
> found. So I would like to find out whether we can edit some
> kind of
> boosting (or something relevant) to achieve this.

Generally if a documents contains more query terms, it will get higher score. But it is not true all times since there are other parameters. For example a short document with only one query term might get higher score than a long document with containing two query terms.

This link can be useful:

http://lucene.apache.org/java/3_0_0/api/core/org/apache/lucene/search/Similarity.html

Hello ,
          Thanks. That clears my doubts.Coming to the point two, Can
you please tell me which part of the Similarity takes care of the
same. Is it possible to implement in such a way that we give more
preference to "number of found terms". Also, here in our case  we need
to give more importance to "length normalisation" than the default? So
any suggestions?
Thanks.

On 15 February 2010 19:50, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
>> 1) Does Solr (Lucene) consider exact match to be something
>> more important ? I mean if the query is
>> "description:organisation", then
>> which one of the following would be returned?
>>        Document A, consiting
>> just "description:organisation" , where
>> as Document B consisting "description:bla bla ...
>> organisation bla
>> bla.. ". Does it consider length of the field-text while
>> ranking ?
>
> It is called length normalization which is done by default. It favors short documents. It punishes long documents.
>
>> 2) Let us assume that our query is "value0 field1:value1" .
>> So here,
>> if we use OR as the default operator its obvious that we
>> may get
>> results in which we might find dominating "value0" and no
>> "field1:value1" at all. We need some kind of mixture of
>> "OR" and
>> "AND", which gives more importance also for the "number of
>> keywords"
>> found. So I would like to find out whether we can edit some
>> kind of
>> boosting (or something relevant) to achieve this.
>
> Generally if a documents contains more query terms, it will get higher score. But it is not true all times since there are other parameters. For example a short document with only one query term might get higher score than a long document with containing two query terms.
>
> This link can be useful:
>
> http://lucene.apache.org/java/3_0_0/api/core/org/apache/lucene/search/Similarity.html
>
>
>
>

> Hello ,
>           Thanks. That clears my
> doubts. Coming to the point two, Can
> you please tell me which part of the Similarity takes care
> of the
> same. Is it possible to implement in such a way that we
> give more
> preference to "number of found terms".

public float coord(int overlap, int maxOverlap) method takes care:

"coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in coord(q,d) by the Similarity in effect at search time."

> Also, here in our case  we need
> to give more importance to "length normalisation" than the
> default?

Do you want to punish *more* long documents?
For example you can return directly 1/numTerms or 1/(numTerms*numTerms) in this method of DefaultSimilarity:

/** Implemented as <code>1/sqrt(numTerms)</code>. */
  @Override
  public float lengthNorm(String fieldName, int numTerms) {
    return (float)(1.0 / Math.sqrt(numTerms));
  }

There will be a trade-off since there are lots of parameters.
If you have two-words query which one is important for you:
A short document containing one word?
A long document containing two word?
Or
A long document containing one query term which is very rare (high idf)
A short document containing one query term which is very common (low idf)

Many combinations...

Hello ,
       Thanks for your detailed explaination.

> Do you want to punish *more* long documents?

    Not alot, but a bit more than default implementation. It seems
"lengthNorm" is field based and pinushing lengthy fields does fit most
of the cases in our project.

> There will be a trade-off since there are lots of parameters.
> If you have two-words query which one is important for you:
> A short document containing one word?
> A long document containing two word?
> Or
> A long document containing one query term which is very rare (high idf)
> A short document containing one query term which is very common (low idf)
>
> Many combinations...

        After getting aware of all these combinations, it seems not
wise to proceed blindly by punushing what ever we want. Thank you very
much for letting me know.

           I have observed "scoring" by enabling debugging. There is a
clear explaination of scoring --> product of x,y,z..
where I can find tf, idf, and fieldNorm() but not  "coord(q,d)  and
queryNorm(q)". Its mentioned that querNorm is the same for all, so no
worries but what about coord(q,d) ?

Thanks.

On 16 February 2010 18:00, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
>
>> Hello ,
>>           Thanks. That clears my
>> doubts. Coming to the point two, Can
>> you please tell me which part of the Similarity takes care
>> of the
>> same. Is it possible to implement in such a way that we
>> give more
>> preference to "number of found terms".
>
> public float coord(int overlap, int maxOverlap) method takes care:
>
> "coord(q,d) is a score factor based on how many of the query terms are found in the specified document. Typically, a document that contains more of the query's terms will receive a higher score than another document with fewer query terms. This is a search time factor computed in coord(q,d) by the Similarity in effect at search time."
>
>> Also, here in our case  we need
>> to give more importance to "length normalisation" than the
>> default?
>
> Do you want to punish *more* long documents?
> For example you can return directly 1/numTerms or 1/(numTerms*numTerms) in this method of DefaultSimilarity:
>
> /** Implemented as <code>1/sqrt(numTerms)</code>. */
>  @Override
>  public float lengthNorm(String fieldName, int numTerms) {
>    return (float)(1.0 / Math.sqrt(numTerms));
>  }
>
> There will be a trade-off since there are lots of parameters.
> If you have two-words query which one is important for you:
> A short document containing one word?
> A long document containing two word?
> Or
> A long document containing one query term which is very rare (high idf)
> A short document containing one query term which is very common (low idf)
>
> Many combinations...
>
>
>
>

> After getting aware of all
> these combinations, it seems not
> wise to proceed blindly by punushing what ever we want.
> Thank you very
> much for letting me know.

Generally most of the people are happy with default solr scoring. Especially in web like search.

I am not sure but you can find this [1] interesting. With this you can say, for example i want at least 75% of the optional clauses must match.

[1]http://wiki.apache.org/solr/DisMaxRequestHandler#mm_.28Minimum_.27Should.27_Match.29

>            I have
> observed "scoring" by enabling debugging. There is a
> clear explaination of scoring --> product of x,y,z..
> where I can find tf, idf, and fieldNorm() but not 
> "coord(q,d)  and
> queryNorm(q)". Its mentioned that querNorm is the same for
> all, so no
> worries but what about coord(q,d) ?

If you want to see coord info like "0.5 = coord(1/2)" enter a query contains two or more terms combined with OR operator, and then go to last page with something like : &rows=10&start=numFound-10&debugQuery=on
You will be able to see it.

Norms are generally not calculated. You need to change the field you
want with this attribute: omitNorms="false".

On Tue, Feb 16, 2010 at 2:38 PM, Ahmet Arslan <[EMAIL PROTECTED]> wrote:
>> After getting aware of all
>> these combinations, it seems not
>> wise to proceed blindly by punushing what ever we want.
>> Thank you very
>> much for letting me know.
>
> Generally most of the people are happy with default solr scoring. Especially in web like search.
>
> I am not sure but you can find this [1] interesting. With this you can say, for example i want at least 75% of the optional clauses must match.
>
> [1]http://wiki.apache.org/solr/DisMaxRequestHandler#mm_.28Minimum_.27Should.27_Match.29
>
>>            I have
>> observed "scoring" by enabling debugging. There is a
>> clear explaination of scoring --> product of x,y,z..
>> where I can find tf, idf, and fieldNorm() but not
>> "coord(q,d)  and
>> queryNorm(q)". Its mentioned that querNorm is the same for
>> all, so no
>> worries but what about coord(q,d) ?
>
> If you want to see coord info like "0.5 = coord(1/2)" enter a query contains two or more terms combined with OR operator, and then go to last page with something like : &rows=10&start=numFound-10&debugQuery=on
> You will be able to see it.
>
>
>
>
>

--
Lance Norskog
[EMAIL PROTECTED]