Hello,
We are trying to get Solr to work for a really huge parts database. Details
of the database
- 55 million parts
- Totally 3700 properties (facets). But each record will not have value for
all properties.
- Most of these facets are defined as dynamic fields within the Solr Index

We were getting really unacceptable timing while doing faceting/searches on
an index created with this database. With only one user using the system,
query times are in excess of 1 minute. With more users concurrently using
the system, the response times are further high.

We thought that by limiting the number of properties that are available for
faceting, the performance can be improved. To test this, we enabled only 6
properties for faceting by setting indexed=true (in schema.xml) for only
these properties. All other properties which are defined as dynamic
properties had indexed=false. The observations after this change :

- Index size reduced by a meagre 5 % only
- Performance did not improve. Infact during PSR run we observed that it
degraded.

My questions:
 - Will reducing the number of facets improve faceting and search
performance ?
- Is there a better way to reduce the number of facets ?
- Will having a large number of properties defined as dynamic fields, reduce
performance ?

Thank you.

Regards
Rahul

Having a large number of fields is not the same as having a large number of
facets. To facets are something you would display to users as aid for query
refinement or navigation. There is no way for a user to use 3700 facets at
the same time. So it more of question on how to determine what facets to
fetch on search time based on the user's actions or based on certain
predefined configurations. I have written an application with 30 some
facetable fields on millions of records, I also ran into the issue of
calculate all facets as the server resources as limited to number of caches
available and CPU cycles available for facet calculations. I then realize
why display all these facet regardless user want to see them or not? I have
then change to approach to only fetch minimum set of facets by default and
make the rest of facets fields open on-demand (using AJAX). I was able to
dramatically increase the response time by spreading the facet loading
overtime. There are still issues of total facet caches when you have a large
number available facets, but you need realistically evaluate what does it
means to a user to have large number of facet. I don't think on typical user
interface having more than 10 filters showing at the same time will be any
more effective than having a small number of filters to begin with and
progressive showing more on-demand (hierarchical facets?)
Rahul R wrote:
>
> Hello,
> We are trying to get Solr to work for a really huge parts database.
> Details
> of the database
> - 55 million parts
> - Totally 3700 properties (facets). But each record will not have value
> for
> all properties.
> - Most of these facets are defined as dynamic fields within the Solr Index
>
> We were getting really unacceptable timing while doing faceting/searches
> on
> an index created with this database. With only one user using the system,
> query times are in excess of 1 minute. With more users concurrently using
> the system, the response times are further high.
>
> We thought that by limiting the number of properties that are available
> for
> faceting, the performance can be improved. To test this, we enabled only 6
> properties for faceting by setting indexed=true (in schema.xml) for only
> these properties. All other properties which are defined as dynamic
> properties had indexed=false. The observations after this change :
>
> - Index size reduced by a meagre 5 % only
> - Performance did not improve. Infact during PSR run we observed that it
> degraded.
>
> My questions:
>  - Will reducing the number of facets improve faceting and search
> performance ?
> - Is there a better way to reduce the number of facets ?
> - Will having a large number of properties defined as dynamic fields,
> reduce
> performance ?
>
> Thank you.
>
> Regards
> Rahul
>
>

--
View this message in context: http://www.nabble.com/Limiting-facets-for-huge-data---setting-indexed%3Dfalse-in-schema.xml-tp24751763p24761778.html
Sent from the Solr - User mailing list archive at Nabble.com.

On Fri, Jul 31, 2009 at 3:19 PM, Yao Ge<[EMAIL PROTECTED]> wrote:
> Having a large number of fields is not the same as having a large number of
> facets. To facets are something you would display to users as aid for query
> refinement or navigation. There is no way for a user to use 3700 facets at
> the same time.

Indeed... it may just be a terminology issue.  Likely it's one field
with 3700 possible values.

-Yonik
http://www.lucidimagination.com

On Jul 31, 2009, at 2:35 AM, Rahul R wrote:

> Hello,
> We are trying to get Solr to work for a really huge parts database.  
> Details
> of the database
> - 55 million parts
> - Totally 3700 properties (facets). But each record will not have  
> value for
> all properties.
> - Most of these facets are defined as dynamic fields within the Solr  
> Index
>
> We were getting really unacceptable timing while doing faceting/
> searches on
> an index created with this database.

Were you accounting for cache warming?  Were your caches sized  
appropriately?  What kind of hardware and RAM were you using?  What  
were the JVM settings?

And certainly not least important - what version of Solr are you  
running?   The difference in faceting performance and scalability  
between Solr 1.3 and what will be Solr 1.4 is quite dramatic.

> We thought that by limiting the number of properties that are  
> available for
> faceting, the performance can be improved. To test this, we enabled  
> only 6
> properties for faceting by setting indexed=true (in schema.xml) for  
> only
> these properties. All other properties which are defined as dynamic
> properties had indexed=false.

These settings won't matter - what matters in this case is what facets  
you request, not what is actually in the index.
> My questions:
> - Will reducing the number of facets improve faceting and search
> performance ?

Reducing what fields you request will, of course.  But what you  
actually index has no effect on performance until you request it.

> - Is there a better way to reduce the number of facets ?

Hard to say without doing a deeper analysis of your needs.

> - Will having a large number of properties defined as dynamic  
> fields, reduce
> performance ?

Dynamic fields versus statically named fields have no effect on  
performance.

Erik

Erik,
I understand that caching is going to improve performance. Infact we did a
PSR run with caches enabled and we got awesome results. But these wouldn't
be really representative because the PSR scripts will be doing the same
searches again and again. These would be cached and there would be virtually
no evictions. This is not a practical case.

My hardware (in the PSR environment where I am testing) is pretty good - 12
CPU, 24 G RAM, Ultrasparc III 1.2 GHz processors, Solaris 10. We have
allocated 3.2 GB RAM for Weblogic (JVM). This is the maximum that I am able
to allocate for one JVM.
I think I need to go back and check if I am not using all the fields in the
query. I understand that setting indexed=false alone will not ensure that
all fields don't participate in the query.

Thanks a lot for your response.

Regards
Rahul
On Fri, Jul 31, 2009 at 3:33 PM, Erik Hatcher <[EMAIL PROTECTED]>wrote:

>
> On Jul 31, 2009, at 2:35 AM, Rahul R wrote:
>
> Hello,
>> We are trying to get Solr to work for a really huge parts database.
>> Details
>> of the database
>> - 55 million parts
>> - Totally 3700 properties (facets). But each record will not have value
>> for
>> all properties.
>> - Most of these facets are defined as dynamic fields within the Solr Index
>>
>> We were getting really unacceptable timing while doing faceting/searches
>> on
>> an index created with this database.
>>
>
> Were you accounting for cache warming?  Were your caches sized
> appropriately?  What kind of hardware and RAM were you using?  What were the
> JVM settings?
>
> And certainly not least important - what version of Solr are you running?
> The difference in faceting performance and scalability between Solr 1.3 and
> what will be Solr 1.4 is quite dramatic.
>
> We thought that by limiting the number of properties that are available for
>> faceting, the performance can be improved. To test this, we enabled only 6
>> properties for faceting by setting indexed=true (in schema.xml) for only
>> these properties. All other properties which are defined as dynamic
>> properties had indexed=false.
>>
>
> These settings won't matter - what matters in this case is what facets you
> request, not what is actually in the index.
>
>
> My questions:
>> - Will reducing the number of facets improve faceting and search
>> performance ?
>>
>
> Reducing what fields you request will, of course.  But what you actually
> index has no effect on performance until you request it.
>
> - Is there a better way to reduce the number of facets ?
>>
>
> Hard to say without doing a deeper analysis of your needs.
>
> - Will having a large number of properties defined as dynamic fields,
>> reduce
>> performance ?
>>
>
> Dynamic fields versus statically named fields have no effect on
> performance.
>
>        Erik
>
>

On Jul 31, 2009, at 7:17 AM, Rahul R wrote:

> Erik,
> I understand that caching is going to improve performance. Infact we  
> did a
> PSR run with caches enabled and we got awesome results. But these  
> wouldn't
> be really representative because the PSR scripts will be doing the  
> same
> searches again and again. These would be cached and there would be  
> virtually
> no evictions. This is not a practical case.

I don't understand how this is not practical.  Why wouldn't having the  
caches warmed and filled with the facets be practical for your needs?

Erik

In a production environment, having the caches enabled makes a lot of sense.
And most definitely we will be enabling them. However, the primary idea of
this exercise is to verify if limiting the number of facets will actually
improve the performance.

An update on this. I did verify and looks like although I set indexed=false
for most of the properties, I have not blocked them from participating in
the query. I now enabled only 7 properties for faceting. Now at any given
time only a maximum of 7 facets will participate in the query. Performance
has now improved from an erstwhile 60 seconds to around 10 seconds.

This really helped. Thanks a lot !

Regards
Rahul

On Fri, Jul 31, 2009 at 6:34 PM, Erik Hatcher <[EMAIL PROTECTED]>wrote:

>
> On Jul 31, 2009, at 7:17 AM, Rahul R wrote:
>
> Erik,
>> I understand that caching is going to improve performance. Infact we did a
>> PSR run with caches enabled and we got awesome results. But these wouldn't
>> be really representative because the PSR scripts will be doing the same
>> searches again and again. These would be cached and there would be
>> virtually
>> no evictions. This is not a practical case.
>>
>
> I don't understand how this is not practical.  Why wouldn't having the
> caches warmed and filled with the facets be practical for your needs?
>
>        Erik
>
>

What version of Solr?   Try a nightly build if you're at Solr 1.3 or  
earlier and you'll be amazed at the difference.

Erik

On Jul 31, 2009, at 10:00 AM, Rahul R wrote:

> In a production environment, having the caches enabled makes a lot  
> of sense.
> And most definitely we will be enabling them. However, the primary  
> idea of
> this exercise is to verify if limiting the number of facets will  
> actually
> improve the performance.
>
> An update on this. I did verify and looks like although I set  
> indexed=false
> for most of the properties, I have not blocked them from  
> participating in
> the query. I now enabled only 7 properties for faceting. Now at any  
> given
> time only a maximum of 7 facets will participate in the query.  
> Performance
> has now improved from an erstwhile 60 seconds to around 10 seconds.
>
> This really helped. Thanks a lot !
>
> Regards
> Rahul
>
> On Fri, Jul 31, 2009 at 6:34 PM, Erik Hatcher <[EMAIL PROTECTED]
> >wrote:
>
>>
>> On Jul 31, 2009, at 7:17 AM, Rahul R wrote:
>>
>> Erik,
>>> I understand that caching is going to improve performance. Infact  
>>> we did a
>>> PSR run with caches enabled and we got awesome results. But these  
>>> wouldn't
>>> be really representative because the PSR scripts will be doing the  
>>> same
>>> searches again and again. These would be cached and there would be
>>> virtually
>>> no evictions. This is not a practical case.
>>>
>>
>> I don't understand how this is not practical.  Why wouldn't having  
>> the
>> caches warmed and filled with the facets be practical for your needs?
>>
>>       Erik
>>
>>

We are using 1.3.0. Thanks for the suggestion. Will see if I can try one of
the ngihtly builds.

On Fri, Jul 31, 2009 at 7:49 PM, Erik Hatcher <[EMAIL PROTECTED]>wrote:

> What version of Solr?   Try a nightly build if you're at Solr 1.3 or
> earlier and you'll be amazed at the difference.
>
>        Erik
>
>
> On Jul 31, 2009, at 10:00 AM, Rahul R wrote:
>
> In a production environment, having the caches enabled makes a lot of
>> sense.
>> And most definitely we will be enabling them. However, the primary idea of
>> this exercise is to verify if limiting the number of facets will actually
>> improve the performance.
>>
>> An update on this. I did verify and looks like although I set
>> indexed=false
>> for most of the properties, I have not blocked them from participating in
>> the query. I now enabled only 7 properties for faceting. Now at any given
>> time only a maximum of 7 facets will participate in the query. Performance
>> has now improved from an erstwhile 60 seconds to around 10 seconds.
>>
>> This really helped. Thanks a lot !
>>
>> Regards
>> Rahul
>>
>> On Fri, Jul 31, 2009 at 6:34 PM, Erik Hatcher <[EMAIL PROTECTED]
>> >wrote:
>>
>>
>>> On Jul 31, 2009, at 7:17 AM, Rahul R wrote:
>>>
>>> Erik,
>>>
>>>> I understand that caching is going to improve performance. Infact we did
>>>> a
>>>> PSR run with caches enabled and we got awesome results. But these
>>>> wouldn't
>>>> be really representative because the PSR scripts will be doing the same
>>>> searches again and again. These would be cached and there would be
>>>> virtually
>>>> no evictions. This is not a practical case.
>>>>
>>>>
>>> I don't understand how this is not practical.  Why wouldn't having the
>>> caches warmed and filled with the facets be practical for your needs?
>>>
>>>      Erik
>>>
>>>
>>>
>