Hi
I have a Very large number (say 3 million) of frequently changing Small
indexes. 90% of these indexes contain about 50 documents, while a few 2-3%
indexes have about 100,000 documents each (these being the more frequently
used indexes).
Each index belongs to a signed in user, thus can have unpredictable index
changes, in short periods of time, or can be stagnant for a long time.
What kind of indexing policy (mergefactor, maxmergedocs) would be optimal
for this kind of index. Is optimizing for this kind of index needed? or
wise?, if yes, what would be a good way to optimize (please note number of
indexes are very large).

Any suggestions would be very helpful.

--
Thanks
Naveen Kumar

Hi
I have one more question, does Lucene purge the deleted documents before
merging the segments, or purging of deleted  documents done only when
optimized?

On Thu, Sep 30, 2010 at 4:59 PM, Naveen Kumar <[EMAIL PROTECTED]> wrote:

> Hi
> I have a Very large number (say 3 million) of frequently changing Small
> indexes. 90% of these indexes contain about 50 documents, while a few 2-3%
> indexes have about 100,000 documents each (these being the more frequently
> used indexes).
> Each index belongs to a signed in user, thus can have unpredictable index
> changes, in short periods of time, or can be stagnant for a long time.
> What kind of indexing policy (mergefactor, maxmergedocs) would be optimal
> for this kind of index. Is optimizing for this kind of index needed? or
> wise?, if yes, what would be a good way to optimize (please note number of
> indexes are very large).
>
> Any suggestions would be very helpful.
>
> --
> Thanks
> Naveen Kumar
>
>
>
>
--
Thanks
Naveen Kumar

Deleted docs will be removed by lucene at some point - there is no
need to run optimize.
Read the javadocs for IndexWriter for details.  See also
expungeDeletes().  That may be just what you need.
--
Ian.
On Tue, Oct 5, 2010 at 7:48 AM, Naveen Kumar <[EMAIL PROTECTED]> wrote:
> Hi
> I have one more question, does Lucene purge the deleted documents before
> merging the segments, or purging of deleted  documents done only when
> optimized?
>
> On Thu, Sep 30, 2010 at 4:59 PM, Naveen Kumar <[EMAIL PROTECTED]> wrote:
>
>> Hi
>> I have a Very large number (say 3 million) of frequently changing Small
>> indexes. 90% of these indexes contain about 50 documents, while a few 2-3%
>> indexes have about 100,000 documents each (these being the more frequently
>> used indexes).
>> Each index belongs to a signed in user, thus can have unpredictable index
>> changes, in short periods of time, or can be stagnant for a long time.
>> What kind of indexing policy (mergefactor, maxmergedocs) would be optimal
>> for this kind of index. Is optimizing for this kind of index needed? or
>> wise?, if yes, what would be a good way to optimize (please note number of
>> indexes are very large).
>>
>> Any suggestions would be very helpful.
>>
>> --
>> Thanks
>> Naveen Kumar
>>
>>
>>
>>
>
>
> --
> Thanks
> Naveen Kumar
>

---------------------------------------------------------------------

Thank you, Ian
I have a large number of dynamically changing Index, so calling
expungeDeletes() and optimize() is very costly.
At this point I am opting to just set a optimum merge factor and skip
optimize()

On Tue, Oct 5, 2010 at 2:54 PM, Ian Lea <[EMAIL PROTECTED]> wrote:

> Deleted docs will be removed by lucene at some point - there is no
> need to run optimize.
> Read the javadocs for IndexWriter for details.  See also
> expungeDeletes().  That may be just what you need.
>
>
> --
> Ian.
>
>
> On Tue, Oct 5, 2010 at 7:48 AM, Naveen Kumar <[EMAIL PROTECTED]> wrote:
> > Hi
> > I have one more question, does Lucene purge the deleted documents before
> > merging the segments, or purging of deleted  documents done only when
> > optimized?
> >
> > On Thu, Sep 30, 2010 at 4:59 PM, Naveen Kumar <[EMAIL PROTECTED]> wrote:
> >
> >> Hi
> >> I have a Very large number (say 3 million) of frequently changing Small
> >> indexes. 90% of these indexes contain about 50 documents, while a few
> 2-3%
> >> indexes have about 100,000 documents each (these being the more
> frequently
> >> used indexes).
> >> Each index belongs to a signed in user, thus can have unpredictable
> index
> >> changes, in short periods of time, or can be stagnant for a long time.
> >> What kind of indexing policy (mergefactor, maxmergedocs) would be
> optimal
> >> for this kind of index. Is optimizing for this kind of index needed? or
> >> wise?, if yes, what would be a good way to optimize (please note number
> of
> >> indexes are very large).
> >>
> >> Any suggestions would be very helpful.
> >>
> >> --
> >> Thanks
> >> Naveen Kumar
> >>
> >>
> >>
> >>
> >
> >
> > --
> > Thanks
> > Naveen Kumar
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
Naveen Kumar

Have you considered having fewer indexes, each storing data for
multiple users?  Obviously with some indexed field that you can use
for restricting searches to data for that user.

I believe that is more common practice for this sort of scenario and
is known to work well.  You seem to be adding possibly unnecessary
complexity and overhead for all the tiny indexes.
--
Ian.
On Tue, Oct 5, 2010 at 11:18 AM, Naveen Kumar <[EMAIL PROTECTED]> wrote:
> Thank you, Ian
> I have a large number of dynamically changing Index, so calling
> expungeDeletes() and optimize() is very costly.
> At this point I am opting to just set a optimum merge factor and skip
> optimize()
>
> On Tue, Oct 5, 2010 at 2:54 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>
>> Deleted docs will be removed by lucene at some point - there is no
>> need to run optimize.
>> Read the javadocs for IndexWriter for details.  See also
>> expungeDeletes().  That may be just what you need.
>>
>>
>> --
>> Ian.
>>
>>
>> On Tue, Oct 5, 2010 at 7:48 AM, Naveen Kumar <[EMAIL PROTECTED]> wrote:
>> > Hi
>> > I have one more question, does Lucene purge the deleted documents before
>> > merging the segments, or purging of deleted  documents done only when
>> > optimized?
>> >
>> > On Thu, Sep 30, 2010 at 4:59 PM, Naveen Kumar <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hi
>> >> I have a Very large number (say 3 million) of frequently changing Small
>> >> indexes. 90% of these indexes contain about 50 documents, while a few
>> 2-3%
>> >> indexes have about 100,000 documents each (these being the more
>> frequently
>> >> used indexes).
>> >> Each index belongs to a signed in user, thus can have unpredictable
>> index
>> >> changes, in short periods of time, or can be stagnant for a long time.
>> >> What kind of indexing policy (mergefactor, maxmergedocs) would be
>> optimal
>> >> for this kind of index. Is optimizing for this kind of index needed? or
>> >> wise?, if yes, what would be a good way to optimize (please note number
>> of
>> >> indexes are very large).
>> >>
>> >> Any suggestions would be very helpful.
>> >>
>> >> --
>> >> Thanks
>> >> Naveen Kumar
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > Thanks
>> > Naveen Kumar
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>
>
>
> Naveen Kumar
>

---------------------------------------------------------------------