| clear query|facets|time |
Search criteria: TieredMergePolicy.
Results from 1 to 10 from
174 (5.552s).
|
|
|
Loading phrases to help you
refine your search...
|
|
MergePolicy Thresholds - Lucene - [mail # dev]
|
|
...Hi Today, LogMP allows you to set different thresholds for segments sizes, thereby allowing you to control the largest segment that will be considered for merge + the largest segment your...
|
|
... index will hold (=~ threshold * mergeFactor). So, if you want to end up w/ say 20GB segments, you can set maxMergeMB(ForOptimize) to 2GB and mergeFactor=10. However, this often does...
|
[+ show more]
[- hide]
| ... not achieve your desired goal -- if the index contains 5 and 7 GB segments, they will never be merged b/c they are bigger than the threshold. I am willing to spend the CPU and IO resources to end up w... |
| .../ 20 GB segments, whether I'm merging 10 segments together or only 2. After I reach a 20GB segment, it can rest peacefully, at least until I increase the threshold. So I wonder, first... |
| ... instead of doing weird calculus that depend on maxMergeMB(ForOptimize) and mergeFactor? Second, should this be an addition to LogMP, or a different type of MP. One that adheres to only those two... |
|
|
Author: Shai Erera,
2011-05-02, 13:03
|
|
|
Re: MergePolicy Thresholds - Lucene - [mail # dev]
|
|
...Thanks Mike. I'll take a look at TieredMP. Does it depend on trunk in any way, or do you think it can easily be ported to 3x? Shai On Mon, May 2, 2011 at 6:34 PM, Michael Mc...
|
|
|
Author: Shai Erera,
2011-05-02, 18:16
|
|
|
Re: MergePolicy Thresholds - Lucene - [mail # dev]
|
|
...Hey Tom, Mike back-ported the changes to 3x, so you can try it out. FYI, Shai On Tue, May 3, 2011 at 9:33 PM, Burton-West, Tom wrote: ...
|
|
|
Author: Shai Erera,
2011-05-08, 03:32
|
|
|
Re: MergePolicy Thresholds - Lucene - [mail # dev]
|
|
...Hi I looked into porting it to 3x, and prepared the attached patch. It only contains the new TieredMP and Test, as well as the necessary changes to LuceneTestCase and IndexWriter. I guess...
|
|
|
Author: Shai Erera,
2011-05-03, 09:29
|
|
|
Re: MergePolicy Thresholds - Lucene - [mail # dev]
|
|
...Large/SmallNumSegments + maxMergeSizeMB) I can achieve something, but it's not very clear what is the right combination. Which is related to one of the points -- is it not more intuitive for an ap p to set...
|
|
... this threshold (if it needs any thresholds), than tweaking all of those parameters? If so, then we only need two thresholds (size + mergeFactor), and we can reuse BalancedMP's findBalancedMerges logic...
|
[+ show more]
[- hide]
| ... (perhaps w/ some adaptations) to derive a merge plan. Shai On Mon, May 2, 2011 at 4:42 PM, Earwin Burrfoot wrote: es r . ... |
|
|
Author: Shai Erera,
2011-05-02, 13:56
|
|
|
Re: MergePolicy Thresholds - Lucene - [mail # dev]
|
|
... I agree. I wonder tough if the knobs we give on LogMP are intuitive enough. It neatly avoids uber-merges I didn't see that I can define what "uber-merge" is, right? Can I tell it t o...
|
|
... stop merging segments of some size? E.g., if my index grew to 100 segments, 40GB each, I don't think that merging 10 40GB segments (to create 400GB segment) is going to speed up my search...
|
[+ show more]
[- hide]
| ..., for instance. A 40GB segment (probably much less) is already big enough to not be touched anymore. Will BalancedMP stop merging such segments (if all segments are of that order of magnitude)? Shai... |
|
|
Author: Shai Erera,
2011-05-02, 14:31
|
|
|
Re: MergePolicy Thresholds - Lucene - [mail # dev]
|
|
...I uploaded a patch to LUCENE-1076. Tom, apparently the patch I've attached before cannot be used, because there are dependencies (in earlier commits on LUCENE-1076) that need to be bac...
|
|
|
Author: Shai Erera,
2011-05-03, 12:10
|
|
|
Re: MergePolicy Thresholds - Lucene - [mail # dev]
|
|
...Mike, if you want, I can back-port it, as I've already started this when preparing the patch. I noticed that you added a "throws IOE" to IW.setInfoStream -- is it ok on 3x too? It'll b...
|
|
|
Author: Shai Erera,
2011-05-03, 09:51
|
|
|
Unnecessary messages creation by LogMergePolicy - Lucene - [mail # dev]
|
|
...Hi As I looked at the code in LogMergePolicy (and its sub-classes), I came across such lines: message("findMergesToExpungeDeletes: " + numSegments + " segments"); Those lines print...
|
|
... you'd like to give it) is enabled before attempting to output any message. The LogMergePolicy classes are one example that I've run at, but I'm sure there are other places in the code. I don...
|
|
|
Author: Shai Erera,
2008-12-05, 13:13
|
|
|
Re: ConcurrentMergeScheduler and MergePolicy question - Lucene - [mail # dev]
|
|
...I think that when LUCENE-1750 is finished, you will be able to: 1) Create a MergePolicy that limits the segments size it's about to merge to a certain size. 2) Then have a daemon...
|
|
... or something that runs on "idle" times and call optimize(maxNumSegments), or even open a new writer w/ the default merge policy and allow it to merge? Shai On Thu, Jul 30, 2009 at 5:48 PM, Grant...
|
|
|
Author: Shai Erera,
2009-07-30, 14:56
|
|
|
|
