> lucene can rebuild index using his internal info and how ? or in have to
> reinsert all in other way?
>

> I'm afraid it's not possible to rebuild index. It's important to maintain a
> backup policy because of that.
>
>
> On Mon, Mar 20, 2017 at 5:12 PM Cristian Lorenzetto <
> [EMAIL PROTECTED]> wrote:
>
> > lucene can rebuild index using his internal info and how ? or in have to
> > reinsert all in other way?
> >
> --
> Marco Reis
> Software Architect
> http://marcoreis.net
> https://github.com/masreis
> +55 61 9 81194620
>

> Hi, I think you forgot to CC the lucene user's list (
> [EMAIL PROTECTED]) in your reply?  Can you resend?
>
> Thanks.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Wed, Mar 22, 2017 at 9:02 AM, Cristian Lorenzetto <
> [EMAIL PROTECTED]> wrote:
>
>> hi , i m thinking about what you told me in previous message and how to
>> solve the corruption problem and the problem about commit operation
>> executed in async way.
>>
>> I m thinking to create a simple transaction log in a file.
>> i use a long atomic sequence for a ordinable transaction id.
>>
>> when i make a new operation
>> 1) generate new incremental transaction id
>> 2) save the operation abstract info in transaction log associated to id.
>>     2.a insert ,update with the a serialized version of the object to
>> save
>>     2b delete the query serialized where apply delete
>> 3) execute same operation in lucene adding before property transactionId
>> (executed in ram)
>>
>> 4) in async way commit is executed. After the commit the transaction log
>> until last transaction id is deleted.(i dont know how insert block after
>> commit , using near real time reader and SearcherManager) I might
>>  introduce a logic in the way a commit is done. The order is simlilar to a
>> queue so it follows the transactionId order. i Is there a example about
>> possibility to commit a specific set of uncommit operations?
>>
>> 5) i need the warrenty after a crud operation the data in available in
>> memory  in a possible imminent research so i think i might execute
>> flush/refreshReader after every CUD operations
>>
>> if there is a failure transaction log will be not empty. But i can
>> rexecute operations not executed after restartup.
>> Maybe it could be usefull also for fixing a corruption but it is sure the
>> corrution dont touch also segments already commited completely in the past?
>> or maybe for a stable solution i might anyway save data in a secondary
>> repository ?
>>
>>
>>
>> for your opinion this solution will be sufficient . It is a good solution
>> for you, i m forgetting some aspects?

> Errata corridge/integration for questions related to previous my post
>
> I studied a bit this lucene classes for understanding:
> 1) setCommitData is designed for versioning the index , not for passing a
> transaction log. However if userdata is different for every transactionid
> it is equivalent .
> 2) NRT refresh automatically searcher/reader it dont call commit. I based
> my implementation using nrt on http://stackoverflow.com/
> questions/17993960/lucene-4-4-0-new-controlledrealtimereopenthread
> -sample-usage. In this example commit is executed for every crud
> operation in synchronous way but in general it is advised to use a batch
> thread because the commit is a long operation. *So it is not clear how to
> do the commit in a near-real time system with a indefinite index size.*
>      2.a if the commit is synchronous , i can use user data because it is
> used before a commit, every commit has a different user data and i can
> trace the transactions changes.But in general a commit can requires also
> minutes for be completed so then it dont seams a real solution in a near
> real time solution.
>     2.b if the commit is async, it is executed every X times (or better
> how memory if full) , the commit can not be used for tracing the
> transactions and i can pass a trnsaction id associated with a lucene
> commit. I can add a mutex in crud ( when i loading uncommit data) i m sure
> the last uncummit Index is aligned to the last transaction id X, so there
> is no overlappind and the crud block is very fast when happens.But how to
> grant that the commit is related to the last CommitIndex what i loaded?
> Maybe if i introduce that mutex in a custom mergePolicy?
> It is right what i wrote until now ?The best solution is 2.b? In this case
> how to grant the commit is done based on the uncommit data loaded in a
> specific commitIndex?
>
>
>
>
>
> 2017-03-22 15:32 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]>:
>
>> Hi, I think you forgot to CC the lucene user's list (
>> [EMAIL PROTECTED]) in your reply?  Can you resend?
>>
>> Thanks.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Wed, Mar 22, 2017 at 9:02 AM, Cristian Lorenzetto <
>> [EMAIL PROTECTED]> wrote:
>>
>>> hi , i m thinking about what you told me in previous message and how to
>>> solve the corruption problem and the problem about commit operation
>>> executed in async way.
>>>
>>> I m thinking to create a simple transaction log in a file.
>>> i use a long atomic sequence for a ordinable transaction id.
>>>
>>> when i make a new operation
>>> 1) generate new incremental transaction id
>>> 2) save the operation abstract info in transaction log associated to id.
>>>     2.a insert ,update with the a serialized version of the object to
>>> save
>>>     2b delete the query serialized where apply delete
>>> 3) execute same operation in lucene adding before property transactionId
>>> (executed in ram)
>>>
>>> 4) in async way commit is executed. After the commit the transaction log
>>> until last transaction id is deleted.(i dont know how insert block after
>>> commit , using near real time reader and SearcherManager) I might
>>>  introduce a logic in the way a commit is done. The order is simlilar to a
>>> queue so it follows the transactionId order. i Is there a example about
>>> possibility to commit a specific set of uncommit operations?

> Errata corridge/integration for questions related to previous my post
>
> I studied a bit this lucene classes for understanding:
> 1) setCommitData is designed for versioning the index , not for passing a
> transaction log. However if userdata is different for every transactionid
> it is equivalent .
> 2) NRT refresh automatically searcher/reader it dont call commit. I based
> my implementation using nrt on http://stackoverflow.com/
> questions/17993960/lucene-4-4-0-new-controlledrealtimereopenthread
> -sample-usage. In this example commit is executed for every crud
> operation in synchronous way but in general it is advised to use a batch
> thread because the commit is a long operation. *So it is not clear how to
> do the commit in a near-real time system with a indefinite index size.*
>      2.a if the commit is synchronous , i can use user data because it is
> used before a commit, every commit has a different user data and i can
> trace the transactions changes.But in general a commit can requires also
> minutes for be completed so then it dont seams a real solution in a near
> real time solution.
>     2.b if the commit is async, it is executed every X times (or better
> how memory if full) , the commit can not be used for tracing the
> transactions and i can pass a trnsaction id associated with a lucene
> commit. I can add a mutex in crud ( when i loading uncommit data) i m sure
> the last uncummit Index is aligned to the last transaction id X, so there
> is no overlappind and the crud block is very fast when happens.But how to
> grant that the commit is related to the last CommitIndex what i loaded?
> Maybe if i introduce that mutex in a custom mergePolicy?
> It is right what i wrote until now ?The best solution is 2.b? In this case
> how to grant the commit is done based on the uncommit data loaded in a
> specific commitIndex?
>
>
>
>
>
> 2017-03-22 15:32 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]>:
>
>> Hi, I think you forgot to CC the lucene user's list (
>> [EMAIL PROTECTED]) in your reply?  Can you resend?
>>
>> Thanks.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Wed, Mar 22, 2017 at 9:02 AM, Cristian Lorenzetto <
>> [EMAIL PROTECTED]> wrote:
>>
>>> hi , i m thinking about what you told me in previous message and how to
>>> solve the corruption problem and the problem about commit operation
>>> executed in async way.
>>>
>>> I m thinking to create a simple transaction log in a file.
>>> i use a long atomic sequence for a ordinable transaction id.
>>>
>>> when i make a new operation
>>> 1) generate new incremental transaction id
>>> 2) save the operation abstract info in transaction log associated to id.
>>>     2.a insert ,update with the a serialized version of the object to
>>> save
>>>     2b delete the query serialized where apply delete
>>> 3) execute same operation in lucene adding before property transactionId
>>> (executed in ram)
>>>
>>> 4) in async way commit is executed. After the commit the transaction log
>>> until last transaction id is deleted.(i dont know how insert block after
>>> commit , using near real time reader and SearcherManager) I might
>>>  introduce a logic in the way a commit is done. The order is simlilar to a
>>> queue so it follows the transactionId order. i Is there a example about
>>> possibility to commit a specific set of uncommit operations?
>>>
>>> 5) i need the warrenty after a crud operation the data in available in
>>> memory  in a possible imminent research so i think i might execute
>>> flush/refreshReader after every CUD operations
>>>
>>> if there is a failure transaction log will be not empty. But i can

> Yes exactly. I saw, working in the past in systems using lucene (for
> example alfresco projects),  lucene corruption happens sometimes and every
> time the building requires a lot of times ... so i thougth a way for
> accelerating the fixing of a corruption index. In addition there is a rare
> case not described here ( If after a database commit lucene throws a
> exception for exampe disk is full ) there is a possibility of a
>  disalignement from the database and the lucene index. With this system
> these problems could be solved automatically. In database every row has a
> property with trasaction id.  So if i know in lucene is missing a segment 6
> , corrisponds to   transactions range[ 1000, 1050] so i can reload in a
> query in database just corrisponding rows.
>
> 2017-03-23 14:59 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]>:
>
>> You should be able to use the sequence numbers returned by IndexWriter
>> operations to "know" which operations made it into the commit and which did
>> not, and then on disaster recovery replay only those operations that didn't
>> make it?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Thu, Mar 23, 2017 at 5:53 AM, Cristian Lorenzetto <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Errata corridge/integration for questions related to previous my post
>>>
>>> I studied a bit this lucene classes for understanding:
>>> 1) setCommitData is designed for versioning the index , not for passing
>>> a transaction log. However if userdata is different for every transactionid
>>> it is equivalent .
>>> 2) NRT refresh automatically searcher/reader it dont call commit. I
>>> based my implementation using nrt on http://stackoverflow.com/qu
>>> estions/17993960/lucene-4-4-0-new-controlledrealtimereopenth
>>> read-sample-usage. In this example commit is executed for every crud
>>> operation in synchronous way but in general it is advised to use a batch
>>> thread because the commit is a long operation. *So it is not clear how
>>> to do the commit in a near-real time system with a indefinite index size.*
>>>      2.a if the commit is synchronous , i can use user data because it
>>> is used before a commit, every commit has a different user data and i can
>>> trace the transactions changes.But in general a commit can requires also
>>> minutes for be completed so then it dont seams a real solution in a near
>>> real time solution.
>>>     2.b if the commit is async, it is executed every X times (or better
>>> how memory if full) , the commit can not be used for tracing the
>>> transactions and i can pass a trnsaction id associated with a lucene
>>> commit. I can add a mutex in crud ( when i loading uncommit data) i m sure
>>> the last uncummit Index is aligned to the last transaction id X, so there
>>> is no overlappind and the crud block is very fast when happens.But how to
>>> grant that the commit is related to the last CommitIndex what i loaded?
>>> Maybe if i introduce that mutex in a custom mergePolicy?
>>> It is right what i wrote until now ?The best solution is 2.b? In this
>>> case how to grant the commit is done based on the uncommit data loaded in a
>>> specific commitIndex?
>>>
>>>
>>>
>>>
>>>
>>> 2017-03-22 15:32 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]
>>> >:
>>>
>>>> Hi, I think you forgot to CC the lucene user's list (
>>>> [EMAIL PROTECTED]) in your reply?  Can you resend?

> I deduce the transaction range not using the segment corrupted but the
> corrected segments. The transaction id is incremental and i imagine segment
> are saved sequentelly so if it is missing the segment 5 , reading the
> correct segment 4 i can find the maximunn transaction id A , reading the
> segment 6 i can find the minimum transaction id B so i can deduce the hole
> , the range is [A+1,B-1] ... making a query in db i reaload the
> corrisponding document and i add again in lucene this missing documents.
>
>
> 2017-03-23 15:17 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]>:
>
>> Lucene corruption should be rare and only due to bad hardware; if you are
>> seeing otherwise we really should get to the root cause.
>>
>> Mapping documents to each segment will not be easy in general, especially
>> if that segment is now corrupted so you can't search it.
>>
>> Documents lost because of power loss / OS crash while indexing can be
>> more common, and its for that use case that the sequence numbers /
>> transaction log should be helpful.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Thu, Mar 23, 2017 at 10:12 AM, Cristian Lorenzetto <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Yes exactly. I saw, working in the past in systems using lucene (for
>>> example alfresco projects),  lucene corruption happens sometimes and every
>>> time the building requires a lot of times ... so i thougth a way for
>>> accelerating the fixing of a corruption index. In addition there is a rare
>>> case not described here ( If after a database commit lucene throws a
>>> exception for exampe disk is full ) there is a possibility of a
>>>  disalignement from the database and the lucene index. With this system
>>> these problems could be solved automatically. In database every row has a
>>> property with trasaction id.  So if i know in lucene is missing a segment 6
>>> , corrisponds to   transactions range[ 1000, 1050] so i can reload in a
>>> query in database just corrisponding rows.
>>>
>>> 2017-03-23 14:59 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]
>>> >:
>>>
>>>> You should be able to use the sequence numbers returned by IndexWriter
>>>> operations to "know" which operations made it into the commit and which did
>>>> not, and then on disaster recovery replay only those operations that didn't
>>>> make it?
>>>>
>>>> Mike McCandless
>>>>
>>>> http://blog.mikemccandless.com
>>>>
>>>> On Thu, Mar 23, 2017 at 5:53 AM, Cristian Lorenzetto <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Errata corridge/integration for questions related to previous my post
>>>>>
>>>>> I studied a bit this lucene classes for understanding:
>>>>> 1) setCommitData is designed for versioning the index , not for
>>>>> passing a transaction log. However if userdata is different for every
>>>>> transactionid it is equivalent .
>>>>> 2) NRT refresh automatically searcher/reader it dont call commit. I
>>>>> based my implementation using nrt on http://stackoverflow.com/qu
>>>>> estions/17993960/lucene-4-4-0-new-controlledrealtimereopenth
>>>>> read-sample-usage. In this example commit is executed for every crud
>>>>> operation in synchronous way but in general it is advised to use a batch
>>>>> thread because the commit is a long operation. *So it is not clear
>>>>> how to do the commit in a near-real time system with a indefinite index
>>>>> size.*
>>>>>      2.a if the commit is synchronous , i can use user data because it

> You should be able to use the sequence numbers returned by IndexWriter
> operations to "know" which operations made it into the commit and which did
> not, and then on disaster recovery replay only those operations that didn't
> make it?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Thu, Mar 23, 2017 at 5:53 AM, Cristian Lorenzetto <
> [EMAIL PROTECTED]> wrote:
>
>> Errata corridge/integration for questions related to previous my post
>>
>> I studied a bit this lucene classes for understanding:
>> 1) setCommitData is designed for versioning the index , not for passing a
>> transaction log. However if userdata is different for every transactionid
>> it is equivalent .
>> 2) NRT refresh automatically searcher/reader it dont call commit. I based
>> my implementation using nrt on http://stackoverflow.com/qu
>> estions/17993960/lucene-4-4-0-new-controlledrealtimereopenth
>> read-sample-usage. In this example commit is executed for every crud
>> operation in synchronous way but in general it is advised to use a batch
>> thread because the commit is a long operation. *So it is not clear how
>> to do the commit in a near-real time system with a indefinite index size.*
>>      2.a if the commit is synchronous , i can use user data because it is
>> used before a commit, every commit has a different user data and i can
>> trace the transactions changes.But in general a commit can requires also
>> minutes for be completed so then it dont seams a real solution in a near
>> real time solution.
>>     2.b if the commit is async, it is executed every X times (or better
>> how memory if full) , the commit can not be used for tracing the
>> transactions and i can pass a trnsaction id associated with a lucene
>> commit. I can add a mutex in crud ( when i loading uncommit data) i m sure
>> the last uncummit Index is aligned to the last transaction id X, so there
>> is no overlappind and the crud block is very fast when happens.But how to
>> grant that the commit is related to the last CommitIndex what i loaded?
>> Maybe if i introduce that mutex in a custom mergePolicy?
>> It is right what i wrote until now ?The best solution is 2.b? In this
>> case how to grant the commit is done based on the uncommit data loaded in a
>> specific commitIndex?
>>
>>
>>
>>
>>
>> 2017-03-22 15:32 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]>
>> :
>>
>>> Hi, I think you forgot to CC the lucene user's list (
>>> [EMAIL PROTECTED]) in your reply?  Can you resend?
>>>
>>> Thanks.
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Wed, Mar 22, 2017 at 9:02 AM, Cristian Lorenzetto <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> hi , i m thinking about what you told me in previous message and how to
>>>> solve the corruption problem and the problem about commit operation
>>>> executed in async way.
>>>>
>>>> I m thinking to create a simple transaction log in a file.
>>>> i use a long atomic sequence for a ordinable transaction id.
>>>>
>>>> when i make a new operation
>>>> 1) generate new incremental transaction id
>>>> 2) save the operation abstract info in transaction log associated to

> I deduce the transaction range not using the segment corrupted but the
> corrected segments. The transaction id is incremental and i imagine segment
> are saved sequentelly so if it is missing the segment 5 , reading the
> correct segment 4 i can find the maximunn transaction id A , reading the
> segment 6 i can find the minimum transaction id B so i can deduce the hole
> , the range is [A+1,B-1] ... making a query in db i reaload the
> corrisponding document and i add again in lucene this missing documents.
>
>
> 2017-03-23 15:28 GMT+01:00 Cristian Lorenzetto <
> [EMAIL PROTECTED]>:
>
>> I deduce the transaction range not using the segment corrupted but the
>> corrected segments. The transaction id is incremental and i imagine segment
>> are saved sequentelly so if it is missing the segment 5 , reading the
>> correct segment 4 i can find the maximunn transaction id A , reading the
>> segment 6 i can find the minimum transaction id B so i can deduce the hole
>> , the range is [A+1,B-1] ... making a query in db i reaload the
>> corrisponding document and i add again in lucene this missing documents.
>>
>>
>> 2017-03-23 15:17 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]>
>> :
>>
>>> Lucene corruption should be rare and only due to bad hardware; if you
>>> are seeing otherwise we really should get to the root cause.
>>>
>>> Mapping documents to each segment will not be easy in general,
>>> especially if that segment is now corrupted so you can't search it.
>>>
>>> Documents lost because of power loss / OS crash while indexing can be
>>> more common, and its for that use case that the sequence numbers /
>>> transaction log should be helpful.
>>>
>>> Mike McCandless
>>>
>>> http://blog.mikemccandless.com
>>>
>>> On Thu, Mar 23, 2017 at 10:12 AM, Cristian Lorenzetto <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Yes exactly. I saw, working in the past in systems using lucene (for
>>>> example alfresco projects),  lucene corruption happens sometimes and every
>>>> time the building requires a lot of times ... so i thougth a way for
>>>> accelerating the fixing of a corruption index. In addition there is a rare
>>>> case not described here ( If after a database commit lucene throws a
>>>> exception for exampe disk is full ) there is a possibility of a
>>>>  disalignement from the database and the lucene index. With this system
>>>> these problems could be solved automatically. In database every row has a
>>>> property with trasaction id.  So if i know in lucene is missing a segment 6
>>>> , corrisponds to   transactions range[ 1000, 1050] so i can reload in a
>>>> query in database just corrisponding rows.
>>>>
>>>> 2017-03-23 14:59 GMT+01:00 Michael McCandless <
>>>> [EMAIL PROTECTED]>:
>>>>
>>>>> You should be able to use the sequence numbers returned by IndexWriter
>>>>> operations to "know" which operations made it into the commit and which did
>>>>> not, and then on disaster recovery replay only those operations that didn't
>>>>> make it?
>>>>>
>>>>> Mike McCandless
>>>>>
>>>>> http://blog.mikemccandless.com
>>>>>
>>>>> On Thu, Mar 23, 2017 at 5:53 AM, Cristian Lorenzetto <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Errata corridge/integration for questions related to previous my post
>>>>>>
>>>>>> I studied a bit this lucene classes for understanding:
>>>>>> 1) setCommitData is designed for versioning the index , not for
>>>>>> passing a transaction log. However if userdata is different for every
>>>>>> transactionid it is equivalent .
>>>>>> 2) NRT refresh automatically searcher/reader it dont call commit. I

> If you use a single thread then, yes, segments are sequential.
>
> But if e.g. you are updating documents, then deletions (because a document
> was replaced) are recorded against different segments, so merely dropping
> the corrupted segment will mean you don't drop the deletions.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Thu, Mar 23, 2017 at 10:29 AM, Cristian Lorenzetto <
> [EMAIL PROTECTED]> wrote:
>
>> I deduce the transaction range not using the segment corrupted but the
>> corrected segments. The transaction id is incremental and i imagine segment
>> are saved sequentelly so if it is missing the segment 5 , reading the
>> correct segment 4 i can find the maximunn transaction id A , reading the
>> segment 6 i can find the minimum transaction id B so i can deduce the hole
>> , the range is [A+1,B-1] ... making a query in db i reaload the
>> corrisponding document and i add again in lucene this missing documents.
>>
>>
>> 2017-03-23 15:28 GMT+01:00 Cristian Lorenzetto <
>> [EMAIL PROTECTED]>:
>>
>>> I deduce the transaction range not using the segment corrupted but the
>>> corrected segments. The transaction id is incremental and i imagine segment
>>> are saved sequentelly so if it is missing the segment 5 , reading the
>>> correct segment 4 i can find the maximunn transaction id A , reading the
>>> segment 6 i can find the minimum transaction id B so i can deduce the hole
>>> , the range is [A+1,B-1] ... making a query in db i reaload the
>>> corrisponding document and i add again in lucene this missing documents.
>>>
>>>
>>> 2017-03-23 15:17 GMT+01:00 Michael McCandless <[EMAIL PROTECTED]
>>> >:
>>>
>>>> Lucene corruption should be rare and only due to bad hardware; if you
>>>> are seeing otherwise we really should get to the root cause.
>>>>
>>>> Mapping documents to each segment will not be easy in general,
>>>> especially if that segment is now corrupted so you can't search it.
>>>>
>>>> Documents lost because of power loss / OS crash while indexing can be
>>>> more common, and its for that use case that the sequence numbers /
>>>> transaction log should be helpful.
>>>>
>>>> Mike McCandless
>>>>
>>>> http://blog.mikemccandless.com
>>>>
>>>> On Thu, Mar 23, 2017 at 10:12 AM, Cristian Lorenzetto <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> Yes exactly. I saw, working in the past in systems using lucene (for
>>>>> example alfresco projects),  lucene corruption happens sometimes and every
>>>>> time the building requires a lot of times ... so i thougth a way for
>>>>> accelerating the fixing of a corruption index. In addition there is a rare
>>>>> case not described here ( If after a database commit lucene throws a