Hi

 

My application is cluster in jobss application servers & lucene
directory was shared.

 

Concurrently 5 user access same lucene directory for searching document

That time I got bellow exception

org.apache.lucene.store.AlreadyClosedException: this IndexReader is
closed

 

is there a way to handle such error

 

Thanks & Regards

Yogesh

You don't say exactly how you are dealing with the concurrent access
(one shared Reader/Searcher? Each user with own Reader/Searcher?
Something else?) but the underlying problem is that the reader has
been closed while something else is still using it.  This can easily
happen in a multi-threaded setup: thread A might close the reader
while thread B is still using it.

Options include

 not closing the old reader straight away when open a new one, but
instead stash it somewhere and close it later when you can be
confident nothing else will be using it.

 using the SearcherManager class from Lucene In Action 2nd edition, or
an equivalent.

 catching the AlreadyClosed exception and restarting the search with a
current reader
--
Ian.
On Tue, Apr 5, 2011 at 12:56 PM, Yogesh Dabhi <[EMAIL PROTECTED]> wrote:
> Hi
>
>
>
> My application is cluster in jobss application servers & lucene
> directory was shared.
>
>
>
> Concurrently 5 user access same lucene directory for searching document
>
> That time I got bellow exception
>
> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
>
>
>
> is there a way to handle such error
>
>
>
> Thanks & Regards
>
> Yogesh
>
>

---------------------------------------------------------------------

Only to second this explanation.

I got the same exception in a web application with a single IndexReader, accessed by many threads.

The index gets updated every half hour or so, so I closed the old IndexReader and opened a new one every now and then.

Even though the method for obtaining the IndexReader was synchronized, with tens of thousands of queries a day,
I started getting the AlreadyClosed exception when one thread triggered the closure of the IndexReader which another thread was still using.

Piotr

 
On 5 Apr 2011, at 15:46, Ian Lea wrote:

> You don't say exactly how you are dealing with the concurrent access
> (one shared Reader/Searcher? Each user with own Reader/Searcher?
> Something else?) but the underlying problem is that the reader has
> been closed while something else is still using it.  This can easily
> happen in a multi-threaded setup: thread A might close the reader
> while thread B is still using it.
>
> Options include
>
> not closing the old reader straight away when open a new one, but
> instead stash it somewhere and close it later when you can be
> confident nothing else will be using it.
>
> using the SearcherManager class from Lucene In Action 2nd edition, or
> an equivalent.
>
> catching the AlreadyClosed exception and restarting the search with a
> current reader
>
>
> --
> Ian.
>
>
> On Tue, Apr 5, 2011 at 12:56 PM, Yogesh Dabhi <[EMAIL PROTECTED]> wrote:
>> Hi
>>
>>
>>
>> My application is cluster in jobss application servers & lucene
>> directory was shared.
>>
>>
>>
>> Concurrently 5 user access same lucene directory for searching document
>>
>> That time I got bellow exception
>>
>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>> closed
>>
>>
>>
>> is there a way to handle such error
>>
>>
>>
>> Thanks & Regards
>>
>> Yogesh
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

You are trying to access the reader which is already closed by some other
thread.

1. Keep a reference count for the reader you create.
2. Have a common function through which all functions will retrieve Reader
objects
3. Once the index got changed, create a new reader, do warmup
4. When the new reader is ready then the common function should return this
for furture request and old reader should go in passive mode.
5. The old reader should be closed when the reference count is zero.

Regards
Aditya
www.findbestopensource.com
On Wed, Apr 6, 2011 at 7:20 PM, Piotr Pezik <[EMAIL PROTECTED]> wrote:

> Only to second this explanation.
>
> I got the same exception in a web application with a single IndexReader,
> accessed by many threads.
>
> The index gets updated every half hour or so, so I closed the old
> IndexReader and opened a new one every now and then.
>
> Even though the method for obtaining the IndexReader was synchronized, with
> tens of thousands of queries a day,
> I started getting the AlreadyClosed exception when one thread triggered the
> closure of the IndexReader which another thread was still using.
>
>
>
> Piotr
>
>
> On 5 Apr 2011, at 15:46, Ian Lea wrote:
>
> > You don't say exactly how you are dealing with the concurrent access
> > (one shared Reader/Searcher? Each user with own Reader/Searcher?
> > Something else?) but the underlying problem is that the reader has
> > been closed while something else is still using it.  This can easily
> > happen in a multi-threaded setup: thread A might close the reader
> > while thread B is still using it.
> >
> > Options include
> >
> > not closing the old reader straight away when open a new one, but
> > instead stash it somewhere and close it later when you can be
> > confident nothing else will be using it.
> >
> > using the SearcherManager class from Lucene In Action 2nd edition, or
> > an equivalent.
> >
> > catching the AlreadyClosed exception and restarting the search with a
> > current reader
> >
> >
> > --
> > Ian.
> >
> >
> > On Tue, Apr 5, 2011 at 12:56 PM, Yogesh Dabhi <[EMAIL PROTECTED]> wrote:
> >> Hi
> >>
> >>
> >>
> >> My application is cluster in jobss application servers & lucene
> >> directory was shared.
> >>
> >>
> >>
> >> Concurrently 5 user access same lucene directory for searching document
> >>
> >> That time I got bellow exception
> >>
> >> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> >> closed
> >>
> >>
> >>
> >> is there a way to handle such error
> >>
> >>
> >>
> >> Thanks & Regards
> >>
> >> Yogesh
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > For additional commands, e-mail: [EMAIL PROTECTED]
> >
>
>

Building on Aditya's comments.

1. Lucene in Action has a SearchManager class which does most of 1-5 and
manages indexreaders.
2. Reference counting is not the best approach, it is too low level and
error prone. Instead
  Use ReadWriteLock
<http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReentrantReadWriteLock.html>manage
index ( or do reference counting for u)

A ReadWriteLock maintains a pair of associated
locks<http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/Lock.html>,
one for read-only operations and one for writing. The read
lock<http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html#readLock%28%29>may
be held simultaneously by multiple reader threads, so long as there
are
no writers. The write
lock<http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html#writeLock%28%29>is
exclusive.

So basically, you wrap the lucene's searcher into your SearchManager class,
which exposes its own API for search and forwards the requests to underlying
searcher.

 The search and reopen will sync up by using ReadWriteLock . search takes
readlock and reopen takes writelock and that will take care of any open
search results etc.

PS:  To speed up the reopening of index use indexreader.reopen() instead of
closing it off and then open again. It is much faster. (Documented)
Thanks & Regards
Umesh Prasad

On Thu, Apr 7, 2011 at 11:38 AM, findbestopensource <
[EMAIL PROTECTED]> wrote:

> You are trying to access the reader which is already closed by some other
> thread.
>
> 1. Keep a reference count for the reader you create.
> 2. Have a common function through which all functions will retrieve Reader
> objects
> 3. Once the index got changed, create a new reader, do warmup
> 4. When the new reader is ready then the common function should return this
> for furture request and old reader should go in passive mode.
> 5. The old reader should be closed when the reference count is zero.
>
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
> On Wed, Apr 6, 2011 at 7:20 PM, Piotr Pezik <[EMAIL PROTECTED]> wrote:
>
> > Only to second this explanation.
> >
> > I got the same exception in a web application with a single IndexReader,
> > accessed by many threads.
> >
> > The index gets updated every half hour or so, so I closed the old
> > IndexReader and opened a new one every now and then.
> >
> > Even though the method for obtaining the IndexReader was synchronized,
> with
> > tens of thousands of queries a day,
> > I started getting the AlreadyClosed exception when one thread triggered
> the
> > closure of the IndexReader which another thread was still using.
> >
> >
> >
> > Piotr
> >
> >
> > On 5 Apr 2011, at 15:46, Ian Lea wrote:
> >
> > > You don't say exactly how you are dealing with the concurrent access
> > > (one shared Reader/Searcher? Each user with own Reader/Searcher?
> > > Something else?) but the underlying problem is that the reader has
> > > been closed while something else is still using it.  This can easily
> > > happen in a multi-threaded setup: thread A might close the reader
> > > while thread B is still using it.
> > >
> > > Options include
> > >
> > > not closing the old reader straight away when open a new one, but
> > > instead stash it somewhere and close it later when you can be
> > > confident nothing else will be using it.
> > >
> > > using the SearcherManager class from Lucene In Action 2nd edition, or
> > > an equivalent.
> > >
> > > catching the AlreadyClosed exception and restarting the search with a
> > > current reader
> > >
> > >
> > > --
> > > Ian.
> > >
> > >
> > > On Tue, Apr 5, 2011 at 12:56 PM, Yogesh Dabhi <[EMAIL PROTECTED]>
> wrote:
> > >> Hi
> > >>
> > >>
> > >>
> > >> My application is cluster in jobss application servers & lucene
> > >> directory was shared.
> > >>
> > >>
> > >>
> > >> Concurrently 5 user access same lucene directory for searching
> document

Thanks & Regards
Umesh Prasad

Actually, you shouldn't need a separate ReadWriteLock.

SearcherManager (from LIA2's source code, freely downloadable from
http://www.manning.com/hatcher3, with Apache License 2.0) handles the
reference counting (so a reader is never closed while any search from
other threads are stil in flight), and also handles reopening and
warming in the background.

You pass it either an IndexWriter (in which case it pulls
near-real-time readers), or a Directory (in which case it opens a
"normal" IndexReader and then uses reopen).

All the app has to do is tell SearcherManager when to reopen.

Mike

http://blog.mikemccandless.com

On Thu, Apr 7, 2011 at 2:08 AM, findbestopensource
<[EMAIL PROTECTED]> wrote:
> You are trying to access the reader which is already closed by some other
> thread.
>
> 1. Keep a reference count for the reader you create.
> 2. Have a common function through which all functions will retrieve Reader
> objects
> 3. Once the index got changed, create a new reader, do warmup
> 4. When the new reader is ready then the common function should return this
> for furture request and old reader should go in passive mode.
> 5. The old reader should be closed when the reference count is zero.
>
> Regards
> Aditya
> www.findbestopensource.com
>
>
> On Wed, Apr 6, 2011 at 7:20 PM, Piotr Pezik <[EMAIL PROTECTED]> wrote:
>
>> Only to second this explanation.
>>
>> I got the same exception in a web application with a single IndexReader,
>> accessed by many threads.
>>
>> The index gets updated every half hour or so, so I closed the old
>> IndexReader and opened a new one every now and then.
>>
>> Even though the method for obtaining the IndexReader was synchronized, with
>> tens of thousands of queries a day,
>> I started getting the AlreadyClosed exception when one thread triggered the
>> closure of the IndexReader which another thread was still using.
>>
>>
>>
>> Piotr
>>
>>
>> On 5 Apr 2011, at 15:46, Ian Lea wrote:
>>
>> > You don't say exactly how you are dealing with the concurrent access
>> > (one shared Reader/Searcher? Each user with own Reader/Searcher?
>> > Something else?) but the underlying problem is that the reader has
>> > been closed while something else is still using it.  This can easily
>> > happen in a multi-threaded setup: thread A might close the reader
>> > while thread B is still using it.
>> >
>> > Options include
>> >
>> > not closing the old reader straight away when open a new one, but
>> > instead stash it somewhere and close it later when you can be
>> > confident nothing else will be using it.
>> >
>> > using the SearcherManager class from Lucene In Action 2nd edition, or
>> > an equivalent.
>> >
>> > catching the AlreadyClosed exception and restarting the search with a
>> > current reader
>> >
>> >
>> > --
>> > Ian.
>> >
>> >
>> > On Tue, Apr 5, 2011 at 12:56 PM, Yogesh Dabhi <[EMAIL PROTECTED]> wrote:
>> >> Hi
>> >>
>> >>
>> >>
>> >> My application is cluster in jobss application servers & lucene
>> >> directory was shared.
>> >>
>> >>
>> >>
>> >> Concurrently 5 user access same lucene directory for searching document
>> >>
>> >> That time I got bellow exception
>> >>
>> >> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>> >> closed
>> >>
>> >>
>> >>
>> >> is there a way to handle such error
>> >>
>> >>
>> >>
>> >> Thanks & Regards
>> >>
>> >> Yogesh
>> >>
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: [EMAIL PROTECTED]
>> > For additional commands, e-mail: [EMAIL PROTECTED]
>> >
>>
>>
>

---------------------------------------------------------------------

Thanks for the tips.

Does the LIA SearcherManager provide/manage access to the IndexReader as well?

In other words, is it as easy as:

searcher.getIndexReader()

?

I need some of the the Reader methods in addition to the Searcher ones.

Piotr

On 8 Apr 2011, at 12:21, Michael McCandless wrote:

> Actually, you shouldn't need a separate ReadWriteLock.
>
> SearcherManager (from LIA2's source code, freely downloadable from
> http://www.manning.com/hatcher3, with Apache License 2.0) handles the
> reference counting (so a reader is never closed while any search from
> other threads are stil in flight), and also handles reopening and
> warming in the background.
>
> You pass it either an IndexWriter (in which case it pulls
> near-real-time readers), or a Directory (in which case it opens a
> "normal" IndexReader and then uses reopen).
>
> All the app has to do is tell SearcherManager when to reopen.
>
> Mike
>
> http://blog.mikemccandless.com
>
> On Thu, Apr 7, 2011 at 2:08 AM, findbestopensource
> <[EMAIL PROTECTED]> wrote:
>> You are trying to access the reader which is already closed by some other
>> thread.
>>
>> 1. Keep a reference count for the reader you create.
>> 2. Have a common function through which all functions will retrieve Reader
>> objects
>> 3. Once the index got changed, create a new reader, do warmup
>> 4. When the new reader is ready then the common function should return this
>> for furture request and old reader should go in passive mode.
>> 5. The old reader should be closed when the reference count is zero.
>>
>> Regards
>> Aditya
>> www.findbestopensource.com
>>
>>
>> On Wed, Apr 6, 2011 at 7:20 PM, Piotr Pezik <[EMAIL PROTECTED]> wrote:
>>
>>> Only to second this explanation.
>>>
>>> I got the same exception in a web application with a single IndexReader,
>>> accessed by many threads.
>>>
>>> The index gets updated every half hour or so, so I closed the old
>>> IndexReader and opened a new one every now and then.
>>>
>>> Even though the method for obtaining the IndexReader was synchronized, with
>>> tens of thousands of queries a day,
>>> I started getting the AlreadyClosed exception when one thread triggered the
>>> closure of the IndexReader which another thread was still using.
>>>
>>>
>>>
>>> Piotr
>>>
>>>
>>> On 5 Apr 2011, at 15:46, Ian Lea wrote:
>>>
>>>> You don't say exactly how you are dealing with the concurrent access
>>>> (one shared Reader/Searcher? Each user with own Reader/Searcher?
>>>> Something else?) but the underlying problem is that the reader has
>>>> been closed while something else is still using it.  This can easily
>>>> happen in a multi-threaded setup: thread A might close the reader
>>>> while thread B is still using it.
>>>>
>>>> Options include
>>>>
>>>> not closing the old reader straight away when open a new one, but
>>>> instead stash it somewhere and close it later when you can be
>>>> confident nothing else will be using it.
>>>>
>>>> using the SearcherManager class from Lucene In Action 2nd edition, or
>>>> an equivalent.
>>>>
>>>> catching the AlreadyClosed exception and restarting the search with a
>>>> current reader
>>>>
>>>>
>>>> --
>>>> Ian.
>>>>
>>>>
>>>> On Tue, Apr 5, 2011 at 12:56 PM, Yogesh Dabhi <[EMAIL PROTECTED]> wrote:
>>>>> Hi
>>>>>
>>>>>
>>>>>
>>>>> My application is cluster in jobss application servers & lucene
>>>>> directory was shared.
>>>>>
>>>>>
>>>>>
>>>>> Concurrently 5 user access same lucene directory for searching document
>>>>>
>>>>> That time I got bellow exception
>>>>>
>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
>>>>> closed
>>>>>
>>>>>
>>>>>
>>>>> is there a way to handle such error
>>>>>
>>>>>
>>>>>
>>>>> Thanks & Regards
>>>>>
>>>>> Yogesh
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>> For additional commands, e-mail: [EMAIL PROTECTED]

Yes, once you get the searcher you're free to access its underlying
IndexReader.  Just don't close it!

And, be sure you call SearcherManager.release in a finally clause!
Else the reader will not be closed...

Mike

http://blog.mikemccandless.com

On Fri, Apr 8, 2011 at 6:54 AM, Piotr Pezik <[EMAIL PROTECTED]> wrote:
> Thanks for the tips.
>
> Does the LIA SearcherManager provide/manage access to the IndexReader as well?
>
> In other words, is it as easy as:
>
> searcher.getIndexReader()
>
> ?
>
> I need some of the the Reader methods in addition to the Searcher ones.
>
> Piotr
>
> On 8 Apr 2011, at 12:21, Michael McCandless wrote:
>
>> Actually, you shouldn't need a separate ReadWriteLock.
>>
>> SearcherManager (from LIA2's source code, freely downloadable from
>> http://www.manning.com/hatcher3, with Apache License 2.0) handles the
>> reference counting (so a reader is never closed while any search from
>> other threads are stil in flight), and also handles reopening and
>> warming in the background.
>>
>> You pass it either an IndexWriter (in which case it pulls
>> near-real-time readers), or a Directory (in which case it opens a
>> "normal" IndexReader and then uses reopen).
>>
>> All the app has to do is tell SearcherManager when to reopen.
>>
>> Mike
>>
>> http://blog.mikemccandless.com
>>
>> On Thu, Apr 7, 2011 at 2:08 AM, findbestopensource
>> <[EMAIL PROTECTED]> wrote:
>>> You are trying to access the reader which is already closed by some other
>>> thread.
>>>
>>> 1. Keep a reference count for the reader you create.
>>> 2. Have a common function through which all functions will retrieve Reader
>>> objects
>>> 3. Once the index got changed, create a new reader, do warmup
>>> 4. When the new reader is ready then the common function should return this
>>> for furture request and old reader should go in passive mode.
>>> 5. The old reader should be closed when the reference count is zero.
>>>
>>> Regards
>>> Aditya
>>> www.findbestopensource.com
>>>
>>>
>>> On Wed, Apr 6, 2011 at 7:20 PM, Piotr Pezik <[EMAIL PROTECTED]> wrote:
>>>
>>>> Only to second this explanation.
>>>>
>>>> I got the same exception in a web application with a single IndexReader,
>>>> accessed by many threads.
>>>>
>>>> The index gets updated every half hour or so, so I closed the old
>>>> IndexReader and opened a new one every now and then.
>>>>
>>>> Even though the method for obtaining the IndexReader was synchronized, with
>>>> tens of thousands of queries a day,
>>>> I started getting the AlreadyClosed exception when one thread triggered the
>>>> closure of the IndexReader which another thread was still using.
>>>>
>>>>
>>>>
>>>> Piotr
>>>>
>>>>
>>>> On 5 Apr 2011, at 15:46, Ian Lea wrote:
>>>>
>>>>> You don't say exactly how you are dealing with the concurrent access
>>>>> (one shared Reader/Searcher? Each user with own Reader/Searcher?
>>>>> Something else?) but the underlying problem is that the reader has
>>>>> been closed while something else is still using it.  This can easily
>>>>> happen in a multi-threaded setup: thread A might close the reader
>>>>> while thread B is still using it.
>>>>>
>>>>> Options include
>>>>>
>>>>> not closing the old reader straight away when open a new one, but
>>>>> instead stash it somewhere and close it later when you can be
>>>>> confident nothing else will be using it.
>>>>>
>>>>> using the SearcherManager class from Lucene In Action 2nd edition, or
>>>>> an equivalent.
>>>>>
>>>>> catching the AlreadyClosed exception and restarting the search with a
>>>>> current reader
>>>>>
>>>>>
>>>>> --
>>>>> Ian.
>>>>>
>>>>>
>>>>> On Tue, Apr 5, 2011 at 12:56 PM, Yogesh Dabhi <[EMAIL PROTECTED]> wrote:
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>>
>>>>>> My application is cluster in jobss application servers & lucene
>>>>>> directory was shared.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Concurrently 5 user access same lucene directory for searching document
>>>>>>
>>>>>> That time I got bellow exception
>>>>>>
>>>>>> org.apache.lucene.store.AlreadyClosedException: this IndexReader is

You might have closed the IndexReader object but trying to access the search
results.

Regards
Aditya
www.findbestopensource.com

On Tue, Apr 5, 2011 at 5:26 PM, Yogesh Dabhi <[EMAIL PROTECTED]> wrote:

> Hi
>
>
>
> My application is cluster in jobss application servers & lucene
> directory was shared.
>
>
>
> Concurrently 5 user access same lucene directory for searching document
>
> That time I got bellow exception
>
> org.apache.lucene.store.AlreadyClosedException: this IndexReader is
> closed
>
>
>
> is there a way to handle such error
>
>
>
> Thanks & Regards
>
> Yogesh
>
>