Hi,

I build an RAMDirectory on a FSDirectory, and would like the writer
associated with the RAMDirectory to periodically write to hard drive.

Is this achievable?

Thanks.

Hi Cheng,

It seems that you use a RAMDirectory for *caching*, otherwise it makes no
sense to write changes back. In recent Lucene versions, this is not a good
idea, especially for large indexes (RAMDirectory eats your heap space,
allocates millions of small byte[] arrays,...). If you need something like a
caching Directory and you are working on a 64bit platform, you can use
MMapDirectory (where the operating system kernel manages the read/write
between disk an memory). MMapDirectory is returned by default for
FSDirectory.open() on most 64 bit platforms. The good thing: the "caching"
space is outside your JVM heap, so does not slowdown the garbage collector.
So be sure to *not* allocate too much heap space (-Xmx) to your search app,
only the minimum needed to execute it and leave the rest of your RAM
available for the OS kernel to manage FS cache.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Cheng [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, February 05, 2012 7:56 AM
> To: [EMAIL PROTECTED]
> Subject: Configure writer to write to FSDirectory?
>
> Hi,
>
> I build an RAMDirectory on a FSDirectory, and would like the writer
associated
> with the RAMDirectory to periodically write to hard drive.
>
> Is this achievable?
>
> Thanks.
---------------------------------------------------------------------

Hi Uwe,

My challenge is that I need to update/modify the indexes frequently while
providing the search capability. I was trying to use FSDirectory, but found
out that the reading and writing from/to FSDirectory is unbearably slow. So
I now am trying the RAMDirectory, which is fast.

I don't know of  MMapDirectory, and wonder if it is as fast as RAMDirectory.
On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:

> Hi Cheng,
>
> It seems that you use a RAMDirectory for *caching*, otherwise it makes no
> sense to write changes back. In recent Lucene versions, this is not a good
> idea, especially for large indexes (RAMDirectory eats your heap space,
> allocates millions of small byte[] arrays,...). If you need something like
> a
> caching Directory and you are working on a 64bit platform, you can use
> MMapDirectory (where the operating system kernel manages the read/write
> between disk an memory). MMapDirectory is returned by default for
> FSDirectory.open() on most 64 bit platforms. The good thing: the "caching"
> space is outside your JVM heap, so does not slowdown the garbage collector.
> So be sure to *not* allocate too much heap space (-Xmx) to your search app,
> only the minimum needed to execute it and leave the rest of your RAM
> available for the OS kernel to manage FS cache.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [EMAIL PROTECTED]
>
>
> > -----Original Message-----
> > From: Cheng [mailto:[EMAIL PROTECTED]]
> > Sent: Sunday, February 05, 2012 7:56 AM
> > To: [EMAIL PROTECTED]
> > Subject: Configure writer to write to FSDirectory?
> >
> > Hi,
> >
> > I build an RAMDirectory on a FSDirectory, and would like the writer
> associated
> > with the RAMDirectory to periodically write to hard drive.
> >
> > Is this achievable?
> >
> > Thanks.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>

Are you using near-real-time readers?

(IndexReader.open(IndexWriter))

Mike McCandless

http://blog.mikemccandless.com

On Sun, Feb 5, 2012 at 9:03 AM, Cheng <[EMAIL PROTECTED]> wrote:
> Hi Uwe,
>
> My challenge is that I need to update/modify the indexes frequently while
> providing the search capability. I was trying to use FSDirectory, but found
> out that the reading and writing from/to FSDirectory is unbearably slow. So
> I now am trying the RAMDirectory, which is fast.
>
> I don't know of  MMapDirectory, and wonder if it is as fast as RAMDirectory.
>
>
> On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
>> Hi Cheng,
>>
>> It seems that you use a RAMDirectory for *caching*, otherwise it makes no
>> sense to write changes back. In recent Lucene versions, this is not a good
>> idea, especially for large indexes (RAMDirectory eats your heap space,
>> allocates millions of small byte[] arrays,...). If you need something like
>> a
>> caching Directory and you are working on a 64bit platform, you can use
>> MMapDirectory (where the operating system kernel manages the read/write
>> between disk an memory). MMapDirectory is returned by default for
>> FSDirectory.open() on most 64 bit platforms. The good thing: the "caching"
>> space is outside your JVM heap, so does not slowdown the garbage collector.
>> So be sure to *not* allocate too much heap space (-Xmx) to your search app,
>> only the minimum needed to execute it and leave the rest of your RAM
>> available for the OS kernel to manage FS cache.
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: [EMAIL PROTECTED]
>>
>>
>> > -----Original Message-----
>> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> > Sent: Sunday, February 05, 2012 7:56 AM
>> > To: [EMAIL PROTECTED]
>> > Subject: Configure writer to write to FSDirectory?
>> >
>> > Hi,
>> >
>> > I build an RAMDirectory on a FSDirectory, and would like the writer
>> associated
>> > with the RAMDirectory to periodically write to hard drive.
>> >
>> > Is this achievable?
>> >
>> > Thanks.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>

---------------------------------------------------------------------

I was trying to, but don't know how to even I read some of your blogs.

On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
[EMAIL PROTECTED]> wrote:

> Are you using near-real-time readers?
>
> (IndexReader.open(IndexWriter))
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > Hi Uwe,
> >
> > My challenge is that I need to update/modify the indexes frequently while
> > providing the search capability. I was trying to use FSDirectory, but
> found
> > out that the reading and writing from/to FSDirectory is unbearably slow.
> So
> > I now am trying the RAMDirectory, which is fast.
> >
> > I don't know of  MMapDirectory, and wonder if it is as fast as
> RAMDirectory.
> >
> >
> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Cheng,
> >>
> >> It seems that you use a RAMDirectory for *caching*, otherwise it makes
> no
> >> sense to write changes back. In recent Lucene versions, this is not a
> good
> >> idea, especially for large indexes (RAMDirectory eats your heap space,
> >> allocates millions of small byte[] arrays,...). If you need something
> like
> >> a
> >> caching Directory and you are working on a 64bit platform, you can use
> >> MMapDirectory (where the operating system kernel manages the read/write
> >> between disk an memory). MMapDirectory is returned by default for
> >> FSDirectory.open() on most 64 bit platforms. The good thing: the
> "caching"
> >> space is outside your JVM heap, so does not slowdown the garbage
> collector.
> >> So be sure to *not* allocate too much heap space (-Xmx) to your search
> app,
> >> only the minimum needed to execute it and leave the rest of your RAM
> >> available for the OS kernel to manage FS cache.
> >>
> >> Uwe
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: [EMAIL PROTECTED]
> >>
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> > Sent: Sunday, February 05, 2012 7:56 AM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Configure writer to write to FSDirectory?
> >> >
> >> > Hi,
> >> >
> >> > I build an RAMDirectory on a FSDirectory, and would like the writer
> >> associated
> >> > with the RAMDirectory to periodically write to hard drive.
> >> >
> >> > Is this achievable?
> >> >
> >> > Thanks.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [EMAIL PROTECTED]
> >> For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>

If you can use NRTManager and SearcherManager things should be easy
and blazingly fast rather than unbearably slow.  The latter phrase is
not one often associated with lucene.

IndexWriter iw = new IndexWriter(whatever - some standard disk index);
NRTManager nrtm = new NRTManager(iw, null);
NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
ropt.setXxx(...);
...
ropt.start();

SearcherManager srchm = nrtm.getSearcherManager(b);

Then add docs to your index via nrtm.addDocument(d), update with
nrtm.updateDocument(...), and to search use

IndexSearcher searcher = srchm.acquire();
try {
  search ...
} finally {
 srchm.release(searcher);
}

All thread safe so you don't have to worry about any complications
there.  And I bet it'll be blindingly fast.

Don't forget to close() things down at the end.
--
Ian.

On Mon, Feb 6, 2012 at 12:15 AM, Cheng <[EMAIL PROTECTED]> wrote:
> I was trying to, but don't know how to even I read some of your blogs.
>
> On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
> [EMAIL PROTECTED]> wrote:
>
>> Are you using near-real-time readers?
>>
>> (IndexReader.open(IndexWriter))
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> > Hi Uwe,
>> >
>> > My challenge is that I need to update/modify the indexes frequently while
>> > providing the search capability. I was trying to use FSDirectory, but
>> found
>> > out that the reading and writing from/to FSDirectory is unbearably slow.
>> So
>> > I now am trying the RAMDirectory, which is fast.
>> >
>> > I don't know of  MMapDirectory, and wonder if it is as fast as
>> RAMDirectory.
>> >
>> >
>> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hi Cheng,
>> >>
>> >> It seems that you use a RAMDirectory for *caching*, otherwise it makes
>> no
>> >> sense to write changes back. In recent Lucene versions, this is not a
>> good
>> >> idea, especially for large indexes (RAMDirectory eats your heap space,
>> >> allocates millions of small byte[] arrays,...). If you need something
>> like
>> >> a
>> >> caching Directory and you are working on a 64bit platform, you can use
>> >> MMapDirectory (where the operating system kernel manages the read/write
>> >> between disk an memory). MMapDirectory is returned by default for
>> >> FSDirectory.open() on most 64 bit platforms. The good thing: the
>> "caching"
>> >> space is outside your JVM heap, so does not slowdown the garbage
>> collector.
>> >> So be sure to *not* allocate too much heap space (-Xmx) to your search
>> app,
>> >> only the minimum needed to execute it and leave the rest of your RAM
>> >> available for the OS kernel to manage FS cache.
>> >>
>> >> Uwe
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: [EMAIL PROTECTED]
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> >> > Sent: Sunday, February 05, 2012 7:56 AM
>> >> > To: [EMAIL PROTECTED]
>> >> > Subject: Configure writer to write to FSDirectory?
>> >> >
>> >> > Hi,
>> >> >
>> >> > I build an RAMDirectory on a FSDirectory, and would like the writer
>> >> associated
>> >> > with the RAMDirectory to periodically write to hard drive.
>> >> >
>> >> > Is this achievable?
>> >> >
>> >> > Thanks.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> >> For additional commands, e-mail: [EMAIL PROTECTED]
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>

---------------------------------------------------------------------

I don't understand this following portion:

IndexWriter iw = new IndexWriter(whatever - some standard disk index);
NRTManager nrtm = new NRTManager(iw, null);
NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
ropt.setXxx(...);
....
ropt.start();

I have a java ExecutorServices instance running which take care of my own
applications. I don't know how this NRTManagerReopenThread works with my
own ExecutorService instance.

Can both work together? How can the NRTManagerReopenThread instance ropt be
plugged into my own multithreading framework?

On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:

> If you can use NRTManager and SearcherManager things should be easy
> and blazingly fast rather than unbearably slow.  The latter phrase is
> not one often associated with lucene.
>
> IndexWriter iw = new IndexWriter(whatever - some standard disk index);
> NRTManager nrtm = new NRTManager(iw, null);
> NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
> ropt.setXxx(...);
> ...
> ropt.start();
>
> SearcherManager srchm = nrtm.getSearcherManager(b);
>
> Then add docs to your index via nrtm.addDocument(d), update with
> nrtm.updateDocument(...), and to search use
>
> IndexSearcher searcher = srchm.acquire();
> try {
>  search ...
> } finally {
>  srchm.release(searcher);
> }
>
> All thread safe so you don't have to worry about any complications
> there.  And I bet it'll be blindingly fast.
>
> Don't forget to close() things down at the end.
>
>
> --
> Ian.
>
>
>
> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > I was trying to, but don't know how to even I read some of your blogs.
> >
> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Are you using near-real-time readers?
> >>
> >> (IndexReader.open(IndexWriter))
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <[EMAIL PROTECTED]> wrote:
> >> > Hi Uwe,
> >> >
> >> > My challenge is that I need to update/modify the indexes frequently
> while
> >> > providing the search capability. I was trying to use FSDirectory, but
> >> found
> >> > out that the reading and writing from/to FSDirectory is unbearably
> slow.
> >> So
> >> > I now am trying the RAMDirectory, which is fast.
> >> >
> >> > I don't know of  MMapDirectory, and wonder if it is as fast as
> >> RAMDirectory.
> >> >
> >> >
> >> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> >> Hi Cheng,
> >> >>
> >> >> It seems that you use a RAMDirectory for *caching*, otherwise it
> makes
> >> no
> >> >> sense to write changes back. In recent Lucene versions, this is not a
> >> good
> >> >> idea, especially for large indexes (RAMDirectory eats your heap
> space,
> >> >> allocates millions of small byte[] arrays,...). If you need something
> >> like
> >> >> a
> >> >> caching Directory and you are working on a 64bit platform, you can
> use
> >> >> MMapDirectory (where the operating system kernel manages the
> read/write
> >> >> between disk an memory). MMapDirectory is returned by default for
> >> >> FSDirectory.open() on most 64 bit platforms. The good thing: the
> >> "caching"
> >> >> space is outside your JVM heap, so does not slowdown the garbage
> >> collector.
> >> >> So be sure to *not* allocate too much heap space (-Xmx) to your
> search
> >> app,
> >> >> only the minimum needed to execute it and leave the rest of your RAM
> >> >> available for the OS kernel to manage FS cache.
> >> >>
> >> >> Uwe
> >> >>
> >> >> -----
> >> >> Uwe Schindler
> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> >> http://www.thetaphi.de
> >> >> eMail: [EMAIL PROTECTED]
> >> >>
> >> >>
> >> >> > -----Original Message-----
> >> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> >> > Sent: Sunday, February 05, 2012 7:56 AM
> >> >> > To: [EMAIL PROTECTED]
> >> >> > Subject: Configure writer to write to FSDirectory?

You would use NRTManagerReopenThread as a standalone thread, not
plugged into your Executor stuff.  It is a utility class which you
don't have to use.  See the javadocs.

But in your case I'd use it, to start with anyway.  Fire it up with
suitable settings and forget about it, except to call close()
eventually. Once you've got things up and running you can tweak things
as much as you want but you appear to be having trouble getting up and
running.

So ... somewhere in the initialisation code of your app, create an
IndexWriter, NRTManager + ReopenThread and SearcherManager as outlined
before.  Then pass the NRTManager to any/all write methods or threads
and the SearcherManager instance to any/all search methods or threads
and you're done.  If you want to use threads that are part of your
ExecutorService, fine.  Just wrap it all together in whatever
combination of Thread or Runnable instances you want.
Does that help?
--
Ian.
> I don't understand this following portion:
>
> IndexWriter iw = new IndexWriter(whatever - some standard disk index);
> NRTManager nrtm = new NRTManager(iw, null);
> NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
> ropt.setXxx(...);
> ....
> ropt.start();
>
> I have a java ExecutorServices instance running which take care of my own
> applications. I don't know how this NRTManagerReopenThread works with my
> own ExecutorService instance.
>
> Can both work together? How can the NRTManagerReopenThread instance ropt be
> plugged into my own multithreading framework?
>
> On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>
>> If you can use NRTManager and SearcherManager things should be easy
>> and blazingly fast rather than unbearably slow.  The latter phrase is
>> not one often associated with lucene.
>>
>> IndexWriter iw = new IndexWriter(whatever - some standard disk index);
>> NRTManager nrtm = new NRTManager(iw, null);
>> NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
>> ropt.setXxx(...);
>> ...
>> ropt.start();
>>
>> SearcherManager srchm = nrtm.getSearcherManager(b);
>>
>> Then add docs to your index via nrtm.addDocument(d), update with
>> nrtm.updateDocument(...), and to search use
>>
>> IndexSearcher searcher = srchm.acquire();
>> try {
>>  search ...
>> } finally {
>>  srchm.release(searcher);
>> }
>>
>> All thread safe so you don't have to worry about any complications
>> there.  And I bet it'll be blindingly fast.
>>
>> Don't forget to close() things down at the end.
>>
>>
>> --
>> Ian.
>>
>>
>>
>> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> > I was trying to, but don't know how to even I read some of your blogs.
>> >
>> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> Are you using near-real-time readers?
>> >>
>> >> (IndexReader.open(IndexWriter))
>> >>
>> >> Mike McCandless
>> >>
>> >> http://blog.mikemccandless.com
>> >>
>> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> >> > Hi Uwe,
>> >> >
>> >> > My challenge is that I need to update/modify the indexes frequently
>> while
>> >> > providing the search capability. I was trying to use FSDirectory, but
>> >> found
>> >> > out that the reading and writing from/to FSDirectory is unbearably
>> slow.
>> >> So
>> >> > I now am trying the RAMDirectory, which is fast.
>> >> >
>> >> > I don't know of  MMapDirectory, and wonder if it is as fast as
>> >> RAMDirectory.
>> >> >
>> >> >
>> >> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <[EMAIL PROTECTED]>
>> wrote:
>> >> >
>> >> >> Hi Cheng,
>> >> >>
>> >> >> It seems that you use a RAMDirectory for *caching*, otherwise it
>> makes
>> >> no
>> >> >> sense to write changes back. In recent Lucene versions, this is not a
>> >> good
>> >> >> idea, especially for large indexes (RAMDirectory eats your heap
>> space,
>> >> >> allocates millions of small byte[] arrays,...). If you need something
>> >> like
>> >> >> a
>> >> >> caching Directory and you are working on a 64bit platform, you can

That really helps! I will try it out.

Thanks.

On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:

> You would use NRTManagerReopenThread as a standalone thread, not
> plugged into your Executor stuff.  It is a utility class which you
> don't have to use.  See the javadocs.
>
> But in your case I'd use it, to start with anyway.  Fire it up with
> suitable settings and forget about it, except to call close()
> eventually. Once you've got things up and running you can tweak things
> as much as you want but you appear to be having trouble getting up and
> running.
>
> So ... somewhere in the initialisation code of your app, create an
> IndexWriter, NRTManager + ReopenThread and SearcherManager as outlined
> before.  Then pass the NRTManager to any/all write methods or threads
> and the SearcherManager instance to any/all search methods or threads
> and you're done.  If you want to use threads that are part of your
> ExecutorService, fine.  Just wrap it all together in whatever
> combination of Thread or Runnable instances you want.
>
>
> Does that help?
>
>
> --
> Ian.
>
>
> > I don't understand this following portion:
> >
> > IndexWriter iw = new IndexWriter(whatever - some standard disk index);
> > NRTManager nrtm = new NRTManager(iw, null);
> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
> > ropt.setXxx(...);
> > ....
> > ropt.start();
> >
> > I have a java ExecutorServices instance running which take care of my own
> > applications. I don't know how this NRTManagerReopenThread works with my
> > own ExecutorService instance.
> >
> > Can both work together? How can the NRTManagerReopenThread instance ropt
> be
> > plugged into my own multithreading framework?
> >
> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> >
> >> If you can use NRTManager and SearcherManager things should be easy
> >> and blazingly fast rather than unbearably slow.  The latter phrase is
> >> not one often associated with lucene.
> >>
> >> IndexWriter iw = new IndexWriter(whatever - some standard disk index);
> >> NRTManager nrtm = new NRTManager(iw, null);
> >> NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
> >> ropt.setXxx(...);
> >> ...
> >> ropt.start();
> >>
> >> SearcherManager srchm = nrtm.getSearcherManager(b);
> >>
> >> Then add docs to your index via nrtm.addDocument(d), update with
> >> nrtm.updateDocument(...), and to search use
> >>
> >> IndexSearcher searcher = srchm.acquire();
> >> try {
> >>  search ...
> >> } finally {
> >>  srchm.release(searcher);
> >> }
> >>
> >> All thread safe so you don't have to worry about any complications
> >> there.  And I bet it'll be blindingly fast.
> >>
> >> Don't forget to close() things down at the end.
> >>
> >>
> >> --
> >> Ian.
> >>
> >>
> >>
> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <[EMAIL PROTECTED]> wrote:
> >> > I was trying to, but don't know how to even I read some of your blogs.
> >> >
> >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> Are you using near-real-time readers?
> >> >>
> >> >> (IndexReader.open(IndexWriter))
> >> >>
> >> >> Mike McCandless
> >> >>
> >> >> http://blog.mikemccandless.com
> >> >>
> >> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <[EMAIL PROTECTED]>
> wrote:
> >> >> > Hi Uwe,
> >> >> >
> >> >> > My challenge is that I need to update/modify the indexes frequently
> >> while
> >> >> > providing the search capability. I was trying to use FSDirectory,
> but
> >> >> found
> >> >> > out that the reading and writing from/to FSDirectory is unbearably
> >> slow.
> >> >> So
> >> >> > I now am trying the RAMDirectory, which is fast.
> >> >> >
> >> >> > I don't know of  MMapDirectory, and wonder if it is as fast as
> >> >> RAMDirectory.
> >> >> >
> >> >> >
> >> >> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <[EMAIL PROTECTED]>
> >> wrote:
> >> >> >
> >> >> >> Hi Cheng,
> >> >> >>
> >> >> >> It seems that you use a RAMDirectory for *caching*, otherwise it

Ian,

I encountered an issue that I need to frequently update the index. The
NRTManager seems not very helpful on this front as the speed is slower than
RAMDirectory is used.

Any improvement advice?

On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]> wrote:

> That really helps! I will try it out.
>
> Thanks.
>
>
> On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>
>> You would use NRTManagerReopenThread as a standalone thread, not
>> plugged into your Executor stuff.  It is a utility class which you
>> don't have to use.  See the javadocs.
>>
>> But in your case I'd use it, to start with anyway.  Fire it up with
>> suitable settings and forget about it, except to call close()
>> eventually. Once you've got things up and running you can tweak things
>> as much as you want but you appear to be having trouble getting up and
>> running.
>>
>> So ... somewhere in the initialisation code of your app, create an
>> IndexWriter, NRTManager + ReopenThread and SearcherManager as outlined
>> before.  Then pass the NRTManager to any/all write methods or threads
>> and the SearcherManager instance to any/all search methods or threads
>> and you're done.  If you want to use threads that are part of your
>> ExecutorService, fine.  Just wrap it all together in whatever
>> combination of Thread or Runnable instances you want.
>>
>>
>> Does that help?
>>
>>
>> --
>> Ian.
>>
>>
>> > I don't understand this following portion:
>> >
>> > IndexWriter iw = new IndexWriter(whatever - some standard disk index);
>> > NRTManager nrtm = new NRTManager(iw, null);
>> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
>> > ropt.setXxx(...);
>> > ....
>> > ropt.start();
>> >
>> > I have a java ExecutorServices instance running which take care of my
>> own
>> > applications. I don't know how this NRTManagerReopenThread works with my
>> > own ExecutorService instance.
>> >
>> > Can both work together? How can the NRTManagerReopenThread instance
>> ropt be
>> > plugged into my own multithreading framework?
>> >
>> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>> >
>> >> If you can use NRTManager and SearcherManager things should be easy
>> >> and blazingly fast rather than unbearably slow.  The latter phrase is
>> >> not one often associated with lucene.
>> >>
>> >> IndexWriter iw = new IndexWriter(whatever - some standard disk index);
>> >> NRTManager nrtm = new NRTManager(iw, null);
>> >> NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
>> >> ropt.setXxx(...);
>> >> ...
>> >> ropt.start();
>> >>
>> >> SearcherManager srchm = nrtm.getSearcherManager(b);
>> >>
>> >> Then add docs to your index via nrtm.addDocument(d), update with
>> >> nrtm.updateDocument(...), and to search use
>> >>
>> >> IndexSearcher searcher = srchm.acquire();
>> >> try {
>> >>  search ...
>> >> } finally {
>> >>  srchm.release(searcher);
>> >> }
>> >>
>> >> All thread safe so you don't have to worry about any complications
>> >> there.  And I bet it'll be blindingly fast.
>> >>
>> >> Don't forget to close() things down at the end.
>> >>
>> >>
>> >> --
>> >> Ian.
>> >>
>> >>
>> >>
>> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <[EMAIL PROTECTED]>
>> wrote:
>> >> > I was trying to, but don't know how to even I read some of your
>> blogs.
>> >> >
>> >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
>> >> > [EMAIL PROTECTED]> wrote:
>> >> >
>> >> >> Are you using near-real-time readers?
>> >> >>
>> >> >> (IndexReader.open(IndexWriter))
>> >> >>
>> >> >> Mike McCandless
>> >> >>
>> >> >> http://blog.mikemccandless.com
>> >> >>
>> >> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <[EMAIL PROTECTED]>
>> wrote:
>> >> >> > Hi Uwe,
>> >> >> >
>> >> >> > My challenge is that I need to update/modify the indexes
>> frequently
>> >> while
>> >> >> > providing the search capability. I was trying to use FSDirectory,
>> but
>> >> >> found
>> >> >> > out that the reading and writing from/to FSDirectory is unbearably

Please review the following articles about NRT, absolutely instant updates
that are visible as they are done are almost impossible (even with
RAMDirectory):

http://goo.gl/mzAHt
http://goo.gl/5RoPx
http://goo.gl/vSJ7x

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]

> -----Original Message-----
> From: Cheng [mailto:[EMAIL PROTECTED]]
> Sent: Monday, February 06, 2012 4:27 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Configure writer to write to FSDirectory?
>
> Ian,
>
> I encountered an issue that I need to frequently update the index. The
> NRTManager seems not very helpful on this front as the speed is slower
than
> RAMDirectory is used.
>
> Any improvement advice?
>
>
>
> On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]> wrote:
>
> > That really helps! I will try it out.
> >
> > Thanks.
> >
> >
> > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> >
> >> You would use NRTManagerReopenThread as a standalone thread, not
> >> plugged into your Executor stuff.  It is a utility class which you
> >> don't have to use.  See the javadocs.
> >>
> >> But in your case I'd use it, to start with anyway.  Fire it up with
> >> suitable settings and forget about it, except to call close()
> >> eventually. Once you've got things up and running you can tweak
> >> things as much as you want but you appear to be having trouble
> >> getting up and running.
> >>
> >> So ... somewhere in the initialisation code of your app, create an
> >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
> >> outlined before.  Then pass the NRTManager to any/all write methods
> >> or threads and the SearcherManager instance to any/all search methods
> >> or threads and you're done.  If you want to use threads that are part
> >> of your ExecutorService, fine.  Just wrap it all together in whatever
> >> combination of Thread or Runnable instances you want.
> >>
> >>
> >> Does that help?
> >>
> >>
> >> --
> >> Ian.
> >>
> >>
> >> > I don't understand this following portion:
> >> >
> >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
> >> > index); NRTManager nrtm = new NRTManager(iw, null);
> >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm,
> >> > ...); ropt.setXxx(...); ....
> >> > ropt.start();
> >> >
> >> > I have a java ExecutorServices instance running which take care of
> >> > my
> >> own
> >> > applications. I don't know how this NRTManagerReopenThread works
> >> > with my own ExecutorService instance.
> >> >
> >> > Can both work together? How can the NRTManagerReopenThread
> instance
> >> ropt be
> >> > plugged into my own multithreading framework?
> >> >
> >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> >> >
> >> >> If you can use NRTManager and SearcherManager things should be
> >> >> easy and blazingly fast rather than unbearably slow.  The latter
> >> >> phrase is not one often associated with lucene.
> >> >>
> >> >> IndexWriter iw = new IndexWriter(whatever - some standard disk
> >> >> index); NRTManager nrtm = new NRTManager(iw, null);
> >> >> NRTManagerReopenThread ropt = new
> NRTManagerReopenThread(nrtm,
> >> >> ...); ropt.setXxx(...); ...
> >> >> ropt.start();
> >> >>
> >> >> SearcherManager srchm = nrtm.getSearcherManager(b);
> >> >>
> >> >> Then add docs to your index via nrtm.addDocument(d), update with
> >> >> nrtm.updateDocument(...), and to search use
> >> >>
> >> >> IndexSearcher searcher = srchm.acquire(); try {  search ...
> >> >> } finally {
> >> >>  srchm.release(searcher);
> >> >> }
> >> >>
> >> >> All thread safe so you don't have to worry about any complications
> >> >> there.  And I bet it'll be blindingly fast.
> >> >>
> >> >> Don't forget to close() things down at the end.
> >> >>
> >> >>
> >> >> --
> >> >> Ian.
> >> >>
> >> >>
> >> >>
> >> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <[EMAIL PROTECTED]>
> >> wrote:
> >> >> > I was trying to, but don't know how to even I read some of your
[EMAIL PROTECTED]

What exactly do you mean by the "speed is slower"?  Time taken to
update the index?  Time taken for updates to become visible in search
results?  Time taken for searches to run on the IndexSearcher returned
from SearcherManager?  Something else?
--
Ian.
On Mon, Feb 6, 2012 at 3:27 PM, Cheng <[EMAIL PROTECTED]> wrote:
> Ian,
>
> I encountered an issue that I need to frequently update the index. The
> NRTManager seems not very helpful on this front as the speed is slower than
> RAMDirectory is used.
>
> Any improvement advice?
>
>
>
> On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]> wrote:
>
>> That really helps! I will try it out.
>>
>> Thanks.
>>
>>
>> On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>>
>>> You would use NRTManagerReopenThread as a standalone thread, not
>>> plugged into your Executor stuff.  It is a utility class which you
>>> don't have to use.  See the javadocs.
>>>
>>> But in your case I'd use it, to start with anyway.  Fire it up with
>>> suitable settings and forget about it, except to call close()
>>> eventually. Once you've got things up and running you can tweak things
>>> as much as you want but you appear to be having trouble getting up and
>>> running.
>>>
>>> So ... somewhere in the initialisation code of your app, create an
>>> IndexWriter, NRTManager + ReopenThread and SearcherManager as outlined
>>> before.  Then pass the NRTManager to any/all write methods or threads
>>> and the SearcherManager instance to any/all search methods or threads
>>> and you're done.  If you want to use threads that are part of your
>>> ExecutorService, fine.  Just wrap it all together in whatever
>>> combination of Thread or Runnable instances you want.
>>>
>>>
>>> Does that help?
>>>
>>>
>>> --
>>> Ian.
>>>
>>>
>>> > I don't understand this following portion:
>>> >
>>> > IndexWriter iw = new IndexWriter(whatever - some standard disk index);
>>> > NRTManager nrtm = new NRTManager(iw, null);
>>> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
>>> > ropt.setXxx(...);
>>> > ....
>>> > ropt.start();
>>> >
>>> > I have a java ExecutorServices instance running which take care of my
>>> own
>>> > applications. I don't know how this NRTManagerReopenThread works with my
>>> > own ExecutorService instance.
>>> >
>>> > Can both work together? How can the NRTManagerReopenThread instance
>>> ropt be
>>> > plugged into my own multithreading framework?
>>> >
>>> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>>> >
>>> >> If you can use NRTManager and SearcherManager things should be easy
>>> >> and blazingly fast rather than unbearably slow.  The latter phrase is
>>> >> not one often associated with lucene.
>>> >>
>>> >> IndexWriter iw = new IndexWriter(whatever - some standard disk index);
>>> >> NRTManager nrtm = new NRTManager(iw, null);
>>> >> NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
>>> >> ropt.setXxx(...);
>>> >> ...
>>> >> ropt.start();
>>> >>
>>> >> SearcherManager srchm = nrtm.getSearcherManager(b);
>>> >>
>>> >> Then add docs to your index via nrtm.addDocument(d), update with
>>> >> nrtm.updateDocument(...), and to search use
>>> >>
>>> >> IndexSearcher searcher = srchm.acquire();
>>> >> try {
>>> >>  search ...
>>> >> } finally {
>>> >>  srchm.release(searcher);
>>> >> }
>>> >>
>>> >> All thread safe so you don't have to worry about any complications
>>> >> there.  And I bet it'll be blindingly fast.
>>> >>
>>> >> Don't forget to close() things down at the end.
>>> >>
>>> >>
>>> >> --
>>> >> Ian.
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <[EMAIL PROTECTED]>
>>> wrote:
>>> >> > I was trying to, but don't know how to even I read some of your
>>> blogs.
>>> >> >
>>> >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
>>> >> > [EMAIL PROTECTED]> wrote:
>>> >> >
>>> >> >> Are you using near-real-time readers?
>>> >> >>
>>> >> >> (IndexReader.open(IndexWriter))
>>> >> >>

Uwe, when I meant speed is slow, I didn't refer to instant visibility of
changes, but that the changes may be synchronized with FSDirectory when I
use writer.commit().

When I use RAMDirectory, the writer.commit() seems much faster than using
NRTManager built upon FSDirectory. So, I am guessing the difference is the
index synchronization.

On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:

> Please review the following articles about NRT, absolutely instant updates
> that are visible as they are done are almost impossible (even with
> RAMDirectory):
>
> http://goo.gl/mzAHt
> http://goo.gl/5RoPx
> http://goo.gl/vSJ7x
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [EMAIL PROTECTED]
>
> > -----Original Message-----
> > From: Cheng [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, February 06, 2012 4:27 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Configure writer to write to FSDirectory?
> >
> > Ian,
> >
> > I encountered an issue that I need to frequently update the index. The
> > NRTManager seems not very helpful on this front as the speed is slower
> than
> > RAMDirectory is used.
> >
> > Any improvement advice?
> >
> >
> >
> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]> wrote:
> >
> > > That really helps! I will try it out.
> > >
> > > Thanks.
> > >
> > >
> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> > >
> > >> You would use NRTManagerReopenThread as a standalone thread, not
> > >> plugged into your Executor stuff.  It is a utility class which you
> > >> don't have to use.  See the javadocs.
> > >>
> > >> But in your case I'd use it, to start with anyway.  Fire it up with
> > >> suitable settings and forget about it, except to call close()
> > >> eventually. Once you've got things up and running you can tweak
> > >> things as much as you want but you appear to be having trouble
> > >> getting up and running.
> > >>
> > >> So ... somewhere in the initialisation code of your app, create an
> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
> > >> outlined before.  Then pass the NRTManager to any/all write methods
> > >> or threads and the SearcherManager instance to any/all search methods
> > >> or threads and you're done.  If you want to use threads that are part
> > >> of your ExecutorService, fine.  Just wrap it all together in whatever
> > >> combination of Thread or Runnable instances you want.
> > >>
> > >>
> > >> Does that help?
> > >>
> > >>
> > >> --
> > >> Ian.
> > >>
> > >>
> > >> > I don't understand this following portion:
> > >> >
> > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
> > >> > index); NRTManager nrtm = new NRTManager(iw, null);
> > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm,
> > >> > ...); ropt.setXxx(...); ....
> > >> > ropt.start();
> > >> >
> > >> > I have a java ExecutorServices instance running which take care of
> > >> > my
> > >> own
> > >> > applications. I don't know how this NRTManagerReopenThread works
> > >> > with my own ExecutorService instance.
> > >> >
> > >> > Can both work together? How can the NRTManagerReopenThread
> > instance
> > >> ropt be
> > >> > plugged into my own multithreading framework?
> > >> >
> > >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> > >> >
> > >> >> If you can use NRTManager and SearcherManager things should be
> > >> >> easy and blazingly fast rather than unbearably slow.  The latter
> > >> >> phrase is not one often associated with lucene.
> > >> >>
> > >> >> IndexWriter iw = new IndexWriter(whatever - some standard disk
> > >> >> index); NRTManager nrtm = new NRTManager(iw, null);
> > >> >> NRTManagerReopenThread ropt = new
> > NRTManagerReopenThread(nrtm,
> > >> >> ...); ropt.setXxx(...); ...
> > >> >> ropt.start();
> > >> >>
> > >> >> SearcherManager srchm = nrtm.getSearcherManager(b);
> > >> >>
> > >> >> Then add docs to your index via nrtm.addDocument(d), update with

I meant that when I use NRTManager and use commit(), the speed is slower
than when I use RAMDirectory.

In my case, NRTManager instance not only perform search but update/modify
indexes which should be visible to other threads. In RAMDirectory, the
commit() doesn't synchronize indexes with the FSDirectory. The slower speed
of using NRTManager built upon FSDirectory may be caused by the frequent
updates or modification of indexes.

That is my guess.

On Mon, Feb 6, 2012 at 11:41 PM, Ian Lea <[EMAIL PROTECTED]> wrote:

> What exactly do you mean by the "speed is slower"?  Time taken to
> update the index?  Time taken for updates to become visible in search
> results?  Time taken for searches to run on the IndexSearcher returned
> from SearcherManager?  Something else?
>
>
> --
> Ian.
>
>
> On Mon, Feb 6, 2012 at 3:27 PM, Cheng <[EMAIL PROTECTED]> wrote:
> > Ian,
> >
> > I encountered an issue that I need to frequently update the index. The
> > NRTManager seems not very helpful on this front as the speed is slower
> than
> > RAMDirectory is used.
> >
> > Any improvement advice?
> >
> >
> >
> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]> wrote:
> >
> >> That really helps! I will try it out.
> >>
> >> Thanks.
> >>
> >>
> >> On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> >>
> >>> You would use NRTManagerReopenThread as a standalone thread, not
> >>> plugged into your Executor stuff.  It is a utility class which you
> >>> don't have to use.  See the javadocs.
> >>>
> >>> But in your case I'd use it, to start with anyway.  Fire it up with
> >>> suitable settings and forget about it, except to call close()
> >>> eventually. Once you've got things up and running you can tweak things
> >>> as much as you want but you appear to be having trouble getting up and
> >>> running.
> >>>
> >>> So ... somewhere in the initialisation code of your app, create an
> >>> IndexWriter, NRTManager + ReopenThread and SearcherManager as outlined
> >>> before.  Then pass the NRTManager to any/all write methods or threads
> >>> and the SearcherManager instance to any/all search methods or threads
> >>> and you're done.  If you want to use threads that are part of your
> >>> ExecutorService, fine.  Just wrap it all together in whatever
> >>> combination of Thread or Runnable instances you want.
> >>>
> >>>
> >>> Does that help?
> >>>
> >>>
> >>> --
> >>> Ian.
> >>>
> >>>
> >>> > I don't understand this following portion:
> >>> >
> >>> > IndexWriter iw = new IndexWriter(whatever - some standard disk
> index);
> >>> > NRTManager nrtm = new NRTManager(iw, null);
> >>> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
> >>> > ropt.setXxx(...);
> >>> > ....
> >>> > ropt.start();
> >>> >
> >>> > I have a java ExecutorServices instance running which take care of my
> >>> own
> >>> > applications. I don't know how this NRTManagerReopenThread works
> with my
> >>> > own ExecutorService instance.
> >>> >
> >>> > Can both work together? How can the NRTManagerReopenThread instance
> >>> ropt be
> >>> > plugged into my own multithreading framework?
> >>> >
> >>> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> >>> >
> >>> >> If you can use NRTManager and SearcherManager things should be easy
> >>> >> and blazingly fast rather than unbearably slow.  The latter phrase
> is
> >>> >> not one often associated with lucene.
> >>> >>
> >>> >> IndexWriter iw = new IndexWriter(whatever - some standard disk
> index);
> >>> >> NRTManager nrtm = new NRTManager(iw, null);
> >>> >> NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
> >>> >> ropt.setXxx(...);
> >>> >> ...
> >>> >> ropt.start();
> >>> >>
> >>> >> SearcherManager srchm = nrtm.getSearcherManager(b);
> >>> >>
> >>> >> Then add docs to your index via nrtm.addDocument(d), update with
> >>> >> nrtm.updateDocument(...), and to search use
> >>> >>
> >>> >> IndexSearcher searcher = srchm.acquire();
> >>> >> try {
> >>> >>  search ...

Well, yes.  What would you expect?  From the javadocs for IndexWriter.commit()

Commits all pending changes (added & deleted documents, segment
merges, added indexes, etc.) to the index, and syncs all referenced
index files ... This may be a costly operation, so you should test the
cost in your application and do it only when really necessary.

If you are using NRTManager why do you care how long this takes?  How
often are you calling it?  Why?
--
Ian.
On Mon, Feb 6, 2012 at 3:45 PM, Cheng <[EMAIL PROTECTED]> wrote:
> Uwe, when I meant speed is slow, I didn't refer to instant visibility of
> changes, but that the changes may be synchronized with FSDirectory when I
> use writer.commit().
>
> When I use RAMDirectory, the writer.commit() seems much faster than using
> NRTManager built upon FSDirectory. So, I am guessing the difference is the
> index synchronization.
>
>
>
> On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
>> Please review the following articles about NRT, absolutely instant updates
>> that are visible as they are done are almost impossible (even with
>> RAMDirectory):
>>
>> http://goo.gl/mzAHt
>> http://goo.gl/5RoPx
>> http://goo.gl/vSJ7x
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: [EMAIL PROTECTED]
>>
>> > -----Original Message-----
>> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> > Sent: Monday, February 06, 2012 4:27 PM
>> > To: [EMAIL PROTECTED]
>> > Subject: Re: Configure writer to write to FSDirectory?
>> >
>> > Ian,
>> >
>> > I encountered an issue that I need to frequently update the index. The
>> > NRTManager seems not very helpful on this front as the speed is slower
>> than
>> > RAMDirectory is used.
>> >
>> > Any improvement advice?
>> >
>> >
>> >
>> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]> wrote:
>> >
>> > > That really helps! I will try it out.
>> > >
>> > > Thanks.
>> > >
>> > >
>> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>> > >
>> > >> You would use NRTManagerReopenThread as a standalone thread, not
>> > >> plugged into your Executor stuff.  It is a utility class which you
>> > >> don't have to use.  See the javadocs.
>> > >>
>> > >> But in your case I'd use it, to start with anyway.  Fire it up with
>> > >> suitable settings and forget about it, except to call close()
>> > >> eventually. Once you've got things up and running you can tweak
>> > >> things as much as you want but you appear to be having trouble
>> > >> getting up and running.
>> > >>
>> > >> So ... somewhere in the initialisation code of your app, create an
>> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
>> > >> outlined before.  Then pass the NRTManager to any/all write methods
>> > >> or threads and the SearcherManager instance to any/all search methods
>> > >> or threads and you're done.  If you want to use threads that are part
>> > >> of your ExecutorService, fine.  Just wrap it all together in whatever
>> > >> combination of Thread or Runnable instances you want.
>> > >>
>> > >>
>> > >> Does that help?
>> > >>
>> > >>
>> > >> --
>> > >> Ian.
>> > >>
>> > >>
>> > >> > I don't understand this following portion:
>> > >> >
>> > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
>> > >> > index); NRTManager nrtm = new NRTManager(iw, null);
>> > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm,
>> > >> > ...); ropt.setXxx(...); ....
>> > >> > ropt.start();
>> > >> >
>> > >> > I have a java ExecutorServices instance running which take care of
>> > >> > my
>> > >> own
>> > >> > applications. I don't know how this NRTManagerReopenThread works
>> > >> > with my own ExecutorService instance.
>> > >> >
>> > >> > Can both work together? How can the NRTManagerReopenThread
>> > instance
>> > >> ropt be
>> > >> > plugged into my own multithreading framework?
>> > >> >
>> > >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:

Hi Cheng,

all pros and cons are explained in those articles written by Mike! As soon
as there are harddisks in the game, there is a slowdown, what do you expect?
If you need it faster, buy SSDs! :-)

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Cheng [mailto:[EMAIL PROTECTED]]
> Sent: Monday, February 06, 2012 4:45 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Configure writer to write to FSDirectory?
>
> Uwe, when I meant speed is slow, I didn't refer to instant visibility of
changes,
> but that the changes may be synchronized with FSDirectory when I use
> writer.commit().
>
> When I use RAMDirectory, the writer.commit() seems much faster than using
> NRTManager built upon FSDirectory. So, I am guessing the difference is the
> index synchronization.
>
>
>
> On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
> > Please review the following articles about NRT, absolutely instant
> > updates that are visible as they are done are almost impossible (even
> > with
> > RAMDirectory):
> >
> > http://goo.gl/mzAHt
> > http://goo.gl/5RoPx
> > http://goo.gl/vSJ7x
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: [EMAIL PROTECTED]
> >
> > > -----Original Message-----
> > > From: Cheng [mailto:[EMAIL PROTECTED]]
> > > Sent: Monday, February 06, 2012 4:27 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: Configure writer to write to FSDirectory?
> > >
> > > Ian,
> > >
> > > I encountered an issue that I need to frequently update the index.
> > > The NRTManager seems not very helpful on this front as the speed is
> > > slower
> > than
> > > RAMDirectory is used.
> > >
> > > Any improvement advice?
> > >
> > >
> > >
> > > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]>
> wrote:
> > >
> > > > That really helps! I will try it out.
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> > > >
> > > >> You would use NRTManagerReopenThread as a standalone thread, not
> > > >> plugged into your Executor stuff.  It is a utility class which
> > > >> you don't have to use.  See the javadocs.
> > > >>
> > > >> But in your case I'd use it, to start with anyway.  Fire it up
> > > >> with suitable settings and forget about it, except to call
> > > >> close() eventually. Once you've got things up and running you can
> > > >> tweak things as much as you want but you appear to be having
> > > >> trouble getting up and running.
> > > >>
> > > >> So ... somewhere in the initialisation code of your app, create
> > > >> an IndexWriter, NRTManager + ReopenThread and SearcherManager as
> > > >> outlined before.  Then pass the NRTManager to any/all write
> > > >> methods or threads and the SearcherManager instance to any/all
> > > >> search methods or threads and you're done.  If you want to use
> > > >> threads that are part of your ExecutorService, fine.  Just wrap
> > > >> it all together in whatever combination of Thread or Runnable
instances
> you want.
> > > >>
> > > >>
> > > >> Does that help?
> > > >>
> > > >>
> > > >> --
> > > >> Ian.
> > > >>
> > > >>
> > > >> > I don't understand this following portion:
> > > >> >
> > > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
> > > >> > index); NRTManager nrtm = new NRTManager(iw, null);
> > > >> > NRTManagerReopenThread ropt = new
> NRTManagerReopenThread(nrtm,
> > > >> > ...); ropt.setXxx(...); ....
> > > >> > ropt.start();
> > > >> >
> > > >> > I have a java ExecutorServices instance running which take care
> > > >> > of my
> > > >> own
> > > >> > applications. I don't know how this NRTManagerReopenThread
> > > >> > works with my own ExecutorService instance.
> > > >> >
> > > >> > Can both work together? How can the NRTManagerReopenThread
> > > instance
> > > >> ropt be
> > > >wrote:
thing:
drive.

My original question is if there exists a way to configure writer when to
writer to FSDirectory. I think there may be something in
the IndexWriterConfig that can helps.

On Mon, Feb 6, 2012 at 11:50 PM, Ian Lea <[EMAIL PROTECTED]> wrote:

> Well, yes.  What would you expect?  From the javadocs for
> IndexWriter.commit()
>
> Commits all pending changes (added & deleted documents, segment
> merges, added indexes, etc.) to the index, and syncs all referenced
> index files ... This may be a costly operation, so you should test the
> cost in your application and do it only when really necessary.
>
> If you are using NRTManager why do you care how long this takes?  How
> often are you calling it?  Why?
>
>
> --
> Ian.
>
>
> On Mon, Feb 6, 2012 at 3:45 PM, Cheng <[EMAIL PROTECTED]> wrote:
> > Uwe, when I meant speed is slow, I didn't refer to instant visibility of
> > changes, but that the changes may be synchronized with FSDirectory when I
> > use writer.commit().
> >
> > When I use RAMDirectory, the writer.commit() seems much faster than using
> > NRTManager built upon FSDirectory. So, I am guessing the difference is
> the
> > index synchronization.
> >
> >
> >
> > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> >> Please review the following articles about NRT, absolutely instant
> updates
> >> that are visible as they are done are almost impossible (even with
> >> RAMDirectory):
> >>
> >> http://goo.gl/mzAHt
> >> http://goo.gl/5RoPx
> >> http://goo.gl/vSJ7x
> >>
> >> Uwe
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: [EMAIL PROTECTED]
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> > Sent: Monday, February 06, 2012 4:27 PM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: Configure writer to write to FSDirectory?
> >> >
> >> > Ian,
> >> >
> >> > I encountered an issue that I need to frequently update the index. The
> >> > NRTManager seems not very helpful on this front as the speed is slower
> >> than
> >> > RAMDirectory is used.
> >> >
> >> > Any improvement advice?
> >> >
> >> >
> >> >
> >> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > That really helps! I will try it out.
> >> > >
> >> > > Thanks.
> >> > >
> >> > >
> >> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> >> > >
> >> > >> You would use NRTManagerReopenThread as a standalone thread, not
> >> > >> plugged into your Executor stuff.  It is a utility class which you
> >> > >> don't have to use.  See the javadocs.
> >> > >>
> >> > >> But in your case I'd use it, to start with anyway.  Fire it up with
> >> > >> suitable settings and forget about it, except to call close()
> >> > >> eventually. Once you've got things up and running you can tweak
> >> > >> things as much as you want but you appear to be having trouble
> >> > >> getting up and running.
> >> > >>
> >> > >> So ... somewhere in the initialisation code of your app, create an
> >> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
> >> > >> outlined before.  Then pass the NRTManager to any/all write methods
> >> > >> or threads and the SearcherManager instance to any/all search
> methods
> >> > >> or threads and you're done.  If you want to use threads that are
> part
> >> > >> of your ExecutorService, fine.  Just wrap it all together in
> whatever
> >> > >> combination of Thread or Runnable instances you want.
> >> > >>
> >> > >>
> >> > >> Does that help?
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Ian.
> >> > >>
> >> > >>
> >> > >> > I don't understand this following portion:
> >> > >> >
> >> > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
> >> > >> > index); NRTManager nrtm = new NRTManager(iw, null);
> >> > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm,
> >> > >> > ...); ropt.setXxx(...); ....
> >> > >> > ropt.start();

Agree.

On Mon, Feb 6, 2012 at 11:53 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:

> Hi Cheng,
>
> all pros and cons are explained in those articles written by Mike! As soon
> as there are harddisks in the game, there is a slowdown, what do you
> expect?
> If you need it faster, buy SSDs! :-)
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [EMAIL PROTECTED]
>
>
> > -----Original Message-----
> > From: Cheng [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, February 06, 2012 4:45 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Configure writer to write to FSDirectory?
> >
> > Uwe, when I meant speed is slow, I didn't refer to instant visibility of
> changes,
> > but that the changes may be synchronized with FSDirectory when I use
> > writer.commit().
> >
> > When I use RAMDirectory, the writer.commit() seems much faster than using
> > NRTManager built upon FSDirectory. So, I am guessing the difference is
> the
> > index synchronization.
> >
> >
> >
> > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> > > Please review the following articles about NRT, absolutely instant
> > > updates that are visible as they are done are almost impossible (even
> > > with
> > > RAMDirectory):
> > >
> > > http://goo.gl/mzAHt
> > > http://goo.gl/5RoPx
> > > http://goo.gl/vSJ7x
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, D-28213 Bremen
> > > http://www.thetaphi.de
> > > eMail: [EMAIL PROTECTED]
> > >
> > > > -----Original Message-----
> > > > From: Cheng [mailto:[EMAIL PROTECTED]]
> > > > Sent: Monday, February 06, 2012 4:27 PM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: Re: Configure writer to write to FSDirectory?
> > > >
> > > > Ian,
> > > >
> > > > I encountered an issue that I need to frequently update the index.
> > > > The NRTManager seems not very helpful on this front as the speed is
> > > > slower
> > > than
> > > > RAMDirectory is used.
> > > >
> > > > Any improvement advice?
> > > >
> > > >
> > > >
> > > > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]>
> > wrote:
> > > >
> > > > > That really helps! I will try it out.
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]>
> wrote:
> > > > >
> > > > >> You would use NRTManagerReopenThread as a standalone thread, not
> > > > >> plugged into your Executor stuff.  It is a utility class which
> > > > >> you don't have to use.  See the javadocs.
> > > > >>
> > > > >> But in your case I'd use it, to start with anyway.  Fire it up
> > > > >> with suitable settings and forget about it, except to call
> > > > >> close() eventually. Once you've got things up and running you can
> > > > >> tweak things as much as you want but you appear to be having
> > > > >> trouble getting up and running.
> > > > >>
> > > > >> So ... somewhere in the initialisation code of your app, create
> > > > >> an IndexWriter, NRTManager + ReopenThread and SearcherManager as
> > > > >> outlined before.  Then pass the NRTManager to any/all write
> > > > >> methods or threads and the SearcherManager instance to any/all
> > > > >> search methods or threads and you're done.  If you want to use
> > > > >> threads that are part of your ExecutorService, fine.  Just wrap
> > > > >> it all together in whatever combination of Thread or Runnable
> instances
> > you want.
> > > > >>
> > > > >>
> > > > >> Does that help?
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Ian.
> > > > >>
> > > > >>
> > > > >> > I don't understand this following portion:
> > > > >> >
> > > > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
> > > > >> > index); NRTManager nrtm = new NRTManager(iw, null);
> > > > >> > NRTManagerReopenThread ropt = new
> > NRTManagerReopenThread(nrtm,
> > > > >> > ...); ropt.setXxx(...); ....
> > > > >> > ropt.start();
> > > > >> >
> > > > >> > I have a java ExecutorServices instance running which take care

You shouldn't call IW.commit when using NRT; that's the point of NRT
(making changes visible w/o calling commit).

Only call commit when you require that all changes be durable (surive
OS / JVM crash, power loss, etc.) on disk.

Also, you can use NRTCachingDirectory which acts like RAMDirectory for
small flushed segments.

Mike McCandless

http://blog.mikemccandless.com

On Mon, Feb 6, 2012 at 10:45 AM, Cheng <[EMAIL PROTECTED]> wrote:
> Uwe, when I meant speed is slow, I didn't refer to instant visibility of
> changes, but that the changes may be synchronized with FSDirectory when I
> use writer.commit().
>
> When I use RAMDirectory, the writer.commit() seems much faster than using
> NRTManager built upon FSDirectory. So, I am guessing the difference is the
> index synchronization.
>
>
>
> On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
>> Please review the following articles about NRT, absolutely instant updates
>> that are visible as they are done are almost impossible (even with
>> RAMDirectory):
>>
>> http://goo.gl/mzAHt
>> http://goo.gl/5RoPx
>> http://goo.gl/vSJ7x
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: [EMAIL PROTECTED]
>>
>> > -----Original Message-----
>> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> > Sent: Monday, February 06, 2012 4:27 PM
>> > To: [EMAIL PROTECTED]
>> > Subject: Re: Configure writer to write to FSDirectory?
>> >
>> > Ian,
>> >
>> > I encountered an issue that I need to frequently update the index. The
>> > NRTManager seems not very helpful on this front as the speed is slower
>> than
>> > RAMDirectory is used.
>> >
>> > Any improvement advice?
>> >
>> >
>> >
>> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]> wrote:
>> >
>> > > That really helps! I will try it out.
>> > >
>> > > Thanks.
>> > >
>> > >
>> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>> > >
>> > >> You would use NRTManagerReopenThread as a standalone thread, not
>> > >> plugged into your Executor stuff.  It is a utility class which you
>> > >> don't have to use.  See the javadocs.
>> > >>
>> > >> But in your case I'd use it, to start with anyway.  Fire it up with
>> > >> suitable settings and forget about it, except to call close()
>> > >> eventually. Once you've got things up and running you can tweak
>> > >> things as much as you want but you appear to be having trouble
>> > >> getting up and running.
>> > >>
>> > >> So ... somewhere in the initialisation code of your app, create an
>> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
>> > >> outlined before.  Then pass the NRTManager to any/all write methods
>> > >> or threads and the SearcherManager instance to any/all search methods
>> > >> or threads and you're done.  If you want to use threads that are part
>> > >> of your ExecutorService, fine.  Just wrap it all together in whatever
>> > >> combination of Thread or Runnable instances you want.
>> > >>
>> > >>
>> > >> Does that help?
>> > >>
>> > >>
>> > >> --
>> > >> Ian.
>> > >>
>> > >>
>> > >> > I don't understand this following portion:
>> > >> >
>> > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
>> > >> > index); NRTManager nrtm = new NRTManager(iw, null);
>> > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm,
>> > >> > ...); ropt.setXxx(...); ....
>> > >> > ropt.start();
>> > >> >
>> > >> > I have a java ExecutorServices instance running which take care of
>> > >> > my
>> > >> own
>> > >> > applications. I don't know how this NRTManagerReopenThread works
>> > >> > with my own ExecutorService instance.
>> > >> >
>> > >> > Can both work together? How can the NRTManagerReopenThread
>> > instance
>> > >> ropt be
>> > >> > plugged into my own multithreading framework?
>> > >> >
>> > >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>> > >> >
>> > >> >> If you can use NRTManager and SearcherManager things should be

Good point. I should remove the commits.

Any difference between NRTCashingDirectory and RAMDirectory? how to define
the "small"?

On Tue, Feb 7, 2012 at 12:42 AM, Michael McCandless <
[EMAIL PROTECTED]> wrote:

> You shouldn't call IW.commit when using NRT; that's the point of NRT
> (making changes visible w/o calling commit).
>
> Only call commit when you require that all changes be durable (surive
> OS / JVM crash, power loss, etc.) on disk.
>
> Also, you can use NRTCachingDirectory which acts like RAMDirectory for
> small flushed segments.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Mon, Feb 6, 2012 at 10:45 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > Uwe, when I meant speed is slow, I didn't refer to instant visibility of
> > changes, but that the changes may be synchronized with FSDirectory when I
> > use writer.commit().
> >
> > When I use RAMDirectory, the writer.commit() seems much faster than using
> > NRTManager built upon FSDirectory. So, I am guessing the difference is
> the
> > index synchronization.
> >
> >
> >
> > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> >> Please review the following articles about NRT, absolutely instant
> updates
> >> that are visible as they are done are almost impossible (even with
> >> RAMDirectory):
> >>
> >> http://goo.gl/mzAHt
> >> http://goo.gl/5RoPx
> >> http://goo.gl/vSJ7x
> >>
> >> Uwe
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: [EMAIL PROTECTED]
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> > Sent: Monday, February 06, 2012 4:27 PM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: Configure writer to write to FSDirectory?
> >> >
> >> > Ian,
> >> >
> >> > I encountered an issue that I need to frequently update the index. The
> >> > NRTManager seems not very helpful on this front as the speed is slower
> >> than
> >> > RAMDirectory is used.
> >> >
> >> > Any improvement advice?
> >> >
> >> >
> >> >
> >> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > That really helps! I will try it out.
> >> > >
> >> > > Thanks.
> >> > >
> >> > >
> >> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
> >> > >
> >> > >> You would use NRTManagerReopenThread as a standalone thread, not
> >> > >> plugged into your Executor stuff.  It is a utility class which you
> >> > >> don't have to use.  See the javadocs.
> >> > >>
> >> > >> But in your case I'd use it, to start with anyway.  Fire it up with
> >> > >> suitable settings and forget about it, except to call close()
> >> > >> eventually. Once you've got things up and running you can tweak
> >> > >> things as much as you want but you appear to be having trouble
> >> > >> getting up and running.
> >> > >>
> >> > >> So ... somewhere in the initialisation code of your app, create an
> >> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
> >> > >> outlined before.  Then pass the NRTManager to any/all write methods
> >> > >> or threads and the SearcherManager instance to any/all search
> methods
> >> > >> or threads and you're done.  If you want to use threads that are
> part
> >> > >> of your ExecutorService, fine.  Just wrap it all together in
> whatever
> >> > >> combination of Thread or Runnable instances you want.
> >> > >>
> >> > >>
> >> > >> Does that help?
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Ian.
> >> > >>
> >> > >>
> >> > >> > I don't understand this following portion:
> >> > >> >
> >> > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk
> >> > >> > index); NRTManager nrtm = new NRTManager(iw, null);
> >> > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm,
> >> > >> > ...); ropt.setXxx(...); ....
> >> > >> > ropt.start();
> >> > >> >
> >> > >> > I have a java ExecutorServices instance running which take care

You tell NRTCachingDirectory how much RAM it's allowed to use, and it
then caches newly flushed segments in a private RAMDirectory.

But you should first test performance w/o it (after removing the
commit calls).  NRT is very fast...

Mike McCandless

http://blog.mikemccandless.com

On Mon, Feb 6, 2012 at 11:46 AM, Cheng <[EMAIL PROTECTED]> wrote:
> Good point. I should remove the commits.
>
> Any difference between NRTCashingDirectory and RAMDirectory? how to define
> the "small"?
>
> On Tue, Feb 7, 2012 at 12:42 AM, Michael McCandless <
> [EMAIL PROTECTED]> wrote:
>
>> You shouldn't call IW.commit when using NRT; that's the point of NRT
>> (making changes visible w/o calling commit).
>>
>> Only call commit when you require that all changes be durable (surive
>> OS / JVM crash, power loss, etc.) on disk.
>>
>> Also, you can use NRTCachingDirectory which acts like RAMDirectory for
>> small flushed segments.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Mon, Feb 6, 2012 at 10:45 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> > Uwe, when I meant speed is slow, I didn't refer to instant visibility of
>> > changes, but that the changes may be synchronized with FSDirectory when I
>> > use writer.commit().
>> >
>> > When I use RAMDirectory, the writer.commit() seems much faster than using
>> > NRTManager built upon FSDirectory. So, I am guessing the difference is
>> the
>> > index synchronization.
>> >
>> >
>> >
>> > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>> >
>> >> Please review the following articles about NRT, absolutely instant
>> updates
>> >> that are visible as they are done are almost impossible (even with
>> >> RAMDirectory):
>> >>
>> >> http://goo.gl/mzAHt
>> >> http://goo.gl/5RoPx
>> >> http://goo.gl/vSJ7x
>> >>
>> >> Uwe
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: [EMAIL PROTECTED]
>> >>
>> >> > -----Original Message-----
>> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> >> > Sent: Monday, February 06, 2012 4:27 PM
>> >> > To: [EMAIL PROTECTED]
>> >> > Subject: Re: Configure writer to write to FSDirectory?
>> >> >
>> >> > Ian,
>> >> >
>> >> > I encountered an issue that I need to frequently update the index. The
>> >> > NRTManager seems not very helpful on this front as the speed is slower
>> >> than
>> >> > RAMDirectory is used.
>> >> >
>> >> > Any improvement advice?
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]>
>> wrote:
>> >> >
>> >> > > That really helps! I will try it out.
>> >> > >
>> >> > > Thanks.
>> >> > >
>> >> > >
>> >> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]> wrote:
>> >> > >
>> >> > >> You would use NRTManagerReopenThread as a standalone thread, not
>> >> > >> plugged into your Executor stuff.  It is a utility class which you
>> >> > >> don't have to use.  See the javadocs.
>> >> > >>
>> >> > >> But in your case I'd use it, to start with anyway.  Fire it up with
>> >> > >> suitable settings and forget about it, except to call close()
>> >> > >> eventually. Once you've got things up and running you can tweak
>> >> > >> things as much as you want but you appear to be having trouble
>> >> > >> getting up and running.
>> >> > >>
>> >> > >> So ... somewhere in the initialisation code of your app, create an
>> >> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
>> >> > >> outlined before.  Then pass the NRTManager to any/all write methods
>> >> > >> or threads and the SearcherManager instance to any/all search
>> methods
>> >> > >> or threads and you're done.  If you want to use threads that are
>> part
>> >> > >> of your ExecutorService, fine.  Just wrap it all together in
>> whatever
>> >> > >> combination of Thread or Runnable instances you want.
>> >> > >>
>> >> > >>
>> >> > >> Does that help?
>> >> > >>
>> >> > >>
>> >> > >

Will do.

On Tue, Feb 7, 2012 at 12:52 AM, Michael McCandless <
[EMAIL PROTECTED]> wrote:

> You tell NRTCachingDirectory how much RAM it's allowed to use, and it
> then caches newly flushed segments in a private RAMDirectory.
>
> But you should first test performance w/o it (after removing the
> commit calls).  NRT is very fast...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Mon, Feb 6, 2012 at 11:46 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > Good point. I should remove the commits.
> >
> > Any difference between NRTCashingDirectory and RAMDirectory? how to
> define
> > the "small"?
> >
> > On Tue, Feb 7, 2012 at 12:42 AM, Michael McCandless <
> > [EMAIL PROTECTED]> wrote:
> >
> >> You shouldn't call IW.commit when using NRT; that's the point of NRT
> >> (making changes visible w/o calling commit).
> >>
> >> Only call commit when you require that all changes be durable (surive
> >> OS / JVM crash, power loss, etc.) on disk.
> >>
> >> Also, you can use NRTCachingDirectory which acts like RAMDirectory for
> >> small flushed segments.
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> On Mon, Feb 6, 2012 at 10:45 AM, Cheng <[EMAIL PROTECTED]> wrote:
> >> > Uwe, when I meant speed is slow, I didn't refer to instant visibility
> of
> >> > changes, but that the changes may be synchronized with FSDirectory
> when I
> >> > use writer.commit().
> >> >
> >> > When I use RAMDirectory, the writer.commit() seems much faster than
> using
> >> > NRTManager built upon FSDirectory. So, I am guessing the difference is
> >> the
> >> > index synchronization.
> >> >
> >> >
> >> >
> >> > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> >> Please review the following articles about NRT, absolutely instant
> >> updates
> >> >> that are visible as they are done are almost impossible (even with
> >> >> RAMDirectory):
> >> >>
> >> >> http://goo.gl/mzAHt
> >> >> http://goo.gl/5RoPx
> >> >> http://goo.gl/vSJ7x
> >> >>
> >> >> Uwe
> >> >>
> >> >> -----
> >> >> Uwe Schindler
> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> >> http://www.thetaphi.de
> >> >> eMail: [EMAIL PROTECTED]
> >> >>
> >> >> > -----Original Message-----
> >> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> >> > Sent: Monday, February 06, 2012 4:27 PM
> >> >> > To: [EMAIL PROTECTED]
> >> >> > Subject: Re: Configure writer to write to FSDirectory?
> >> >> >
> >> >> > Ian,
> >> >> >
> >> >> > I encountered an issue that I need to frequently update the index.
> The
> >> >> > NRTManager seems not very helpful on this front as the speed is
> slower
> >> >> than
> >> >> > RAMDirectory is used.
> >> >> >
> >> >> > Any improvement advice?
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <[EMAIL PROTECTED]>
> >> wrote:
> >> >> >
> >> >> > > That really helps! I will try it out.
> >> >> > >
> >> >> > > Thanks.
> >> >> > >
> >> >> > >
> >> >> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <[EMAIL PROTECTED]>
> wrote:
> >> >> > >
> >> >> > >> You would use NRTManagerReopenThread as a standalone thread, not
> >> >> > >> plugged into your Executor stuff.  It is a utility class which
> you
> >> >> > >> don't have to use.  See the javadocs.
> >> >> > >>
> >> >> > >> But in your case I'd use it, to start with anyway.  Fire it up
> with
> >> >> > >> suitable settings and forget about it, except to call close()
> >> >> > >> eventually. Once you've got things up and running you can tweak
> >> >> > >> things as much as you want but you appear to be having trouble
> >> >> > >> getting up and running.
> >> >> > >>
> >> >> > >> So ... somewhere in the initialisation code of your app, create
> an
> >> >> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as
> >> >> > >> outlined before.  Then pass the NRTManager to any/all write
> methods
> >> >> > >> or threads and the SearcherManager instance to any/all search