Cheng
2012-06-04, 14:08
Jack Krupansky
2012-06-04, 14:30
Cheng
2012-06-04, 14:55
Jason Rutherglen
2012-06-04, 14:58
Cheng
2012-06-04, 15:04
Jason Rutherglen
2012-06-04, 15:12
Uwe Schindler
2012-06-04, 15:42
Cheng
2012-06-04, 16:09
Uwe Schindler
2012-06-04, 23:53
Cheng
2012-06-16, 11:18
Michael McCandless
2012-06-18, 11:32
Cheng
2012-06-18, 11:42
Cheng
2012-06-30, 08:24
Michael McCandless
2012-06-30, 10:24
-
RAMDirectory unexpectedly slowsCheng 2012-06-04, 14:08
Hi,
My apps need to read from and write to some big indexes frequently. So I
use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
size.
I notice that the speed of reading and writing unexpectedly slows as the
size of the indexes increases. Since the usage of RAM is less than 20%, I
think by default the RAMDirectory doesn't take advantage of the memory I
assigned to JVM.
What are the steps to improve the reading and writing speed of
RAMDirectory?
Thanks!
Jeff
My apps need to read from and write to some big indexes frequently. So I
use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
size.
I notice that the speed of reading and writing unexpectedly slows as the
size of the indexes increases. Since the usage of RAM is less than 20%, I
think by default the RAMDirectory doesn't take advantage of the memory I
assigned to JVM.
What are the steps to improve the reading and writing speed of
RAMDirectory?
Thanks!
Jeff
-
Re: RAMDirectory unexpectedly slowsJack Krupansky 2012-06-04, 14:30
>From the javadoc for RAMDirectory:
"Warning: This class is not intended to work with huge indexes. Everything
beyond several hundred megabytes will waste resources (GC cycles), because
it uses an internal buffer size of 1024 bytes, producing millions of
byte[1024] arrays. This class is optimized for small memory-resident
indexes. It also has bad concurrency on multithreaded environments.
It is recommended to materialize large indexes on disk and use
MMapDirectory, which is a high-performance directory implementation working
directly on the file system cache of the operating system, so copying data
to Java heap space is not useful."
-- Jack Krupansky
-----Original Message-----
From: Cheng
Sent: Monday, June 04, 2012 10:08 AM
To: [EMAIL PROTECTED]
Subject: RAMDirectory unexpectedly slows
Hi,
My apps need to read from and write to some big indexes frequently. So I
use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
size.
I notice that the speed of reading and writing unexpectedly slows as the
size of the indexes increases. Since the usage of RAM is less than 20%, I
think by default the RAMDirectory doesn't take advantage of the memory I
assigned to JVM.
What are the steps to improve the reading and writing speed of
RAMDirectory?
Thanks!
Jeff
---------------------------------------------------------------------
"Warning: This class is not intended to work with huge indexes. Everything
beyond several hundred megabytes will waste resources (GC cycles), because
it uses an internal buffer size of 1024 bytes, producing millions of
byte[1024] arrays. This class is optimized for small memory-resident
indexes. It also has bad concurrency on multithreaded environments.
It is recommended to materialize large indexes on disk and use
MMapDirectory, which is a high-performance directory implementation working
directly on the file system cache of the operating system, so copying data
to Java heap space is not useful."
-- Jack Krupansky
-----Original Message-----
From: Cheng
Sent: Monday, June 04, 2012 10:08 AM
To: [EMAIL PROTECTED]
Subject: RAMDirectory unexpectedly slows
Hi,
My apps need to read from and write to some big indexes frequently. So I
use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
size.
I notice that the speed of reading and writing unexpectedly slows as the
size of the indexes increases. Since the usage of RAM is less than 20%, I
think by default the RAMDirectory doesn't take advantage of the memory I
assigned to JVM.
What are the steps to improve the reading and writing speed of
RAMDirectory?
Thanks!
Jeff
---------------------------------------------------------------------
-
Re: RAMDirectory unexpectedly slowsCheng 2012-06-04, 14:55
My indexes are 500MB+. So it seems like that RAMDirectory is not good for
that big a size.
My challenge, on the other side, is that I need to update the indexes very
frequently. So, do you think MMapDirectory is the solution?
Thanks.
On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]>wrote:
> From the javadoc for RAMDirectory:
>
> "Warning: This class is not intended to work with huge indexes. Everything
> beyond several hundred megabytes will waste resources (GC cycles), because
> it uses an internal buffer size of 1024 bytes, producing millions of
> byte[1024] arrays. This class is optimized for small memory-resident
> indexes. It also has bad concurrency on multithreaded environments.
>
> It is recommended to materialize large indexes on disk and use
> MMapDirectory, which is a high-performance directory implementation working
> directly on the file system cache of the operating system, so copying data
> to Java heap space is not useful."
>
> -- Jack Krupansky
>
> -----Original Message----- From: Cheng
> Sent: Monday, June 04, 2012 10:08 AM
> To: [EMAIL PROTECTED]
> Subject: RAMDirectory unexpectedly slows
>
>
> Hi,
>
> My apps need to read from and write to some big indexes frequently. So I
> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> size.
>
> I notice that the speed of reading and writing unexpectedly slows as the
> size of the indexes increases. Since the usage of RAM is less than 20%, I
> think by default the RAMDirectory doesn't take advantage of the memory I
> assigned to JVM.
>
> What are the steps to improve the reading and writing speed of
> RAMDirectory?
>
> Thanks!
> Jeff
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<[EMAIL PROTECTED]>
> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<[EMAIL PROTECTED]>
>
>
that big a size.
My challenge, on the other side, is that I need to update the indexes very
frequently. So, do you think MMapDirectory is the solution?
Thanks.
On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]>wrote:
> From the javadoc for RAMDirectory:
>
> "Warning: This class is not intended to work with huge indexes. Everything
> beyond several hundred megabytes will waste resources (GC cycles), because
> it uses an internal buffer size of 1024 bytes, producing millions of
> byte[1024] arrays. This class is optimized for small memory-resident
> indexes. It also has bad concurrency on multithreaded environments.
>
> It is recommended to materialize large indexes on disk and use
> MMapDirectory, which is a high-performance directory implementation working
> directly on the file system cache of the operating system, so copying data
> to Java heap space is not useful."
>
> -- Jack Krupansky
>
> -----Original Message----- From: Cheng
> Sent: Monday, June 04, 2012 10:08 AM
> To: [EMAIL PROTECTED]
> Subject: RAMDirectory unexpectedly slows
>
>
> Hi,
>
> My apps need to read from and write to some big indexes frequently. So I
> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> size.
>
> I notice that the speed of reading and writing unexpectedly slows as the
> size of the indexes increases. Since the usage of RAM is less than 20%, I
> think by default the RAMDirectory doesn't take advantage of the memory I
> assigned to JVM.
>
> What are the steps to improve the reading and writing speed of
> RAMDirectory?
>
> Thanks!
> Jeff
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<[EMAIL PROTECTED]>
> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<[EMAIL PROTECTED]>
>
>
-
Re: RAMDirectory unexpectedly slowsJason Rutherglen 2012-06-04, 14:58
If you want the index to be stored completely in RAM, there is the
ByteBuffer directory [1]. Though I do not see the point in putting an
index in RAM, it will be cached in RAM regardless in the OS system IO
cache.
1. https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> that big a size.
>
> My challenge, on the other side, is that I need to update the indexes very
> frequently. So, do you think MMapDirectory is the solution?
>
> Thanks.
>
> On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]>wrote:
>
>> From the javadoc for RAMDirectory:
>>
>> "Warning: This class is not intended to work with huge indexes. Everything
>> beyond several hundred megabytes will waste resources (GC cycles), because
>> it uses an internal buffer size of 1024 bytes, producing millions of
>> byte[1024] arrays. This class is optimized for small memory-resident
>> indexes. It also has bad concurrency on multithreaded environments.
>>
>> It is recommended to materialize large indexes on disk and use
>> MMapDirectory, which is a high-performance directory implementation working
>> directly on the file system cache of the operating system, so copying data
>> to Java heap space is not useful."
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Cheng
>> Sent: Monday, June 04, 2012 10:08 AM
>> To: [EMAIL PROTECTED]
>> Subject: RAMDirectory unexpectedly slows
>>
>>
>> Hi,
>>
>> My apps need to read from and write to some big indexes frequently. So I
>> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
>> size.
>>
>> I notice that the speed of reading and writing unexpectedly slows as the
>> size of the indexes increases. Since the usage of RAM is less than 20%, I
>> think by default the RAMDirectory doesn't take advantage of the memory I
>> assigned to JVM.
>>
>> What are the steps to improve the reading and writing speed of
>> RAMDirectory?
>>
>> Thanks!
>> Jeff
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<[EMAIL PROTECTED]>
>> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<[EMAIL PROTECTED]>
>>
>>
---------------------------------------------------------------------
ByteBuffer directory [1]. Though I do not see the point in putting an
index in RAM, it will be cached in RAM regardless in the OS system IO
cache.
1. https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> that big a size.
>
> My challenge, on the other side, is that I need to update the indexes very
> frequently. So, do you think MMapDirectory is the solution?
>
> Thanks.
>
> On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]>wrote:
>
>> From the javadoc for RAMDirectory:
>>
>> "Warning: This class is not intended to work with huge indexes. Everything
>> beyond several hundred megabytes will waste resources (GC cycles), because
>> it uses an internal buffer size of 1024 bytes, producing millions of
>> byte[1024] arrays. This class is optimized for small memory-resident
>> indexes. It also has bad concurrency on multithreaded environments.
>>
>> It is recommended to materialize large indexes on disk and use
>> MMapDirectory, which is a high-performance directory implementation working
>> directly on the file system cache of the operating system, so copying data
>> to Java heap space is not useful."
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Cheng
>> Sent: Monday, June 04, 2012 10:08 AM
>> To: [EMAIL PROTECTED]
>> Subject: RAMDirectory unexpectedly slows
>>
>>
>> Hi,
>>
>> My apps need to read from and write to some big indexes frequently. So I
>> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
>> size.
>>
>> I notice that the speed of reading and writing unexpectedly slows as the
>> size of the indexes increases. Since the usage of RAM is less than 20%, I
>> think by default the RAMDirectory doesn't take advantage of the memory I
>> assigned to JVM.
>>
>> What are the steps to improve the reading and writing speed of
>> RAMDirectory?
>>
>> Thanks!
>> Jeff
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<[EMAIL PROTECTED]>
>> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<[EMAIL PROTECTED]>
>>
>>
---------------------------------------------------------------------
-
Re: RAMDirectory unexpectedly slowsCheng 2012-06-04, 15:04
Please shed more insight into the difference between JVM heap size and the
memory size used by Lucene.
What I am getting at is that no matter however much ram I give my apps,
Lucene can't utilize it. Is that right?
What about the ByteBufferDirectory? Can this specific directory utilize the
2GB memory I grant to the app?
On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
[EMAIL PROTECTED]> wrote:
> If you want the index to be stored completely in RAM, there is the
> ByteBuffer directory [1]. Though I do not see the point in putting an
> index in RAM, it will be cached in RAM regardless in the OS system IO
> cache.
>
> 1.
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>
> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> > that big a size.
> >
> > My challenge, on the other side, is that I need to update the indexes
> very
> > frequently. So, do you think MMapDirectory is the solution?
> >
> > Thanks.
> >
> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]
> >wrote:
> >
> >> From the javadoc for RAMDirectory:
> >>
> >> "Warning: This class is not intended to work with huge indexes.
> Everything
> >> beyond several hundred megabytes will waste resources (GC cycles),
> because
> >> it uses an internal buffer size of 1024 bytes, producing millions of
> >> byte[1024] arrays. This class is optimized for small memory-resident
> >> indexes. It also has bad concurrency on multithreaded environments.
> >>
> >> It is recommended to materialize large indexes on disk and use
> >> MMapDirectory, which is a high-performance directory implementation
> working
> >> directly on the file system cache of the operating system, so copying
> data
> >> to Java heap space is not useful."
> >>
> >> -- Jack Krupansky
> >>
> >> -----Original Message----- From: Cheng
> >> Sent: Monday, June 04, 2012 10:08 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: RAMDirectory unexpectedly slows
> >>
> >>
> >> Hi,
> >>
> >> My apps need to read from and write to some big indexes frequently. So I
> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> >> size.
> >>
> >> I notice that the speed of reading and writing unexpectedly slows as the
> >> size of the indexes increases. Since the usage of RAM is less than 20%,
> I
> >> think by default the RAMDirectory doesn't take advantage of the memory I
> >> assigned to JVM.
> >>
> >> What are the steps to improve the reading and writing speed of
> >> RAMDirectory?
> >>
> >> Thanks!
> >> Jeff
> >>
> >>
> ------------------------------**------------------------------**---------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> [EMAIL PROTECTED]>
> >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
> [EMAIL PROTECTED]>
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
memory size used by Lucene.
What I am getting at is that no matter however much ram I give my apps,
Lucene can't utilize it. Is that right?
What about the ByteBufferDirectory? Can this specific directory utilize the
2GB memory I grant to the app?
On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
[EMAIL PROTECTED]> wrote:
> If you want the index to be stored completely in RAM, there is the
> ByteBuffer directory [1]. Though I do not see the point in putting an
> index in RAM, it will be cached in RAM regardless in the OS system IO
> cache.
>
> 1.
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>
> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> > that big a size.
> >
> > My challenge, on the other side, is that I need to update the indexes
> very
> > frequently. So, do you think MMapDirectory is the solution?
> >
> > Thanks.
> >
> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]
> >wrote:
> >
> >> From the javadoc for RAMDirectory:
> >>
> >> "Warning: This class is not intended to work with huge indexes.
> Everything
> >> beyond several hundred megabytes will waste resources (GC cycles),
> because
> >> it uses an internal buffer size of 1024 bytes, producing millions of
> >> byte[1024] arrays. This class is optimized for small memory-resident
> >> indexes. It also has bad concurrency on multithreaded environments.
> >>
> >> It is recommended to materialize large indexes on disk and use
> >> MMapDirectory, which is a high-performance directory implementation
> working
> >> directly on the file system cache of the operating system, so copying
> data
> >> to Java heap space is not useful."
> >>
> >> -- Jack Krupansky
> >>
> >> -----Original Message----- From: Cheng
> >> Sent: Monday, June 04, 2012 10:08 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: RAMDirectory unexpectedly slows
> >>
> >>
> >> Hi,
> >>
> >> My apps need to read from and write to some big indexes frequently. So I
> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> >> size.
> >>
> >> I notice that the speed of reading and writing unexpectedly slows as the
> >> size of the indexes increases. Since the usage of RAM is less than 20%,
> I
> >> think by default the RAMDirectory doesn't take advantage of the memory I
> >> assigned to JVM.
> >>
> >> What are the steps to improve the reading and writing speed of
> >> RAMDirectory?
> >>
> >> Thanks!
> >> Jeff
> >>
> >>
> ------------------------------**------------------------------**---------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> [EMAIL PROTECTED]>
> >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
> [EMAIL PROTECTED]>
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
-
Re: RAMDirectory unexpectedly slowsJason Rutherglen 2012-06-04, 15:12
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
BBD places the byte objects outside of the heap, so increasing the
heap size is only going to rob the system IO cache. With Lucene the
heap is only used for field caches and the terms dictionary index.
On Mon, Jun 4, 2012 at 11:04 AM, Cheng <[EMAIL PROTECTED]> wrote:
> Please shed more insight into the difference between JVM heap size and the
> memory size used by Lucene.
>
> What I am getting at is that no matter however much ram I give my apps,
> Lucene can't utilize it. Is that right?
>
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
>
> On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> [EMAIL PROTECTED]> wrote:
>
>> If you want the index to be stored completely in RAM, there is the
>> ByteBuffer directory [1]. Though I do not see the point in putting an
>> index in RAM, it will be cached in RAM regardless in the OS system IO
>> cache.
>>
>> 1.
>> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>>
>> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
>> > that big a size.
>> >
>> > My challenge, on the other side, is that I need to update the indexes
>> very
>> > frequently. So, do you think MMapDirectory is the solution?
>> >
>> > Thanks.
>> >
>> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> From the javadoc for RAMDirectory:
>> >>
>> >> "Warning: This class is not intended to work with huge indexes.
>> Everything
>> >> beyond several hundred megabytes will waste resources (GC cycles),
>> because
>> >> it uses an internal buffer size of 1024 bytes, producing millions of
>> >> byte[1024] arrays. This class is optimized for small memory-resident
>> >> indexes. It also has bad concurrency on multithreaded environments.
>> >>
>> >> It is recommended to materialize large indexes on disk and use
>> >> MMapDirectory, which is a high-performance directory implementation
>> working
>> >> directly on the file system cache of the operating system, so copying
>> data
>> >> to Java heap space is not useful."
>> >>
>> >> -- Jack Krupansky
>> >>
>> >> -----Original Message----- From: Cheng
>> >> Sent: Monday, June 04, 2012 10:08 AM
>> >> To: [EMAIL PROTECTED]
>> >> Subject: RAMDirectory unexpectedly slows
>> >>
>> >>
>> >> Hi,
>> >>
>> >> My apps need to read from and write to some big indexes frequently. So I
>> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
>> >> size.
>> >>
>> >> I notice that the speed of reading and writing unexpectedly slows as the
>> >> size of the indexes increases. Since the usage of RAM is less than 20%,
>> I
>> >> think by default the RAMDirectory doesn't take advantage of the memory I
>> >> assigned to JVM.
>> >>
>> >> What are the steps to improve the reading and writing speed of
>> >> RAMDirectory?
>> >>
>> >> Thanks!
>> >> Jeff
>> >>
>> >>
>> ------------------------------**------------------------------**---------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
>> [EMAIL PROTECTED]>
>> >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
>> [EMAIL PROTECTED]>
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>
---------------------------------------------------------------------
> 2GB memory I grant to the app?
BBD places the byte objects outside of the heap, so increasing the
heap size is only going to rob the system IO cache. With Lucene the
heap is only used for field caches and the terms dictionary index.
On Mon, Jun 4, 2012 at 11:04 AM, Cheng <[EMAIL PROTECTED]> wrote:
> Please shed more insight into the difference between JVM heap size and the
> memory size used by Lucene.
>
> What I am getting at is that no matter however much ram I give my apps,
> Lucene can't utilize it. Is that right?
>
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
>
> On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> [EMAIL PROTECTED]> wrote:
>
>> If you want the index to be stored completely in RAM, there is the
>> ByteBuffer directory [1]. Though I do not see the point in putting an
>> index in RAM, it will be cached in RAM regardless in the OS system IO
>> cache.
>>
>> 1.
>> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>>
>> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
>> > that big a size.
>> >
>> > My challenge, on the other side, is that I need to update the indexes
>> very
>> > frequently. So, do you think MMapDirectory is the solution?
>> >
>> > Thanks.
>> >
>> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]
>> >wrote:
>> >
>> >> From the javadoc for RAMDirectory:
>> >>
>> >> "Warning: This class is not intended to work with huge indexes.
>> Everything
>> >> beyond several hundred megabytes will waste resources (GC cycles),
>> because
>> >> it uses an internal buffer size of 1024 bytes, producing millions of
>> >> byte[1024] arrays. This class is optimized for small memory-resident
>> >> indexes. It also has bad concurrency on multithreaded environments.
>> >>
>> >> It is recommended to materialize large indexes on disk and use
>> >> MMapDirectory, which is a high-performance directory implementation
>> working
>> >> directly on the file system cache of the operating system, so copying
>> data
>> >> to Java heap space is not useful."
>> >>
>> >> -- Jack Krupansky
>> >>
>> >> -----Original Message----- From: Cheng
>> >> Sent: Monday, June 04, 2012 10:08 AM
>> >> To: [EMAIL PROTECTED]
>> >> Subject: RAMDirectory unexpectedly slows
>> >>
>> >>
>> >> Hi,
>> >>
>> >> My apps need to read from and write to some big indexes frequently. So I
>> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
>> >> size.
>> >>
>> >> I notice that the speed of reading and writing unexpectedly slows as the
>> >> size of the indexes increases. Since the usage of RAM is less than 20%,
>> I
>> >> think by default the RAMDirectory doesn't take advantage of the memory I
>> >> assigned to JVM.
>> >>
>> >> What are the steps to improve the reading and writing speed of
>> >> RAMDirectory?
>> >>
>> >> Thanks!
>> >> Jeff
>> >>
>> >>
>> ------------------------------**------------------------------**---------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
>> [EMAIL PROTECTED]>
>> >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
>> [EMAIL PROTECTED]>
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail: [EMAIL PROTECTED]
>>
>>
---------------------------------------------------------------------
-
Re: RAMDirectory unexpectedly slowsUwe Schindler 2012-06-04, 15:42
Hi,
If you are using MMapDirectory or this ByteBufferDirectory (which is similar to the first) the used RAM is outside JVM heap, it is in the FS cache of the OS kernel. Giving too much memory to the JVM penalizes the OS cache, so give only as much as the App needs. Lucene and the OS kernel will then utilize the remaining memory for caching.
Please read docs of MMapDirectory and inform yourself about mmap in e.g. Wikipedia.
Uwe
--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de
Cheng <[EMAIL PROTECTED]> schrieb:
Please shed more insight into the difference between JVM heap size and the
memory size used by Lucene.
What I am getting at is that no matter however much ram I give my apps,
Lucene can't utilize it. Is that right?
What about the ByteBufferDirectory? Can this specific directory utilize the
2GB memory I grant to the app?
On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
[EMAIL PROTECTED]> wrote:
> If you want the index to be stored completely in RAM, there is the
> ByteBuffer directory [1]. Though I do not see the point in putting an
> index in RAM, it will be cached in RAM regardless in the OS system IO
> cache.
>
> 1.
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>
> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> > that big a size.
> >
> > My challenge, on the other side, is that I need to update the indexes
> very
> > frequently. So, do you think MMapDirectory is the solution?
> >
> > Thanks.
> >
> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]
> >wrote:
> >
> >> From the javadoc for RAMDirectory:
> >>
> >> "Warning: This class is not intended to work with huge indexes.
> Everything
> >> beyond several hundred megabytes will waste resources (GC cycles),
> because
> >> it uses an internal buffer size of 1024 bytes, producing millions of
> >> byte[1024] arrays. This class is optimized for small memory-resident
> >> indexes. It also has bad concurrency on multithreaded environments.
> >>
> >> It is recommended to materialize large indexes on disk and use
> >> MMapDirectory, which is a high-performance directory implementation
> working
> >> directly on the file system cache of the operating system, so copying
> data
> >> to Java heap space is not useful."
> >>
> >> -- Jack Krupansky
> >>
> >> -----Original Message----- From: Cheng
> >> Sent: Monday, June 04, 2012 10:08 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: RAMDirectory unexpectedly slows
> >>
> >>
> >> Hi,
> >>
> >> My apps need to read from and write to some big indexes frequently. So I
> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> >> size.
> >>
> >> I notice that the speed of reading and writing unexpectedly slows as the
> >> size of the indexes increases. Since the usage of RAM is less than 20%,
> I
> >> think by default the RAMDirectory doesn't take advantage of the memory I
> >> assigned to JVM.
> >>
> >> What are the steps to improve the reading and writing speed of
> >> RAMDirectory?
> >>
> >> Thanks!
> >> Jeff
> >>
> >>
>_____________________________________________
**_____________________________________________
**---------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> [EMAIL PROTECTED]>
> >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
> [EMAIL PROTECTED]>
> >>
> >>
>
>_____________________________________________
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
If you are using MMapDirectory or this ByteBufferDirectory (which is similar to the first) the used RAM is outside JVM heap, it is in the FS cache of the OS kernel. Giving too much memory to the JVM penalizes the OS cache, so give only as much as the App needs. Lucene and the OS kernel will then utilize the remaining memory for caching.
Please read docs of MMapDirectory and inform yourself about mmap in e.g. Wikipedia.
Uwe
--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de
Cheng <[EMAIL PROTECTED]> schrieb:
Please shed more insight into the difference between JVM heap size and the
memory size used by Lucene.
What I am getting at is that no matter however much ram I give my apps,
Lucene can't utilize it. Is that right?
What about the ByteBufferDirectory? Can this specific directory utilize the
2GB memory I grant to the app?
On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
[EMAIL PROTECTED]> wrote:
> If you want the index to be stored completely in RAM, there is the
> ByteBuffer directory [1]. Though I do not see the point in putting an
> index in RAM, it will be cached in RAM regardless in the OS system IO
> cache.
>
> 1.
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>
> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
> > that big a size.
> >
> > My challenge, on the other side, is that I need to update the indexes
> very
> > frequently. So, do you think MMapDirectory is the solution?
> >
> > Thanks.
> >
> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <[EMAIL PROTECTED]
> >wrote:
> >
> >> From the javadoc for RAMDirectory:
> >>
> >> "Warning: This class is not intended to work with huge indexes.
> Everything
> >> beyond several hundred megabytes will waste resources (GC cycles),
> because
> >> it uses an internal buffer size of 1024 bytes, producing millions of
> >> byte[1024] arrays. This class is optimized for small memory-resident
> >> indexes. It also has bad concurrency on multithreaded environments.
> >>
> >> It is recommended to materialize large indexes on disk and use
> >> MMapDirectory, which is a high-performance directory implementation
> working
> >> directly on the file system cache of the operating system, so copying
> data
> >> to Java heap space is not useful."
> >>
> >> -- Jack Krupansky
> >>
> >> -----Original Message----- From: Cheng
> >> Sent: Monday, June 04, 2012 10:08 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: RAMDirectory unexpectedly slows
> >>
> >>
> >> Hi,
> >>
> >> My apps need to read from and write to some big indexes frequently. So I
> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> >> size.
> >>
> >> I notice that the speed of reading and writing unexpectedly slows as the
> >> size of the indexes increases. Since the usage of RAM is less than 20%,
> I
> >> think by default the RAMDirectory doesn't take advantage of the memory I
> >> assigned to JVM.
> >>
> >> What are the steps to improve the reading and writing speed of
> >> RAMDirectory?
> >>
> >> Thanks!
> >> Jeff
> >>
> >>
>_____________________________________________
**_____________________________________________
**---------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> [EMAIL PROTECTED]>
> >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
> [EMAIL PROTECTED]>
> >>
> >>
>
>_____________________________________________
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
-
Re: RAMDirectory unexpectedly slowsCheng 2012-06-04, 16:09
Can I control the size of ram given to either MMapDirectory or
ByteBufferDirectory?
On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> Hi,
>
> If you are using MMapDirectory or this ByteBufferDirectory (which is
> similar to the first) the used RAM is outside JVM heap, it is in the FS
> cache of the OS kernel. Giving too much memory to the JVM penalizes the OS
> cache, so give only as much as the App needs. Lucene and the OS kernel will
> then utilize the remaining memory for caching.
>
> Please read docs of MMapDirectory and inform yourself about mmap in e.g.
> Wikipedia.
>
> Uwe
> --
> Uwe Schindler
> H.-H.-Meier-Allee 63, 28213 Bremen
> http://www.thetaphi.de
>
>
>
> Cheng <[EMAIL PROTECTED]> schrieb:
>
> Please shed more insight into the difference between JVM heap size and the
> memory size used by Lucene.
>
> What I am getting at is that no matter however much ram I give my apps,
> Lucene can't utilize it. Is that right?
>
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
>
> On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> [EMAIL PROTECTED]> wrote:
>
> > If you want the index to be stored completely in RAM, there is the
> > ByteBuffer directory [1]. Though I do not see the point in putting an
> > index in RAM, it will be cached in RAM regardless in the OS system IO
> > cache.
> >
> > 1.
> >
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> >
> > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > > My indexes are 500MB+. So it seems like that RAMDirectory is not good
> for
> > > that big a size.
> > >
> > > My challenge, on the other side, is that I need to update the indexes
> > very
> > > frequently. So, do you think MMapDirectory is the solution?
> > >
> > > Thanks.
> > >
> > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> From the javadoc for RAMDirectory:
> > >>
> > >> "Warning: This class is not intended to work with huge indexes.
> > Everything
> > >> beyond several hundred megabytes will waste resources (GC cycles),
> > because
> > >> it uses an internal buffer size of 1024 bytes, producing millions of
> > >> byte[1024] arrays. This class is optimized for small memory-resident
> > >> indexes. It also has bad concurrency on multithreaded environments.
> > >>
> > >> It is recommended to materialize large indexes on disk and use
> > >> MMapDirectory, which is a high-performance directory implementation
> > working
> > >> directly on the file system cache of the operating system, so copying
> > data
> > >> to Java heap space is not useful."
> > >>
> > >> -- Jack Krupansky
> > >>
> > >> -----Original Message----- From: Cheng
> > >> Sent: Monday, June 04, 2012 10:08 AM
> > >> To: [EMAIL PROTECTED]
> > >> Subject: RAMDirectory unexpectedly slows
> > >>
> > >>
> > >> Hi,
> > >>
> > >> My apps need to read from and write to some big indexes frequently.
> So I
> > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> > >> size.
> > >>
> > >> I notice that the speed of reading and writing unexpectedly slows as
> the
> > >> size of the indexes increases. Since the usage of RAM is less than
> 20%,
> > I
> > >> think by default the RAMDirectory doesn't take advantage of the
> memory I
> > >> assigned to JVM.
> > >>
> > >> What are the steps to improve the reading and writing speed of
> > >> RAMDirectory?
> > >>
> > >> Thanks!
> > >> Jeff
> > >>
> > >>
> >_____________________________________________
> **_____________________________________________
> **---------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> > [EMAIL PROTECTED]>
> > >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
> > [EMAIL PROTECTED]>
> > >>
> > >>
> >
> >_____________________________________________
ByteBufferDirectory?
On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> Hi,
>
> If you are using MMapDirectory or this ByteBufferDirectory (which is
> similar to the first) the used RAM is outside JVM heap, it is in the FS
> cache of the OS kernel. Giving too much memory to the JVM penalizes the OS
> cache, so give only as much as the App needs. Lucene and the OS kernel will
> then utilize the remaining memory for caching.
>
> Please read docs of MMapDirectory and inform yourself about mmap in e.g.
> Wikipedia.
>
> Uwe
> --
> Uwe Schindler
> H.-H.-Meier-Allee 63, 28213 Bremen
> http://www.thetaphi.de
>
>
>
> Cheng <[EMAIL PROTECTED]> schrieb:
>
> Please shed more insight into the difference between JVM heap size and the
> memory size used by Lucene.
>
> What I am getting at is that no matter however much ram I give my apps,
> Lucene can't utilize it. Is that right?
>
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
>
> On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> [EMAIL PROTECTED]> wrote:
>
> > If you want the index to be stored completely in RAM, there is the
> > ByteBuffer directory [1]. Though I do not see the point in putting an
> > index in RAM, it will be cached in RAM regardless in the OS system IO
> > cache.
> >
> > 1.
> >
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> >
> > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > > My indexes are 500MB+. So it seems like that RAMDirectory is not good
> for
> > > that big a size.
> > >
> > > My challenge, on the other side, is that I need to update the indexes
> > very
> > > frequently. So, do you think MMapDirectory is the solution?
> > >
> > > Thanks.
> > >
> > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> From the javadoc for RAMDirectory:
> > >>
> > >> "Warning: This class is not intended to work with huge indexes.
> > Everything
> > >> beyond several hundred megabytes will waste resources (GC cycles),
> > because
> > >> it uses an internal buffer size of 1024 bytes, producing millions of
> > >> byte[1024] arrays. This class is optimized for small memory-resident
> > >> indexes. It also has bad concurrency on multithreaded environments.
> > >>
> > >> It is recommended to materialize large indexes on disk and use
> > >> MMapDirectory, which is a high-performance directory implementation
> > working
> > >> directly on the file system cache of the operating system, so copying
> > data
> > >> to Java heap space is not useful."
> > >>
> > >> -- Jack Krupansky
> > >>
> > >> -----Original Message----- From: Cheng
> > >> Sent: Monday, June 04, 2012 10:08 AM
> > >> To: [EMAIL PROTECTED]
> > >> Subject: RAMDirectory unexpectedly slows
> > >>
> > >>
> > >> Hi,
> > >>
> > >> My apps need to read from and write to some big indexes frequently.
> So I
> > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
> > >> size.
> > >>
> > >> I notice that the speed of reading and writing unexpectedly slows as
> the
> > >> size of the indexes increases. Since the usage of RAM is less than
> 20%,
> > I
> > >> think by default the RAMDirectory doesn't take advantage of the
> memory I
> > >> assigned to JVM.
> > >>
> > >> What are the steps to improve the reading and writing speed of
> > >> RAMDirectory?
> > >>
> > >> Thanks!
> > >> Jeff
> > >>
> > >>
> >_____________________________________________
> **_____________________________________________
> **---------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
> > [EMAIL PROTECTED]>
> > >> For additional commands, e-mail: [EMAIL PROTECTED]he.**org<
> > [EMAIL PROTECTED]>
> > >>
> > >>
> >
> >_____________________________________________
-
RE: RAMDirectory unexpectedly slowsUwe Schindler 2012-06-04, 23:53
This is managed by your operating system. In general OS kernels like Linux
or Windows use all free memory to cache disk accesses.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Cheng [mailto:[EMAIL PROTECTED]]
> Sent: Monday, June 04, 2012 6:10 PM
> To: [EMAIL PROTECTED]
> Subject: Re: RAMDirectory unexpectedly slows
>
> Can I control the size of ram given to either MMapDirectory or
> ByteBufferDirectory?
>
> On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > If you are using MMapDirectory or this ByteBufferDirectory (which is
> > similar to the first) the used RAM is outside JVM heap, it is in the
> > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
> > the OS cache, so give only as much as the App needs. Lucene and the OS
> > kernel will then utilize the remaining memory for caching.
> >
> > Please read docs of MMapDirectory and inform yourself about mmap in e.g.
> > Wikipedia.
> >
> > Uwe
> > --
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, 28213 Bremen
> > http://www.thetaphi.de
> >
> >
> >
> > Cheng <[EMAIL PROTECTED]> schrieb:
> >
> > Please shed more insight into the difference between JVM heap size and
> > the memory size used by Lucene.
> >
> > What I am getting at is that no matter however much ram I give my
> > apps, Lucene can't utilize it. Is that right?
> >
> > What about the ByteBufferDirectory? Can this specific directory
> > utilize the 2GB memory I grant to the app?
> >
> > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> > [EMAIL PROTECTED]> wrote:
> >
> > > If you want the index to be stored completely in RAM, there is the
> > > ByteBuffer directory [1]. Though I do not see the point in putting
> > > an index in RAM, it will be cached in RAM regardless in the OS
> > > system IO cache.
> > >
> > > 1.
> > >
> > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> > >
> > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
> wrote:
> > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> > > > good
> > for
> > > > that big a size.
> > > >
> > > > My challenge, on the other side, is that I need to update the
> > > > indexes
> > > very
> > > > frequently. So, do you think MMapDirectory is the solution?
> > > >
> > > > Thanks.
> > > >
> > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> > [EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > >> From the javadoc for RAMDirectory:
> > > >>
> > > >> "Warning: This class is not intended to work with huge indexes.
> > > Everything
> > > >> beyond several hundred megabytes will waste resources (GC
> > > >> cycles),
> > > because
> > > >> it uses an internal buffer size of 1024 bytes, producing millions
> > > >> of byte[1024] arrays. This class is optimized for small
> > > >> memory-resident indexes. It also has bad concurrency on
multithreaded
> environments.
> > > >>
> > > >> It is recommended to materialize large indexes on disk and use
> > > >> MMapDirectory, which is a high-performance directory
> > > >> implementation
> > > working
> > > >> directly on the file system cache of the operating system, so
> > > >> copying
> > > data
> > > >> to Java heap space is not useful."
> > > >>
> > > >> -- Jack Krupansky
> > > >>
> > > >> -----Original Message----- From: Cheng
> > > >> Sent: Monday, June 04, 2012 10:08 AM
> > > >> To: [EMAIL PROTECTED]
> > > >> Subject: RAMDirectory unexpectedly slows
> > > >>
> > > >>
> > > >> Hi,
> > > >>
> > > >> My apps need to read from and write to some big indexes frequently.
> > So I
> > > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB
> > > >> memory size.
> > > >>
> > > >> I notice that the speed of reading and writing unexpectedly slows
> > > >> as
> > the
> > > >> size of the indexes increases. Since the usage of RAM is less
or Windows use all free memory to cache disk accesses.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Cheng [mailto:[EMAIL PROTECTED]]
> Sent: Monday, June 04, 2012 6:10 PM
> To: [EMAIL PROTECTED]
> Subject: Re: RAMDirectory unexpectedly slows
>
> Can I control the size of ram given to either MMapDirectory or
> ByteBufferDirectory?
>
> On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > If you are using MMapDirectory or this ByteBufferDirectory (which is
> > similar to the first) the used RAM is outside JVM heap, it is in the
> > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
> > the OS cache, so give only as much as the App needs. Lucene and the OS
> > kernel will then utilize the remaining memory for caching.
> >
> > Please read docs of MMapDirectory and inform yourself about mmap in e.g.
> > Wikipedia.
> >
> > Uwe
> > --
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, 28213 Bremen
> > http://www.thetaphi.de
> >
> >
> >
> > Cheng <[EMAIL PROTECTED]> schrieb:
> >
> > Please shed more insight into the difference between JVM heap size and
> > the memory size used by Lucene.
> >
> > What I am getting at is that no matter however much ram I give my
> > apps, Lucene can't utilize it. Is that right?
> >
> > What about the ByteBufferDirectory? Can this specific directory
> > utilize the 2GB memory I grant to the app?
> >
> > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> > [EMAIL PROTECTED]> wrote:
> >
> > > If you want the index to be stored completely in RAM, there is the
> > > ByteBuffer directory [1]. Though I do not see the point in putting
> > > an index in RAM, it will be cached in RAM regardless in the OS
> > > system IO cache.
> > >
> > > 1.
> > >
> > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> > >
> > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
> wrote:
> > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> > > > good
> > for
> > > > that big a size.
> > > >
> > > > My challenge, on the other side, is that I need to update the
> > > > indexes
> > > very
> > > > frequently. So, do you think MMapDirectory is the solution?
> > > >
> > > > Thanks.
> > > >
> > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> > [EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > >> From the javadoc for RAMDirectory:
> > > >>
> > > >> "Warning: This class is not intended to work with huge indexes.
> > > Everything
> > > >> beyond several hundred megabytes will waste resources (GC
> > > >> cycles),
> > > because
> > > >> it uses an internal buffer size of 1024 bytes, producing millions
> > > >> of byte[1024] arrays. This class is optimized for small
> > > >> memory-resident indexes. It also has bad concurrency on
multithreaded
> environments.
> > > >>
> > > >> It is recommended to materialize large indexes on disk and use
> > > >> MMapDirectory, which is a high-performance directory
> > > >> implementation
> > > working
> > > >> directly on the file system cache of the operating system, so
> > > >> copying
> > > data
> > > >> to Java heap space is not useful."
> > > >>
> > > >> -- Jack Krupansky
> > > >>
> > > >> -----Original Message----- From: Cheng
> > > >> Sent: Monday, June 04, 2012 10:08 AM
> > > >> To: [EMAIL PROTECTED]
> > > >> Subject: RAMDirectory unexpectedly slows
> > > >>
> > > >>
> > > >> Hi,
> > > >>
> > > >> My apps need to read from and write to some big indexes frequently.
> > So I
> > > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB
> > > >> memory size.
> > > >>
> > > >> I notice that the speed of reading and writing unexpectedly slows
> > > >> as
> > the
> > > >> size of the indexes increases. Since the usage of RAM is less
-
Re: RAMDirectory unexpectedly slowsCheng 2012-06-16, 11:18
After a number of test, the performance of MMapDirectory is not even close
to that of RAMDirectory, in terms of speed.
My application w/ the former can only deal with 10 tasks per round while it
could handle over 90 w/ RAMDirectory.
I use the application in Linux.
What can be the reasons?
Thanks.
On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> This is managed by your operating system. In general OS kernels like Linux
> or Windows use all free memory to cache disk accesses.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [EMAIL PROTECTED]
>
>
> > -----Original Message-----
> > From: Cheng [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, June 04, 2012 6:10 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: RAMDirectory unexpectedly slows
> >
> > Can I control the size of ram given to either MMapDirectory or
> > ByteBufferDirectory?
> >
> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
> > > similar to the first) the used RAM is outside JVM heap, it is in the
> > > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
> > > the OS cache, so give only as much as the App needs. Lucene and the OS
> > > kernel will then utilize the remaining memory for caching.
> > >
> > > Please read docs of MMapDirectory and inform yourself about mmap in
> e.g.
> > > Wikipedia.
> > >
> > > Uwe
> > > --
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, 28213 Bremen
> > > http://www.thetaphi.de
> > >
> > >
> > >
> > > Cheng <[EMAIL PROTECTED]> schrieb:
> > >
> > > Please shed more insight into the difference between JVM heap size and
> > > the memory size used by Lucene.
> > >
> > > What I am getting at is that no matter however much ram I give my
> > > apps, Lucene can't utilize it. Is that right?
> > >
> > > What about the ByteBufferDirectory? Can this specific directory
> > > utilize the 2GB memory I grant to the app?
> > >
> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > If you want the index to be stored completely in RAM, there is the
> > > > ByteBuffer directory [1]. Though I do not see the point in putting
> > > > an index in RAM, it will be cached in RAM regardless in the OS
> > > > system IO cache.
> > > >
> > > > 1.
> > > >
> > > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> > > >
> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
> > wrote:
> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> > > > > good
> > > for
> > > > > that big a size.
> > > > >
> > > > > My challenge, on the other side, is that I need to update the
> > > > > indexes
> > > > very
> > > > > frequently. So, do you think MMapDirectory is the solution?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> > > [EMAIL PROTECTED]
> > > > >wrote:
> > > > >
> > > > >> From the javadoc for RAMDirectory:
> > > > >>
> > > > >> "Warning: This class is not intended to work with huge indexes.
> > > > Everything
> > > > >> beyond several hundred megabytes will waste resources (GC
> > > > >> cycles),
> > > > because
> > > > >> it uses an internal buffer size of 1024 bytes, producing millions
> > > > >> of byte[1024] arrays. This class is optimized for small
> > > > >> memory-resident indexes. It also has bad concurrency on
> multithreaded
> > environments.
> > > > >>
> > > > >> It is recommended to materialize large indexes on disk and use
> > > > >> MMapDirectory, which is a high-performance directory
> > > > >> implementation
> > > > working
> > > > >> directly on the file system cache of the operating system, so
> > > > >> copying
> > > > data
> > > > >> to Java heap space is not useful."
> > > > >>
> > > > >> -- Jack Krupansky
to that of RAMDirectory, in terms of speed.
My application w/ the former can only deal with 10 tasks per round while it
could handle over 90 w/ RAMDirectory.
I use the application in Linux.
What can be the reasons?
Thanks.
On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> This is managed by your operating system. In general OS kernels like Linux
> or Windows use all free memory to cache disk accesses.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: [EMAIL PROTECTED]
>
>
> > -----Original Message-----
> > From: Cheng [mailto:[EMAIL PROTECTED]]
> > Sent: Monday, June 04, 2012 6:10 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: RAMDirectory unexpectedly slows
> >
> > Can I control the size of ram given to either MMapDirectory or
> > ByteBufferDirectory?
> >
> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
> > > similar to the first) the used RAM is outside JVM heap, it is in the
> > > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
> > > the OS cache, so give only as much as the App needs. Lucene and the OS
> > > kernel will then utilize the remaining memory for caching.
> > >
> > > Please read docs of MMapDirectory and inform yourself about mmap in
> e.g.
> > > Wikipedia.
> > >
> > > Uwe
> > > --
> > > Uwe Schindler
> > > H.-H.-Meier-Allee 63, 28213 Bremen
> > > http://www.thetaphi.de
> > >
> > >
> > >
> > > Cheng <[EMAIL PROTECTED]> schrieb:
> > >
> > > Please shed more insight into the difference between JVM heap size and
> > > the memory size used by Lucene.
> > >
> > > What I am getting at is that no matter however much ram I give my
> > > apps, Lucene can't utilize it. Is that right?
> > >
> > > What about the ByteBufferDirectory? Can this specific directory
> > > utilize the 2GB memory I grant to the app?
> > >
> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > If you want the index to be stored completely in RAM, there is the
> > > > ByteBuffer directory [1]. Though I do not see the point in putting
> > > > an index in RAM, it will be cached in RAM regardless in the OS
> > > > system IO cache.
> > > >
> > > > 1.
> > > >
> > > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> > > >
> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
> > wrote:
> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> > > > > good
> > > for
> > > > > that big a size.
> > > > >
> > > > > My challenge, on the other side, is that I need to update the
> > > > > indexes
> > > > very
> > > > > frequently. So, do you think MMapDirectory is the solution?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> > > [EMAIL PROTECTED]
> > > > >wrote:
> > > > >
> > > > >> From the javadoc for RAMDirectory:
> > > > >>
> > > > >> "Warning: This class is not intended to work with huge indexes.
> > > > Everything
> > > > >> beyond several hundred megabytes will waste resources (GC
> > > > >> cycles),
> > > > because
> > > > >> it uses an internal buffer size of 1024 bytes, producing millions
> > > > >> of byte[1024] arrays. This class is optimized for small
> > > > >> memory-resident indexes. It also has bad concurrency on
> multithreaded
> > environments.
> > > > >>
> > > > >> It is recommended to materialize large indexes on disk and use
> > > > >> MMapDirectory, which is a high-performance directory
> > > > >> implementation
> > > > working
> > > > >> directly on the file system cache of the operating system, so
> > > > >> copying
> > > > data
> > > > >> to Java heap space is not useful."
> > > > >>
> > > > >> -- Jack Krupansky
-
Re: RAMDirectory unexpectedly slowsMichael McCandless 2012-06-18, 11:32
9 fold improvement using RAMDir over MMapDir is much more than I've
seen (~30-40% maybe) in the past.
Can you explain how you are using Lucene?
You may also want to try the CachingRAMDirectory patch on
https://issues.apache.org/jira/browse/LUCENE-4123
Mike McCandless
http://blog.mikemccandless.com
On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
> After a number of test, the performance of MMapDirectory is not even close
> to that of RAMDirectory, in terms of speed.
>
> My application w/ the former can only deal with 10 tasks per round while it
> could handle over 90 w/ RAMDirectory.
>
> I use the application in Linux.
>
> What can be the reasons?
>
> Thanks.
>
>
> On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
>> This is managed by your operating system. In general OS kernels like Linux
>> or Windows use all free memory to cache disk accesses.
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: [EMAIL PROTECTED]
>>
>>
>> > -----Original Message-----
>> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> > Sent: Monday, June 04, 2012 6:10 PM
>> > To: [EMAIL PROTECTED]
>> > Subject: Re: RAMDirectory unexpectedly slows
>> >
>> > Can I control the size of ram given to either MMapDirectory or
>> > ByteBufferDirectory?
>> >
>> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>> >
>> > > Hi,
>> > >
>> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
>> > > similar to the first) the used RAM is outside JVM heap, it is in the
>> > > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
>> > > the OS cache, so give only as much as the App needs. Lucene and the OS
>> > > kernel will then utilize the remaining memory for caching.
>> > >
>> > > Please read docs of MMapDirectory and inform yourself about mmap in
>> e.g.
>> > > Wikipedia.
>> > >
>> > > Uwe
>> > > --
>> > > Uwe Schindler
>> > > H.-H.-Meier-Allee 63, 28213 Bremen
>> > > http://www.thetaphi.de
>> > >
>> > >
>> > >
>> > > Cheng <[EMAIL PROTECTED]> schrieb:
>> > >
>> > > Please shed more insight into the difference between JVM heap size and
>> > > the memory size used by Lucene.
>> > >
>> > > What I am getting at is that no matter however much ram I give my
>> > > apps, Lucene can't utilize it. Is that right?
>> > >
>> > > What about the ByteBufferDirectory? Can this specific directory
>> > > utilize the 2GB memory I grant to the app?
>> > >
>> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
>> > > [EMAIL PROTECTED]> wrote:
>> > >
>> > > > If you want the index to be stored completely in RAM, there is the
>> > > > ByteBuffer directory [1]. Though I do not see the point in putting
>> > > > an index in RAM, it will be cached in RAM regardless in the OS
>> > > > system IO cache.
>> > > >
>> > > > 1.
>> > > >
>> > > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
>> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>> > > >
>> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
>> > wrote:
>> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
>> > > > > good
>> > > for
>> > > > > that big a size.
>> > > > >
>> > > > > My challenge, on the other side, is that I need to update the
>> > > > > indexes
>> > > > very
>> > > > > frequently. So, do you think MMapDirectory is the solution?
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
>> > > [EMAIL PROTECTED]
>> > > > >wrote:
>> > > > >
>> > > > >> From the javadoc for RAMDirectory:
>> > > > >>
>> > > > >> "Warning: This class is not intended to work with huge indexes.
>> > > > Everything
>> > > > >> beyond several hundred megabytes will waste resources (GC
>> > > > >> cycles),
>> > > > because
>> > > > >> it uses an internal buffer size of 1024 bytes, producing millions
>> > > > >> of byte[1024] arrays. This class is optimized for small
seen (~30-40% maybe) in the past.
Can you explain how you are using Lucene?
You may also want to try the CachingRAMDirectory patch on
https://issues.apache.org/jira/browse/LUCENE-4123
Mike McCandless
http://blog.mikemccandless.com
On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
> After a number of test, the performance of MMapDirectory is not even close
> to that of RAMDirectory, in terms of speed.
>
> My application w/ the former can only deal with 10 tasks per round while it
> could handle over 90 w/ RAMDirectory.
>
> I use the application in Linux.
>
> What can be the reasons?
>
> Thanks.
>
>
> On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
>> This is managed by your operating system. In general OS kernels like Linux
>> or Windows use all free memory to cache disk accesses.
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: [EMAIL PROTECTED]
>>
>>
>> > -----Original Message-----
>> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> > Sent: Monday, June 04, 2012 6:10 PM
>> > To: [EMAIL PROTECTED]
>> > Subject: Re: RAMDirectory unexpectedly slows
>> >
>> > Can I control the size of ram given to either MMapDirectory or
>> > ByteBufferDirectory?
>> >
>> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>> >
>> > > Hi,
>> > >
>> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
>> > > similar to the first) the used RAM is outside JVM heap, it is in the
>> > > FS cache of the OS kernel. Giving too much memory to the JVM penalizes
>> > > the OS cache, so give only as much as the App needs. Lucene and the OS
>> > > kernel will then utilize the remaining memory for caching.
>> > >
>> > > Please read docs of MMapDirectory and inform yourself about mmap in
>> e.g.
>> > > Wikipedia.
>> > >
>> > > Uwe
>> > > --
>> > > Uwe Schindler
>> > > H.-H.-Meier-Allee 63, 28213 Bremen
>> > > http://www.thetaphi.de
>> > >
>> > >
>> > >
>> > > Cheng <[EMAIL PROTECTED]> schrieb:
>> > >
>> > > Please shed more insight into the difference between JVM heap size and
>> > > the memory size used by Lucene.
>> > >
>> > > What I am getting at is that no matter however much ram I give my
>> > > apps, Lucene can't utilize it. Is that right?
>> > >
>> > > What about the ByteBufferDirectory? Can this specific directory
>> > > utilize the 2GB memory I grant to the app?
>> > >
>> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
>> > > [EMAIL PROTECTED]> wrote:
>> > >
>> > > > If you want the index to be stored completely in RAM, there is the
>> > > > ByteBuffer directory [1]. Though I do not see the point in putting
>> > > > an index in RAM, it will be cached in RAM regardless in the OS
>> > > > system IO cache.
>> > > >
>> > > > 1.
>> > > >
>> > > https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
>> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>> > > >
>> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
>> > wrote:
>> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
>> > > > > good
>> > > for
>> > > > > that big a size.
>> > > > >
>> > > > > My challenge, on the other side, is that I need to update the
>> > > > > indexes
>> > > > very
>> > > > > frequently. So, do you think MMapDirectory is the solution?
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
>> > > [EMAIL PROTECTED]
>> > > > >wrote:
>> > > > >
>> > > > >> From the javadoc for RAMDirectory:
>> > > > >>
>> > > > >> "Warning: This class is not intended to work with huge indexes.
>> > > > Everything
>> > > > >> beyond several hundred megabytes will waste resources (GC
>> > > > >> cycles),
>> > > > because
>> > > > >> it uses an internal buffer size of 1024 bytes, producing millions
>> > > > >> of byte[1024] arrays. This class is optimized for small
-
Re: RAMDirectory unexpectedly slowsCheng 2012-06-18, 11:42
Lucene is used in the following steps:
1) store interested data in Lucene indexes;
2) search key words against the indexes;
3) write new data into indexes and refresh the reader;
4) use the reader to search key words, and the 2-4 steps repeat.
As you see, there are lots of read, update actions. I guess since MMapDir
needs to synchronize to a local drive that causes it to be slower.
The code is attached:
public class YYTLucene {
private static Logger logger = Logger.getLogger(YYTLuceneImpl.class);
private static FSDirectory indexDir;
private static RAMDirectory ramDir;
// private static MMapDirectory ramDir;
private static IndexWriter iw;
private static IndexSearcher is;
private static IndexReader ir;
private static YYTLucene instance;
public static YYTLucene getInstance(String type) {
if (instance == null) {
instance = new YYTLucene(type);
}
return instance;
}
private YYTLucene(String type) {
try {
indexDir = new NIOFSDirectory(new File(ERConstants.indexFolder1
+ "/" + type));
ramDir = new RAMDirectory(indexDir);
// ramDir = new MMapDirectory(new File(ERConstants.indexFolder1 +
// "/"
// + type));
IndexWriterConfig iwConfig = new IndexWriterConfig(
ERConstants.version, new LimitTokenCountAnalyzer(
ERConstants.analyzer, ERConstants.maxTokenNum));
// iwConfig.setMaxBufferedDocs(ERConstants.maxBufferedDocs);
//
// iwConfig.setRAMBufferSizeMB(ERConstants.RAMBufferSizeMB);
iw = new IndexWriter(ramDir, iwConfig);
iw.commit();
ir = IndexReader.open(iw, true);
is = new IndexSearcher(ir);
} catch (IOException e) {
e.printStackTrace();
logger.info("Can't initiate YYTLuceneImpl...");
}
}
public IndexWriter getIndexWriter() {
return iw;
}
public void setIndexWriter(IndexWriter iw) {
YYTLucene.iw = iw;
}
public IndexSearcher getIndexSearcher() {
return is;
}
public void setIndexSearcher(IndexSearcher is) {
YYTLucene.is = is;
}
public IndexReader getIndexReader() {
return ir;
}
public static void setIndexReader(IndexReader ir) {
YYTLucene.ir = ir;
}
}
On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> 9 fold improvement using RAMDir over MMapDir is much more than I've
> seen (~30-40% maybe) in the past.
>
> Can you explain how you are using Lucene?
>
> You may also want to try the CachingRAMDirectory patch on
> https://issues.apache.org/jira/browse/LUCENE-4123
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > After a number of test, the performance of MMapDirectory is not even
> close
> > to that of RAMDirectory, in terms of speed.
> >
> > My application w/ the former can only deal with 10 tasks per round while
> it
> > could handle over 90 w/ RAMDirectory.
> >
> > I use the application in Linux.
> >
> > What can be the reasons?
> >
> > Thanks.
> >
> >
> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> >> This is managed by your operating system. In general OS kernels like
> Linux
> >> or Windows use all free memory to cache disk accesses.
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: [EMAIL PROTECTED]
> >>
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> > Sent: Monday, June 04, 2012 6:10 PM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: RAMDirectory unexpectedly slows
> >> >
> >> > Can I control the size of ram given to either MMapDirectory or
> >> > ByteBufferDirectory?
> >> >
> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
> >> > > similar to the first) the used RAM is outside JVM heap, it is in the
> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
> penalizes
> >> > > the OS cache, so give only as much as the App needs. Lucene and the
> OS
> >> > > kernel will then utilize the remaining memory for caching.
1) store interested data in Lucene indexes;
2) search key words against the indexes;
3) write new data into indexes and refresh the reader;
4) use the reader to search key words, and the 2-4 steps repeat.
As you see, there are lots of read, update actions. I guess since MMapDir
needs to synchronize to a local drive that causes it to be slower.
The code is attached:
public class YYTLucene {
private static Logger logger = Logger.getLogger(YYTLuceneImpl.class);
private static FSDirectory indexDir;
private static RAMDirectory ramDir;
// private static MMapDirectory ramDir;
private static IndexWriter iw;
private static IndexSearcher is;
private static IndexReader ir;
private static YYTLucene instance;
public static YYTLucene getInstance(String type) {
if (instance == null) {
instance = new YYTLucene(type);
}
return instance;
}
private YYTLucene(String type) {
try {
indexDir = new NIOFSDirectory(new File(ERConstants.indexFolder1
+ "/" + type));
ramDir = new RAMDirectory(indexDir);
// ramDir = new MMapDirectory(new File(ERConstants.indexFolder1 +
// "/"
// + type));
IndexWriterConfig iwConfig = new IndexWriterConfig(
ERConstants.version, new LimitTokenCountAnalyzer(
ERConstants.analyzer, ERConstants.maxTokenNum));
// iwConfig.setMaxBufferedDocs(ERConstants.maxBufferedDocs);
//
// iwConfig.setRAMBufferSizeMB(ERConstants.RAMBufferSizeMB);
iw = new IndexWriter(ramDir, iwConfig);
iw.commit();
ir = IndexReader.open(iw, true);
is = new IndexSearcher(ir);
} catch (IOException e) {
e.printStackTrace();
logger.info("Can't initiate YYTLuceneImpl...");
}
}
public IndexWriter getIndexWriter() {
return iw;
}
public void setIndexWriter(IndexWriter iw) {
YYTLucene.iw = iw;
}
public IndexSearcher getIndexSearcher() {
return is;
}
public void setIndexSearcher(IndexSearcher is) {
YYTLucene.is = is;
}
public IndexReader getIndexReader() {
return ir;
}
public static void setIndexReader(IndexReader ir) {
YYTLucene.ir = ir;
}
}
On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> 9 fold improvement using RAMDir over MMapDir is much more than I've
> seen (~30-40% maybe) in the past.
>
> Can you explain how you are using Lucene?
>
> You may also want to try the CachingRAMDirectory patch on
> https://issues.apache.org/jira/browse/LUCENE-4123
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > After a number of test, the performance of MMapDirectory is not even
> close
> > to that of RAMDirectory, in terms of speed.
> >
> > My application w/ the former can only deal with 10 tasks per round while
> it
> > could handle over 90 w/ RAMDirectory.
> >
> > I use the application in Linux.
> >
> > What can be the reasons?
> >
> > Thanks.
> >
> >
> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> >> This is managed by your operating system. In general OS kernels like
> Linux
> >> or Windows use all free memory to cache disk accesses.
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: [EMAIL PROTECTED]
> >>
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> > Sent: Monday, June 04, 2012 6:10 PM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: RAMDirectory unexpectedly slows
> >> >
> >> > Can I control the size of ram given to either MMapDirectory or
> >> > ByteBufferDirectory?
> >> >
> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
> >> > > similar to the first) the used RAM is outside JVM heap, it is in the
> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
> penalizes
> >> > > the OS cache, so give only as much as the App needs. Lucene and the
> OS
> >> > > kernel will then utilize the remaining memory for caching.
-
Re: RAMDirectory unexpectedly slowsCheng 2012-06-30, 08:24
Hi,
I can't find the CachingRAMDirectory in Lucene 3.6. Is this decaperated?
Thanks
On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> 9 fold improvement using RAMDir over MMapDir is much more than I've
> seen (~30-40% maybe) in the past.
>
> Can you explain how you are using Lucene?
>
> You may also want to try the CachingRAMDirectory patch on
> https://issues.apache.org/jira/browse/LUCENE-4123
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > After a number of test, the performance of MMapDirectory is not even
> close
> > to that of RAMDirectory, in terms of speed.
> >
> > My application w/ the former can only deal with 10 tasks per round while
> it
> > could handle over 90 w/ RAMDirectory.
> >
> > I use the application in Linux.
> >
> > What can be the reasons?
> >
> > Thanks.
> >
> >
> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> >> This is managed by your operating system. In general OS kernels like
> Linux
> >> or Windows use all free memory to cache disk accesses.
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: [EMAIL PROTECTED]
> >>
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> > Sent: Monday, June 04, 2012 6:10 PM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: RAMDirectory unexpectedly slows
> >> >
> >> > Can I control the size of ram given to either MMapDirectory or
> >> > ByteBufferDirectory?
> >> >
> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
> >> > > similar to the first) the used RAM is outside JVM heap, it is in the
> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
> penalizes
> >> > > the OS cache, so give only as much as the App needs. Lucene and the
> OS
> >> > > kernel will then utilize the remaining memory for caching.
> >> > >
> >> > > Please read docs of MMapDirectory and inform yourself about mmap in
> >> e.g.
> >> > > Wikipedia.
> >> > >
> >> > > Uwe
> >> > > --
> >> > > Uwe Schindler
> >> > > H.-H.-Meier-Allee 63, 28213 Bremen
> >> > > http://www.thetaphi.de
> >> > >
> >> > >
> >> > >
> >> > > Cheng <[EMAIL PROTECTED]> schrieb:
> >> > >
> >> > > Please shed more insight into the difference between JVM heap size
> and
> >> > > the memory size used by Lucene.
> >> > >
> >> > > What I am getting at is that no matter however much ram I give my
> >> > > apps, Lucene can't utilize it. Is that right?
> >> > >
> >> > > What about the ByteBufferDirectory? Can this specific directory
> >> > > utilize the 2GB memory I grant to the app?
> >> > >
> >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> >> > > [EMAIL PROTECTED]> wrote:
> >> > >
> >> > > > If you want the index to be stored completely in RAM, there is the
> >> > > > ByteBuffer directory [1]. Though I do not see the point in putting
> >> > > > an index in RAM, it will be cached in RAM regardless in the OS
> >> > > > system IO cache.
> >> > > >
> >> > > > 1.
> >> > > >
> >> > >
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> >> > > >
> >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
> >> > wrote:
> >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> >> > > > > good
> >> > > for
> >> > > > > that big a size.
> >> > > > >
> >> > > > > My challenge, on the other side, is that I need to update the
> >> > > > > indexes
> >> > > > very
> >> > > > > frequently. So, do you think MMapDirectory is the solution?
> >> > > > >
> >> > > > > Thanks.
> >> > > > >
> >> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> >> > > [EMAIL PROTECTED]
I can't find the CachingRAMDirectory in Lucene 3.6. Is this decaperated?
Thanks
On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> 9 fold improvement using RAMDir over MMapDir is much more than I've
> seen (~30-40% maybe) in the past.
>
> Can you explain how you are using Lucene?
>
> You may also want to try the CachingRAMDirectory patch on
> https://issues.apache.org/jira/browse/LUCENE-4123
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
> > After a number of test, the performance of MMapDirectory is not even
> close
> > to that of RAMDirectory, in terms of speed.
> >
> > My application w/ the former can only deal with 10 tasks per round while
> it
> > could handle over 90 w/ RAMDirectory.
> >
> > I use the application in Linux.
> >
> > What can be the reasons?
> >
> > Thanks.
> >
> >
> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> >
> >> This is managed by your operating system. In general OS kernels like
> Linux
> >> or Windows use all free memory to cache disk accesses.
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: [EMAIL PROTECTED]
> >>
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
> >> > Sent: Monday, June 04, 2012 6:10 PM
> >> > To: [EMAIL PROTECTED]
> >> > Subject: Re: RAMDirectory unexpectedly slows
> >> >
> >> > Can I control the size of ram given to either MMapDirectory or
> >> > ByteBufferDirectory?
> >> >
> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
> >> > > similar to the first) the used RAM is outside JVM heap, it is in the
> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
> penalizes
> >> > > the OS cache, so give only as much as the App needs. Lucene and the
> OS
> >> > > kernel will then utilize the remaining memory for caching.
> >> > >
> >> > > Please read docs of MMapDirectory and inform yourself about mmap in
> >> e.g.
> >> > > Wikipedia.
> >> > >
> >> > > Uwe
> >> > > --
> >> > > Uwe Schindler
> >> > > H.-H.-Meier-Allee 63, 28213 Bremen
> >> > > http://www.thetaphi.de
> >> > >
> >> > >
> >> > >
> >> > > Cheng <[EMAIL PROTECTED]> schrieb:
> >> > >
> >> > > Please shed more insight into the difference between JVM heap size
> and
> >> > > the memory size used by Lucene.
> >> > >
> >> > > What I am getting at is that no matter however much ram I give my
> >> > > apps, Lucene can't utilize it. Is that right?
> >> > >
> >> > > What about the ByteBufferDirectory? Can this specific directory
> >> > > utilize the 2GB memory I grant to the app?
> >> > >
> >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> >> > > [EMAIL PROTECTED]> wrote:
> >> > >
> >> > > > If you want the index to be stored completely in RAM, there is the
> >> > > > ByteBuffer directory [1]. Though I do not see the point in putting
> >> > > > an index in RAM, it will be cached in RAM regardless in the OS
> >> > > > system IO cache.
> >> > > >
> >> > > > 1.
> >> > > >
> >> > >
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> >> > > >
> >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
> >> > wrote:
> >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> >> > > > > good
> >> > > for
> >> > > > > that big a size.
> >> > > > >
> >> > > > > My challenge, on the other side, is that I need to update the
> >> > > > > indexes
> >> > > > very
> >> > > > > frequently. So, do you think MMapDirectory is the solution?
> >> > > > >
> >> > > > > Thanks.
> >> > > > >
> >> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> >> > > [EMAIL PROTECTED]
-
Re: RAMDirectory unexpectedly slowsMichael McCandless 2012-06-30, 10:24
It hasn't been committed yet ... it's a patch on
https://issues.apache.org/jira/browse/LUCENE-4123
Mike McCandless
http://blog.mikemccandless.com
On Sat, Jun 30, 2012 at 4:24 AM, Cheng <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I can't find the CachingRAMDirectory in Lucene 3.6. Is this decaperated?
>
> Thanks
>
> On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
> [EMAIL PROTECTED]> wrote:
>
>> 9 fold improvement using RAMDir over MMapDir is much more than I've
>> seen (~30-40% maybe) in the past.
>>
>> Can you explain how you are using Lucene?
>>
>> You may also want to try the CachingRAMDirectory patch on
>> https://issues.apache.org/jira/browse/LUCENE-4123
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> > After a number of test, the performance of MMapDirectory is not even
>> close
>> > to that of RAMDirectory, in terms of speed.
>> >
>> > My application w/ the former can only deal with 10 tasks per round while
>> it
>> > could handle over 90 w/ RAMDirectory.
>> >
>> > I use the application in Linux.
>> >
>> > What can be the reasons?
>> >
>> > Thanks.
>> >
>> >
>> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>> >
>> >> This is managed by your operating system. In general OS kernels like
>> Linux
>> >> or Windows use all free memory to cache disk accesses.
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: [EMAIL PROTECTED]
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> >> > Sent: Monday, June 04, 2012 6:10 PM
>> >> > To: [EMAIL PROTECTED]
>> >> > Subject: Re: RAMDirectory unexpectedly slows
>> >> >
>> >> > Can I control the size of ram given to either MMapDirectory or
>> >> > ByteBufferDirectory?
>> >> >
>> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]>
>> wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
>> >> > > similar to the first) the used RAM is outside JVM heap, it is in the
>> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
>> penalizes
>> >> > > the OS cache, so give only as much as the App needs. Lucene and the
>> OS
>> >> > > kernel will then utilize the remaining memory for caching.
>> >> > >
>> >> > > Please read docs of MMapDirectory and inform yourself about mmap in
>> >> e.g.
>> >> > > Wikipedia.
>> >> > >
>> >> > > Uwe
>> >> > > --
>> >> > > Uwe Schindler
>> >> > > H.-H.-Meier-Allee 63, 28213 Bremen
>> >> > > http://www.thetaphi.de
>> >> > >
>> >> > >
>> >> > >
>> >> > > Cheng <[EMAIL PROTECTED]> schrieb:
>> >> > >
>> >> > > Please shed more insight into the difference between JVM heap size
>> and
>> >> > > the memory size used by Lucene.
>> >> > >
>> >> > > What I am getting at is that no matter however much ram I give my
>> >> > > apps, Lucene can't utilize it. Is that right?
>> >> > >
>> >> > > What about the ByteBufferDirectory? Can this specific directory
>> >> > > utilize the 2GB memory I grant to the app?
>> >> > >
>> >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
>> >> > > [EMAIL PROTECTED]> wrote:
>> >> > >
>> >> > > > If you want the index to be stored completely in RAM, there is the
>> >> > > > ByteBuffer directory [1]. Though I do not see the point in putting
>> >> > > > an index in RAM, it will be cached in RAM regardless in the OS
>> >> > > > system IO cache.
>> >> > > >
>> >> > > > 1.
>> >> > > >
>> >> > >
>> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
>> >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>> >> > > >
>> >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
>> >> > > > > good
>> >> > > for
>> >> > > > > that big a size.
https://issues.apache.org/jira/browse/LUCENE-4123
Mike McCandless
http://blog.mikemccandless.com
On Sat, Jun 30, 2012 at 4:24 AM, Cheng <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I can't find the CachingRAMDirectory in Lucene 3.6. Is this decaperated?
>
> Thanks
>
> On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
> [EMAIL PROTECTED]> wrote:
>
>> 9 fold improvement using RAMDir over MMapDir is much more than I've
>> seen (~30-40% maybe) in the past.
>>
>> Can you explain how you are using Lucene?
>>
>> You may also want to try the CachingRAMDirectory patch on
>> https://issues.apache.org/jira/browse/LUCENE-4123
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <[EMAIL PROTECTED]> wrote:
>> > After a number of test, the performance of MMapDirectory is not even
>> close
>> > to that of RAMDirectory, in terms of speed.
>> >
>> > My application w/ the former can only deal with 10 tasks per round while
>> it
>> > could handle over 90 w/ RAMDirectory.
>> >
>> > I use the application in Linux.
>> >
>> > What can be the reasons?
>> >
>> > Thanks.
>> >
>> >
>> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>> >
>> >> This is managed by your operating system. In general OS kernels like
>> Linux
>> >> or Windows use all free memory to cache disk accesses.
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: [EMAIL PROTECTED]
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: Cheng [mailto:[EMAIL PROTECTED]]
>> >> > Sent: Monday, June 04, 2012 6:10 PM
>> >> > To: [EMAIL PROTECTED]
>> >> > Subject: Re: RAMDirectory unexpectedly slows
>> >> >
>> >> > Can I control the size of ram given to either MMapDirectory or
>> >> > ByteBufferDirectory?
>> >> >
>> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <[EMAIL PROTECTED]>
>> wrote:
>> >> >
>> >> > > Hi,
>> >> > >
>> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
>> >> > > similar to the first) the used RAM is outside JVM heap, it is in the
>> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
>> penalizes
>> >> > > the OS cache, so give only as much as the App needs. Lucene and the
>> OS
>> >> > > kernel will then utilize the remaining memory for caching.
>> >> > >
>> >> > > Please read docs of MMapDirectory and inform yourself about mmap in
>> >> e.g.
>> >> > > Wikipedia.
>> >> > >
>> >> > > Uwe
>> >> > > --
>> >> > > Uwe Schindler
>> >> > > H.-H.-Meier-Allee 63, 28213 Bremen
>> >> > > http://www.thetaphi.de
>> >> > >
>> >> > >
>> >> > >
>> >> > > Cheng <[EMAIL PROTECTED]> schrieb:
>> >> > >
>> >> > > Please shed more insight into the difference between JVM heap size
>> and
>> >> > > the memory size used by Lucene.
>> >> > >
>> >> > > What I am getting at is that no matter however much ram I give my
>> >> > > apps, Lucene can't utilize it. Is that right?
>> >> > >
>> >> > > What about the ByteBufferDirectory? Can this specific directory
>> >> > > utilize the 2GB memory I grant to the app?
>> >> > >
>> >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
>> >> > > [EMAIL PROTECTED]> wrote:
>> >> > >
>> >> > > > If you want the index to be stored completely in RAM, there is the
>> >> > > > ByteBuffer directory [1]. Though I do not see the point in putting
>> >> > > > an index in RAM, it will be cached in RAM regardless in the OS
>> >> > > > system IO cache.
>> >> > > >
>> >> > > > 1.
>> >> > > >
>> >> > >
>> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
>> >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>> >> > > >
>> >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <[EMAIL PROTECTED]>
>> >> > wrote:
>> >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
>> >> > > > > good
>> >> > > for
>> >> > > > > that big a size.
