Lucene, mail # user - MMapDirectory vs RAMDirectory

Re: MMapDirectory vs RAMDirectory

eks dev 2006-05-28, 08:28

If you can use all that memory for index, I would say RAM. For long running indexes (to get os cache populated), MMAP will do just as good if you have any file system worth using.

----- Original Message ----
From: Michael Chan <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, 28 May, 2006 10:09:30 AM
Subject: MMapDirectory vs RAMDirectory

Hi,

On a 64-bit platform with 30gb RAM and 8 real CPUs, should
MMapDirectory or RAMDirectory provide better search performance on a
5gb index?

Cheers,

Michael

---------------------------------------------------------------------

---------------------------------------------------------------------

Re: MMapDirectory vs RAMDirectory

Peter Keegan 2006-06-05, 13:54

I'm reposting this from java-dev to java-user for greater exposure.

My search process is using MMapDirectory on a read-only index via:

-Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory

Another indexing process is building the next version of the index in a
different directory. When it's time to switch to the new index, the search
process closes the old IndexSearcher, MultiReader (2) and FSDirectories (2)
and opens the new index. Subsequently, attempts to delete the old index
files fail because there are still references to them from the old
MMapDirectory (in contrast, the deletes succeed when using FSDirectory).

There is no 'unmap' method, so my understanding is that the file mapping is
valid until the underlying buffer is garbage-collected. However, forcing the
gc doesn't help.

I found this bug listed at Sun:
http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4724038
Synopsis: (fs) Add unmap method to MappedByteBuffer)

The file deletes don't fail on Linux, but I'm wondering if there is still a
memory leak?

Has anyone discovered a way of releasing the old memory mapped files so that
they can be deleted sucessfully on Windows? My current workaround is to
periodically reattempt to delete the old index files.

Peter

Re: MMapDirectory vs RAMDirectory

Daniel Noll 2006-06-06, 00:05

Peter Keegan wrote:

> There is no 'unmap' method, so my understanding is that the file mapping is
> valid until the underlying buffer is garbage-collected. However, forcing
> the gc doesn't help.

You're half right.

The file mapping is indeed valid until the underlying buffer is garbage
collected, but you can't "force" the GC -- there is no API which does that.

Note the wording in the Javadoc for System.gc():

   "Calling the gc method **suggests** that the Java Virtual Machine
   expend effort toward recycling unused objects in order to make the
   memory they currently occupy available for quick reuse. When control
   returns from the method call, the Java Virtual Machine has made a
   best effort to reclaim space from all discarded objects."

> The file deletes don't fail on Linux, but I'm wondering if there is still a
> memory leak?

Linux allows you to delete a file while someone has the file descriptor
open, but the file descriptor will remain valid (i.e. the delete doesn't
actually occur) until everyone releases the file descriptor.

I ran into similar issues as these when working on other things, and
eventually ended up switching to using a RandomAccessFile, as those can
be closed. Otherwise you're right -- the workaround is to routinely try
to delete the file.

Daniel

--
Daniel Noll

Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com.au/ Fax: +61 2 9212 6902

This message is intended only for the named recipient. If you are not
the intended recipient you are notified that disclosing, copying,
distributing or taking any action in reliance on the contents of this
message or attachment is strictly prohibited.

---------------------------------------------------------------------

Re: MMapDirectory vs RAMDirectory

Peter Keegan 2006-06-07, 15:04

I was able to improve the behavior by setting the mapped ByteBuffer to null
in the close method of MMapIndexInput. This seems to be a strong enough
'suggestion' to the gc, as I can see the references go away with process
explorer, and the index files can be deleted, usually. Occasionally, a
reference to the '.tis' file remains.

Peter
On 6/5/06, Daniel Noll <[EMAIL PROTECTED]> wrote:
>
> Peter Keegan wrote:
>
> > There is no 'unmap' method, so my understanding is that the file mapping
> is
> > valid until the underlying buffer is garbage-collected. However, forcing
> > the gc doesn't help.
>
> You're half right.
>
> The file mapping is indeed valid until the underlying buffer is garbage
> collected, but you can't "force" the GC -- there is no API which does
> that.
>
> Note the wording in the Javadoc for System.gc():
>
> "Calling the gc method **suggests** that the Java Virtual Machine
> expend effort toward recycling unused objects in order to make the
> memory they currently occupy available for quick reuse. When control
> returns from the method call, the Java Virtual Machine has made a
> best effort to reclaim space from all discarded objects."
>
> > The file deletes don't fail on Linux, but I'm wondering if there is
> still a
> > memory leak?
>
> Linux allows you to delete a file while someone has the file descriptor
> open, but the file descriptor will remain valid (i.e. the delete doesn't
> actually occur) until everyone releases the file descriptor.
>
> I ran into similar issues as these when working on other things, and
> eventually ended up switching to using a RandomAccessFile, as those can
> be closed. Otherwise you're right -- the workaround is to routinely try
> to delete the file.
>
> Daniel
>
> --
> Daniel Noll
>
> Nuix Pty Ltd
> Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
> Web: http://www.nuix.com.au/ Fax: +61 2 9212 6902
>
> This message is intended only for the named recipient. If you are not
> the intended recipient you are notified that disclosing, copying,
> distributing or taking any action in reliance on the contents of this
> message or attachment is strictly prohibited.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>