Gopal Patwa
2012-03-31, 05:26
Michael McCandless
2012-03-31, 10:15
Gopal Patwa
2012-04-11, 03:35
Michael McCandless
2012-04-11, 09:20
Gopal Patwa
2012-04-11, 22:59
Michael McCandless
2012-04-12, 10:07
Mark Miller
2012-04-12, 16:04
Gopal Patwa
2012-04-14, 16:39
Boon Low
2012-04-19, 14:42
Gopal Patwa
2012-04-20, 13:33
Michael McCandless
2012-04-22, 12:29
-
Large Index and OutOfMemoryError: Map failedGopal Patwa 2012-03-31, 05:26
*I need help!!*
*
*
*I am using Solr 4.0 nightly build with NRT and I often get this error
during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I
have search this forum and what I found it is related to OS ulimit
setting, please se below my ulimit settings. I am not sure what ulimit
setting I should have? and we also get "**java.net.SocketException:*
*Too* *many* *open* *files" NOT sure how many open file we need to
set?*
I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 -
15GB, with Single shard
*
*
*We update the index every 5 seconds, soft commit every 1 second and
hard commit every 15 minutes*
*
*
*Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB*
*
*
ulimit:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 401408
max locked memory (kbytes, -l) 1024
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 401408
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*
*
*ERROR:*
*
*
*2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=*
*thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto*
*commit* *error...:java.io.IOException:* *Map* *failed*
*at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*)
*at* *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*)
*at* *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*)
*at* *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*)
*at* *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*)
*at* *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*)
*at* *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*)
*at* *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*)
*at* *org.apache.lucene.index.IndexWriter.maybeApplyDeletes*(*IndexWriter.java:2930*)
*at* *org.apache.lucene.index.IndexWriter.prepareCommit*(*IndexWriter.java:2681*)
*at* *org.apache.lucene.index.IndexWriter.commitInternal*(*IndexWriter.java:2804*)
*at* *org.apache.lucene.index.IndexWriter.commit*(*IndexWriter.java:2786*)
*at* *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:391*)
*at* *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*)
*at* *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*)
*at* *java.util.concurrent.FutureTask$Sync.innerRun*(*FutureTask.java:303*)
*at* *java.util.concurrent.FutureTask.run*(*FutureTask.java:138*)
*at* *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301*(*ScheduledThreadPoolExecutor.java:98*)
*at* *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run*(*ScheduledThreadPoolExecutor.java:206*)
*at* *java.util.concurrent.ThreadPoolExecutor$Worker.runTask*(*ThreadPoolExecutor.java:886*)
*at* *java.util.concurrent.ThreadPoolExecutor$Worker.run*(*ThreadPoolExecutor.java:908*)
*at* *java.lang.Thread.run*(*Thread.java:662*)*Caused* *by:*
*java.lang.OutOfMemoryError:* *Map* *failed*
*at* *sun.nio.ch.FileChannelImpl.map0*(*Native* *Method*)
*at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:745*)
*...* *28* *more*
*
*
*
*
*
SolrConfig.xml:
<indexDefaults>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength-->
<ramBufferSizeMB>4096</ramBufferSizeMB>
<maxThreadStates>10</maxThreadStates>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
<lockType>single</lockType>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<double name="forceMergeDeletesPctAllowed">0.0</double>
<double name="reclaimDeletesWeight">10.0</double>
</mergePolicy>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="keepOptimizedOnly">false</str>
<str name="maxCommitsToKeep">0</str>
</deletionPolicy>
</indexDefaults>
<updateHandler class="solr.DirectUpdateHandler2">
<maxPendingDeletes>1000</maxPendingDeletes>
<autoCommit>
<maxTime>900000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${inventory.solr.softcommit.duration:1000}</maxTime>
</autoSoftCommit>
</updateHandler>
Thanks
Gopal Patwa
*
*
*
*I am using Solr 4.0 nightly build with NRT and I often get this error
during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I
have search this forum and what I found it is related to OS ulimit
setting, please se below my ulimit settings. I am not sure what ulimit
setting I should have? and we also get "**java.net.SocketException:*
*Too* *many* *open* *files" NOT sure how many open file we need to
set?*
I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 -
15GB, with Single shard
*
*
*We update the index every 5 seconds, soft commit every 1 second and
hard commit every 15 minutes*
*
*
*Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB*
*
*
ulimit:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 401408
max locked memory (kbytes, -l) 1024
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 401408
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*
*
*ERROR:*
*
*
*2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=*
*thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto*
*commit* *error...:java.io.IOException:* *Map* *failed*
*at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*)
*at* *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*)
*at* *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*)
*at* *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*)
*at* *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*)
*at* *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*)
*at* *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*)
*at* *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*)
*at* *org.apache.lucene.index.IndexWriter.maybeApplyDeletes*(*IndexWriter.java:2930*)
*at* *org.apache.lucene.index.IndexWriter.prepareCommit*(*IndexWriter.java:2681*)
*at* *org.apache.lucene.index.IndexWriter.commitInternal*(*IndexWriter.java:2804*)
*at* *org.apache.lucene.index.IndexWriter.commit*(*IndexWriter.java:2786*)
*at* *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:391*)
*at* *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*)
*at* *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*)
*at* *java.util.concurrent.FutureTask$Sync.innerRun*(*FutureTask.java:303*)
*at* *java.util.concurrent.FutureTask.run*(*FutureTask.java:138*)
*at* *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301*(*ScheduledThreadPoolExecutor.java:98*)
*at* *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run*(*ScheduledThreadPoolExecutor.java:206*)
*at* *java.util.concurrent.ThreadPoolExecutor$Worker.runTask*(*ThreadPoolExecutor.java:886*)
*at* *java.util.concurrent.ThreadPoolExecutor$Worker.run*(*ThreadPoolExecutor.java:908*)
*at* *java.lang.Thread.run*(*Thread.java:662*)*Caused* *by:*
*java.lang.OutOfMemoryError:* *Map* *failed*
*at* *sun.nio.ch.FileChannelImpl.map0*(*Native* *Method*)
*at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:745*)
*...* *28* *more*
*
*
*
*
*
SolrConfig.xml:
<indexDefaults>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength-->
<ramBufferSizeMB>4096</ramBufferSizeMB>
<maxThreadStates>10</maxThreadStates>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
<lockType>single</lockType>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<double name="forceMergeDeletesPctAllowed">0.0</double>
<double name="reclaimDeletesWeight">10.0</double>
</mergePolicy>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="keepOptimizedOnly">false</str>
<str name="maxCommitsToKeep">0</str>
</deletionPolicy>
</indexDefaults>
<updateHandler class="solr.DirectUpdateHandler2">
<maxPendingDeletes>1000</maxPendingDeletes>
<autoCommit>
<maxTime>900000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${inventory.solr.softcommit.duration:1000}</maxTime>
</autoSoftCommit>
</updateHandler>
Thanks
Gopal Patwa
*
-
Re: Large Index and OutOfMemoryError: Map failedMichael McCandless 2012-03-31, 10:15
It's the virtual memory limit that matters; yours says unlimited below
(good!), but, are you certain that's really the limit your Solr
process runs with?
On Linux, there is also a per-process map count:
cat /proc/sys/vm/max_map_count
I think it typically defaults to 65,536 but you should check on your
env. If a process tries to map more than this many regions, you'll
hit that exception.
I think you can:
cat /proc/<pid>/maps | wc
to see how many maps your Solr process currently has... if that is
anywhere near the limit then it could be the cause.
Mike McCandless
http://blog.mikemccandless.com
On Sat, Mar 31, 2012 at 1:26 AM, Gopal Patwa <[EMAIL PROTECTED]> wrote:
> *I need help!!*
>
> *
> *
>
> *I am using Solr 4.0 nightly build with NRT and I often get this error
> during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I
> have search this forum and what I found it is related to OS ulimit
> setting, please se below my ulimit settings. I am not sure what ulimit
> setting I should have? and we also get "**java.net.SocketException:*
> *Too* *many* *open* *files" NOT sure how many open file we need to
> set?*
>
>
> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 -
> 15GB, with Single shard
>
> *
> *
>
> *We update the index every 5 seconds, soft commit every 1 second and
> hard commit every 15 minutes*
>
> *
> *
>
> *Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB*
>
> *
> *
>
> ulimit:
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 401408
> max locked memory (kbytes, -l) 1024
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 401408
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> *
> *
>
> *ERROR:*
>
> *
> *
>
> *2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=*
> *thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto*
> *commit* *error...:java.io.IOException:* *Map* *failed*
> *at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*)
> *at* *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*)
> *at* *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*)
> *at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*)
> *at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*)
> *at* *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*)
> *at* *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*)
> *at* *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*)
> *at* *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*)
> *at* *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*)
(good!), but, are you certain that's really the limit your Solr
process runs with?
On Linux, there is also a per-process map count:
cat /proc/sys/vm/max_map_count
I think it typically defaults to 65,536 but you should check on your
env. If a process tries to map more than this many regions, you'll
hit that exception.
I think you can:
cat /proc/<pid>/maps | wc
to see how many maps your Solr process currently has... if that is
anywhere near the limit then it could be the cause.
Mike McCandless
http://blog.mikemccandless.com
On Sat, Mar 31, 2012 at 1:26 AM, Gopal Patwa <[EMAIL PROTECTED]> wrote:
> *I need help!!*
>
> *
> *
>
> *I am using Solr 4.0 nightly build with NRT and I often get this error
> during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I
> have search this forum and what I found it is related to OS ulimit
> setting, please se below my ulimit settings. I am not sure what ulimit
> setting I should have? and we also get "**java.net.SocketException:*
> *Too* *many* *open* *files" NOT sure how many open file we need to
> set?*
>
>
> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 -
> 15GB, with Single shard
>
> *
> *
>
> *We update the index every 5 seconds, soft commit every 1 second and
> hard commit every 15 minutes*
>
> *
> *
>
> *Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB*
>
> *
> *
>
> ulimit:
>
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 401408
> max locked memory (kbytes, -l) 1024
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 401408
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> *
> *
>
> *ERROR:*
>
> *
> *
>
> *2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=*
> *thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto*
> *commit* *error...:java.io.IOException:* *Map* *failed*
> *at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*)
> *at* *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*)
> *at* *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*)
> *at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*)
> *at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*)
> *at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*)
> *at* *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*)
> *at* *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*)
> *at* *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*)
> *at* *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*)
> *at* *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*)
-
Re: Large Index and OutOfMemoryError: Map failedGopal Patwa 2012-04-11, 03:35
Michael, Thanks for response
it was 65K as you mention the default value for "cat
/proc/sys/vm/max_map_count" . How we determine what value this should be?
is it number of document during hard commit in my case it is 15 minutes?
or it is number of index file or number of documents we have in all cores.
I have raised the number to 140K but I still get when it reaches to 140K,
we have to restart jboss server to free up the map count, sometime OOM
error happen during "*Error opening new searcher"*
is making this number to unlimited is only solution?''
Error log:
*location=CommitTracker line=93 auto commit
error...:org.apache.solr.common.SolrException: Error opening new
searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:197)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)Caused by:
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:293)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.<init>(PerFieldPostingsFormat.java:262)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$1.<init>(PerFieldPostingsFormat.java:316)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.files(PerFieldPostingsFormat.java:316)
at org.apache.lucene.codecs.Codec.files(Codec.java:56)
at org.apache.lucene.index.SegmentInfo.files(SegmentInfo.java:423)
at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:215)
at org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2220)
at org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:497)
at org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:477)
at org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:201)
at org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119)
at org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:148)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:438)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:553)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:354)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:258)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:243)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:250)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1091)
... 11 moreCaused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)*
And one more issue we came across i.e
On Sat, Mar 31, 2012 at 3:15 AM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> It's the virtual memory limit that matters; yours says unlimited below
> (good!), but, are you certain that's really the limit your Solr
it was 65K as you mention the default value for "cat
/proc/sys/vm/max_map_count" . How we determine what value this should be?
is it number of document during hard commit in my case it is 15 minutes?
or it is number of index file or number of documents we have in all cores.
I have raised the number to 140K but I still get when it reaches to 140K,
we have to restart jboss server to free up the map count, sometime OOM
error happen during "*Error opening new searcher"*
is making this number to unlimited is only solution?''
Error log:
*location=CommitTracker line=93 auto commit
error...:org.apache.solr.common.SolrException: Error opening new
searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:197)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)Caused by:
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:293)
at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.<init>(PerFieldPostingsFormat.java:262)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$1.<init>(PerFieldPostingsFormat.java:316)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.files(PerFieldPostingsFormat.java:316)
at org.apache.lucene.codecs.Codec.files(Codec.java:56)
at org.apache.lucene.index.SegmentInfo.files(SegmentInfo.java:423)
at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:215)
at org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2220)
at org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:497)
at org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:477)
at org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:201)
at org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119)
at org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:148)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:438)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:553)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:354)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:258)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:243)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:250)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1091)
... 11 moreCaused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)*
And one more issue we came across i.e
On Sat, Mar 31, 2012 at 3:15 AM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> It's the virtual memory limit that matters; yours says unlimited below
> (good!), but, are you certain that's really the limit your Solr
-
Re: Large Index and OutOfMemoryError: Map failedMichael McCandless 2012-04-11, 09:20
Hi,
65K is already a very large number and should have been sufficient...
However: have you increased the merge factor? Doing so increases the
open files (maps) required.
Have you disabled compound file format? (Hmmm: I think Solr does so
by default... which is dangerous). Maybe try enabling compound file
format?
Can you "ls -l" your index dir and post the results?
It's also possible Solr isn't closing the old searchers quickly enough
... I don't know the details on when Solr closes old searchers...
Mike McCandless
http://blog.mikemccandless.com
On Tue, Apr 10, 2012 at 11:35 PM, Gopal Patwa <[EMAIL PROTECTED]> wrote:
> Michael, Thanks for response
>
> it was 65K as you mention the default value for "cat
> /proc/sys/vm/max_map_count" . How we determine what value this should be?
> is it number of document during hard commit in my case it is 15 minutes?
> or it is number of index file or number of documents we have in all cores.
>
> I have raised the number to 140K but I still get when it reaches to 140K,
> we have to restart jboss server to free up the map count, sometime OOM
> error happen during "*Error opening new searcher"*
>
> is making this number to unlimited is only solution?''
>
>
> Error log:
>
> *location=CommitTracker line=93 auto commit
> error...:org.apache.solr.common.SolrException: Error opening new
> searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)Caused by:
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:293)
> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.<init>(PerFieldPostingsFormat.java:262)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$1.<init>(PerFieldPostingsFormat.java:316)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.files(PerFieldPostingsFormat.java:316)
> at org.apache.lucene.codecs.Codec.files(Codec.java:56)
> at org.apache.lucene.index.SegmentInfo.files(SegmentInfo.java:423)
> at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:215)
> at org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2220)
> at org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:497)
> at org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:477)
> at org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:201)
> at org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119)
> at org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:148)
> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:438)
> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:553)
65K is already a very large number and should have been sufficient...
However: have you increased the merge factor? Doing so increases the
open files (maps) required.
Have you disabled compound file format? (Hmmm: I think Solr does so
by default... which is dangerous). Maybe try enabling compound file
format?
Can you "ls -l" your index dir and post the results?
It's also possible Solr isn't closing the old searchers quickly enough
... I don't know the details on when Solr closes old searchers...
Mike McCandless
http://blog.mikemccandless.com
On Tue, Apr 10, 2012 at 11:35 PM, Gopal Patwa <[EMAIL PROTECTED]> wrote:
> Michael, Thanks for response
>
> it was 65K as you mention the default value for "cat
> /proc/sys/vm/max_map_count" . How we determine what value this should be?
> is it number of document during hard commit in my case it is 15 minutes?
> or it is number of index file or number of documents we have in all cores.
>
> I have raised the number to 140K but I still get when it reaches to 140K,
> we have to restart jboss server to free up the map count, sometime OOM
> error happen during "*Error opening new searcher"*
>
> is making this number to unlimited is only solution?''
>
>
> Error log:
>
> *location=CommitTracker line=93 auto commit
> error...:org.apache.solr.common.SolrException: Error opening new
> searcher
> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138)
> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251)
> at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409)
> at org.apache.solr.update.CommitTracker.run(CommitTracker.java:197)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)Caused by:
> java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
> at org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:293)
> at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.<init>(PerFieldPostingsFormat.java:262)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$1.<init>(PerFieldPostingsFormat.java:316)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.files(PerFieldPostingsFormat.java:316)
> at org.apache.lucene.codecs.Codec.files(Codec.java:56)
> at org.apache.lucene.index.SegmentInfo.files(SegmentInfo.java:423)
> at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:215)
> at org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2220)
> at org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:497)
> at org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:477)
> at org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:201)
> at org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119)
> at org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:148)
> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:438)
> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:553)
-
Re: Large Index and OutOfMemoryError: Map failedGopal Patwa 2012-04-11, 22:59
I have not change the mergefactor, it was 10. Compound index file is
disable in my config but I read from below post, that some one had similar
issue and it was resolved by switching from compound index file format to
non-compound index file.
and some folks resolved by "changing lucene code to disable MMapDirectory."
Is this best practice to do, if so is this can be done in configuration?
http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
I have index document of core1 = 5 million, core2=8million and
core3=3million and all index are hosted in single Solr instance
I am going to use Solr for our site StubHub.com, see attached "ls -l" list
of index files for all core
*
SolrConfig.xml:
<indexDefaults>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength-->
<ramBufferSizeMB>4096</ramBufferSizeMB>
<maxThreadStates>10</maxThreadStates>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
<lockType>single</lockType>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<double name="forceMergeDeletesPctAllowed">0.0</double>
<double name="reclaimDeletesWeight">10.0</double>
</mergePolicy>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="keepOptimizedOnly">false</str>
<str name="maxCommitsToKeep">0</str>
</deletionPolicy>
</indexDefaults>
<updateHandler class="solr.DirectUpdateHandler2">
<maxPendingDeletes>1000</maxPendingDeletes>
<autoCommit>
<maxTime>900000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${inventory.solr.softcommit.duration:1000}</maxTime>
</autoSoftCommit>
</updateHandler>
*
Forwarded conversation
Subject: Large Index and OutOfMemoryError: Map failed
------------------------
From: *Gopal Patwa* <[EMAIL PROTECTED]>
Date: Fri, Mar 30, 2012 at 10:26 PM
To: [EMAIL PROTECTED]
*I need help!!*
*
*
*I am using Solr 4.0 nightly build with NRT and I often get this error
during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I
have search this forum and what I found it is related to OS ulimit
setting, please se below my ulimit settings. I am not sure what ulimit
setting I should have? and we also get "**java.net.SocketException:*
*Too* *many* *open* *files" NOT sure how many open file we need to
set?*
I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 -
15GB, with Single shard
*
*
*We update the index every 5 seconds, soft commit every 1 second and
hard commit every 15 minutes*
*
*
*Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB*
*
*
ulimit:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 401408
max locked memory (kbytes, -l) 1024
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 401408
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*
*
*ERROR:*
*
*
*2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=*
*thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto*
*commit* *error...:java.io.IOException:* *Map* *failed*
*at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*)
*at* *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*)
*at* *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*)
*at* *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*)
*at* *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*)
*at* *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*)
*at* *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*)
*at* *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*)
*at* *org.apache.lucene.index.IndexWriter.maybeApplyDeletes*(*IndexWriter.java:2930*)
*at* *org.apache.lucene.index.IndexWriter.prepareCommit*(*IndexWriter.java:2681*)
*at* *org.apache.lucene.index.IndexWriter.commitInternal*(*IndexWriter.java:2804*)
*at* *org.apache.lucene.index.IndexWriter.commit*(*IndexWriter.java:2786*)
*at* *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:391*)
*at* *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*)
*at* *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*)
*at*
...
[Message clipped]
From: *Michael McCandless* <[EMAIL PROTECTED]>
Date: Sat, Mar 31, 2012 at 3:15 AM
To: solr-us
disable in my config but I read from below post, that some one had similar
issue and it was resolved by switching from compound index file format to
non-compound index file.
and some folks resolved by "changing lucene code to disable MMapDirectory."
Is this best practice to do, if so is this can be done in configuration?
http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
I have index document of core1 = 5 million, core2=8million and
core3=3million and all index are hosted in single Solr instance
I am going to use Solr for our site StubHub.com, see attached "ls -l" list
of index files for all core
*
SolrConfig.xml:
<indexDefaults>
<useCompoundFile>false</useCompoundFile>
<mergeFactor>10</mergeFactor>
<maxMergeDocs>2147483647</maxMergeDocs>
<maxFieldLength>10000</maxFieldLength-->
<ramBufferSizeMB>4096</ramBufferSizeMB>
<maxThreadStates>10</maxThreadStates>
<writeLockTimeout>1000</writeLockTimeout>
<commitLockTimeout>10000</commitLockTimeout>
<lockType>single</lockType>
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<double name="forceMergeDeletesPctAllowed">0.0</double>
<double name="reclaimDeletesWeight">10.0</double>
</mergePolicy>
<deletionPolicy class="solr.SolrDeletionPolicy">
<str name="keepOptimizedOnly">false</str>
<str name="maxCommitsToKeep">0</str>
</deletionPolicy>
</indexDefaults>
<updateHandler class="solr.DirectUpdateHandler2">
<maxPendingDeletes>1000</maxPendingDeletes>
<autoCommit>
<maxTime>900000</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${inventory.solr.softcommit.duration:1000}</maxTime>
</autoSoftCommit>
</updateHandler>
*
Forwarded conversation
Subject: Large Index and OutOfMemoryError: Map failed
------------------------
From: *Gopal Patwa* <[EMAIL PROTECTED]>
Date: Fri, Mar 30, 2012 at 10:26 PM
To: [EMAIL PROTECTED]
*I need help!!*
*
*
*I am using Solr 4.0 nightly build with NRT and I often get this error
during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I
have search this forum and what I found it is related to OS ulimit
setting, please se below my ulimit settings. I am not sure what ulimit
setting I should have? and we also get "**java.net.SocketException:*
*Too* *many* *open* *files" NOT sure how many open file we need to
set?*
I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 -
15GB, with Single shard
*
*
*We update the index every 5 seconds, soft commit every 1 second and
hard commit every 15 minutes*
*
*
*Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB*
*
*
ulimit:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 401408
max locked memory (kbytes, -l) 1024
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 401408
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*
*
*ERROR:*
*
*
*2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=*
*thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto*
*commit* *error...:java.io.IOException:* *Map* *failed*
*at* *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*)
*at* *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*)
*at* *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*)
*at* *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*)
*at* *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*)
*at* *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*)
*at* *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*)
*at* *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*)
*at* *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*)
*at* *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*)
*at* *org.apache.lucene.index.IndexWriter.maybeApplyDeletes*(*IndexWriter.java:2930*)
*at* *org.apache.lucene.index.IndexWriter.prepareCommit*(*IndexWriter.java:2681*)
*at* *org.apache.lucene.index.IndexWriter.commitInternal*(*IndexWriter.java:2804*)
*at* *org.apache.lucene.index.IndexWriter.commit*(*IndexWriter.java:2786*)
*at* *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:391*)
*at* *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*)
*at* *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*)
*at*
...
[Message clipped]
From: *Michael McCandless* <[EMAIL PROTECTED]>
Date: Sat, Mar 31, 2012 at 3:15 AM
To: solr-us
-
Re: Large Index and OutOfMemoryError: Map failedMichael McCandless 2012-04-12, 10:07
Your largest index has 66 segments (690 files) ... biggish but not
insane. With 64K maps you should be able to have ~47 searchers open
on each core.
Enabling compound file format (not the opposite!) will mean fewer maps
... ie should improve this situation.
I don't understand why Solr defaults to compound file off... that
seems dangerous.
Really we need a Solr dev here... to answer "how long is a stale
searcher kept open". Is it somehow possible 46 old searchers are
being left open...?
I don't see any other reason why you'd run out of maps. Hmm, unless
MMapDirectory didn't think it could safely invoke unmap in your JVM.
Which exact JVM are you using? If you can print the
MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
Yes, switching away from MMapDir will sidestep the "too many maps"
issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
there really is a leak here (Solr not closing the old searchers or a
Lucene bug or something...) then you'll eventually run out of file
descriptors (ie, same problem, different manifestation).
Mike McCandless
http://blog.mikemccandless.com
2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
>
> I have not change the mergefactor, it was 10. Compound index file is disable
> in my config but I read from below post, that some one had similar issue and
> it was resolved by switching from compound index file format to non-compound
> index file.
>
> and some folks resolved by "changing lucene code to disable MMapDirectory."
> Is this best practice to do, if so is this can be done in configuration?
>
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
>
> I have index document of core1 = 5 million, core2=8million and
> core3=3million and all index are hosted in single Solr instance
>
> I am going to use Solr for our site StubHub.com, see attached "ls -l" list
> of index files for all core
>
> SolrConfig.xml:
>
>
> <indexDefaults>
> <useCompoundFile>false</useCompoundFile>
> <mergeFactor>10</mergeFactor>
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength-->
> <ramBufferSizeMB>4096</ramBufferSizeMB>
> <maxThreadStates>10</maxThreadStates>
> <writeLockTimeout>1000</writeLockTimeout>
> <commitLockTimeout>10000</commitLockTimeout>
> <lockType>single</lockType>
>
> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> <double name="forceMergeDeletesPctAllowed">0.0</double>
> <double name="reclaimDeletesWeight">10.0</double>
> </mergePolicy>
>
> <deletionPolicy class="solr.SolrDeletionPolicy">
> <str name="keepOptimizedOnly">false</str>
> <str name="maxCommitsToKeep">0</str>
> </deletionPolicy>
>
> </indexDefaults>
>
>
> <updateHandler class="solr.DirectUpdateHandler2">
> <maxPendingDeletes>1000</maxPendingDeletes>
> <autoCommit>
> <maxTime>900000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>${inventory.solr.softcommit.duration:1000}</maxTime>
> </autoSoftCommit>
>
> </updateHandler>
>
>
> Forwarded conversation
> Subject: Large Index and OutOfMemoryError: Map failed
> ------------------------
>
> From: Gopal Patwa <[EMAIL PROTECTED]>
> Date: Fri, Mar 30, 2012 at 10:26 PM
> To: [EMAIL PROTECTED]
>
>
> I need help!!
>
>
>
>
>
> I am using Solr 4.0 nightly build with NRT and I often get this error during
> auto commit "java.lang.OutOfMemoryError: Map failed". I have search this
> forum and what I found it is related to OS ulimit setting, please se below
> my ulimit settings. I am not sure what ulimit setting I should have? and we
> also get "java.net.SocketException: Too many open files" NOT sure how many
> open file we need to set?
>
>
> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 - 15GB,
> with Single shard
>
>
> We update the index every 5 seconds, soft commit every 1 second and hard
insane. With 64K maps you should be able to have ~47 searchers open
on each core.
Enabling compound file format (not the opposite!) will mean fewer maps
... ie should improve this situation.
I don't understand why Solr defaults to compound file off... that
seems dangerous.
Really we need a Solr dev here... to answer "how long is a stale
searcher kept open". Is it somehow possible 46 old searchers are
being left open...?
I don't see any other reason why you'd run out of maps. Hmm, unless
MMapDirectory didn't think it could safely invoke unmap in your JVM.
Which exact JVM are you using? If you can print the
MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
Yes, switching away from MMapDir will sidestep the "too many maps"
issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
there really is a leak here (Solr not closing the old searchers or a
Lucene bug or something...) then you'll eventually run out of file
descriptors (ie, same problem, different manifestation).
Mike McCandless
http://blog.mikemccandless.com
2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
>
> I have not change the mergefactor, it was 10. Compound index file is disable
> in my config but I read from below post, that some one had similar issue and
> it was resolved by switching from compound index file format to non-compound
> index file.
>
> and some folks resolved by "changing lucene code to disable MMapDirectory."
> Is this best practice to do, if so is this can be done in configuration?
>
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
>
> I have index document of core1 = 5 million, core2=8million and
> core3=3million and all index are hosted in single Solr instance
>
> I am going to use Solr for our site StubHub.com, see attached "ls -l" list
> of index files for all core
>
> SolrConfig.xml:
>
>
> <indexDefaults>
> <useCompoundFile>false</useCompoundFile>
> <mergeFactor>10</mergeFactor>
> <maxMergeDocs>2147483647</maxMergeDocs>
> <maxFieldLength>10000</maxFieldLength-->
> <ramBufferSizeMB>4096</ramBufferSizeMB>
> <maxThreadStates>10</maxThreadStates>
> <writeLockTimeout>1000</writeLockTimeout>
> <commitLockTimeout>10000</commitLockTimeout>
> <lockType>single</lockType>
>
> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> <double name="forceMergeDeletesPctAllowed">0.0</double>
> <double name="reclaimDeletesWeight">10.0</double>
> </mergePolicy>
>
> <deletionPolicy class="solr.SolrDeletionPolicy">
> <str name="keepOptimizedOnly">false</str>
> <str name="maxCommitsToKeep">0</str>
> </deletionPolicy>
>
> </indexDefaults>
>
>
> <updateHandler class="solr.DirectUpdateHandler2">
> <maxPendingDeletes>1000</maxPendingDeletes>
> <autoCommit>
> <maxTime>900000</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>${inventory.solr.softcommit.duration:1000}</maxTime>
> </autoSoftCommit>
>
> </updateHandler>
>
>
> Forwarded conversation
> Subject: Large Index and OutOfMemoryError: Map failed
> ------------------------
>
> From: Gopal Patwa <[EMAIL PROTECTED]>
> Date: Fri, Mar 30, 2012 at 10:26 PM
> To: [EMAIL PROTECTED]
>
>
> I need help!!
>
>
>
>
>
> I am using Solr 4.0 nightly build with NRT and I often get this error during
> auto commit "java.lang.OutOfMemoryError: Map failed". I have search this
> forum and what I found it is related to OS ulimit setting, please se below
> my ulimit settings. I am not sure what ulimit setting I should have? and we
> also get "java.net.SocketException: Too many open files" NOT sure how many
> open file we need to set?
>
>
> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 - 15GB,
> with Single shard
>
>
> We update the index every 5 seconds, soft commit every 1 second and hard
-
Re: Large Index and OutOfMemoryError: Map failedMark Miller 2012-04-12, 16:04
On Apr 12, 2012, at 6:07 AM, Michael McCandless wrote:
> Your largest index has 66 segments (690 files) ... biggish but not
> insane. With 64K maps you should be able to have ~47 searchers open
> on each core.
>
> Enabling compound file format (not the opposite!) will mean fewer maps
> ... ie should improve this situation.
>
> I don't understand why Solr defaults to compound file off... that
> seems dangerous.
>
> Really we need a Solr dev here... to answer "how long is a stale
> searcher kept open". Is it somehow possible 46 old searchers are
> being left open...?
Probably only if there is a bug. When a new Searcher is opened, any previous Searcher is closed as soon as there are no more references to it (eg all in flight requests to that Searcher finish).
>
> I don't see any other reason why you'd run out of maps. Hmm, unless
> MMapDirectory didn't think it could safely invoke unmap in your JVM.
> Which exact JVM are you using? If you can print the
> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>
> Yes, switching away from MMapDir will sidestep the "too many maps"
> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
> there really is a leak here (Solr not closing the old searchers or a
> Lucene bug or something...) then you'll eventually run out of file
> descriptors (ie, same problem, different manifestation).
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
>>
>> I have not change the mergefactor, it was 10. Compound index file is disable
>> in my config but I read from below post, that some one had similar issue and
>> it was resolved by switching from compound index file format to non-compound
>> index file.
>>
>> and some folks resolved by "changing lucene code to disable MMapDirectory."
>> Is this best practice to do, if so is this can be done in configuration?
>>
>> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
>>
>> I have index document of core1 = 5 million, core2=8million and
>> core3=3million and all index are hosted in single Solr instance
>>
>> I am going to use Solr for our site StubHub.com, see attached "ls -l" list
>> of index files for all core
>>
>> SolrConfig.xml:
>>
>>
>> <indexDefaults>
>> <useCompoundFile>false</useCompoundFile>
>> <mergeFactor>10</mergeFactor>
>> <maxMergeDocs>2147483647</maxMergeDocs>
>> <maxFieldLength>10000</maxFieldLength-->
>> <ramBufferSizeMB>4096</ramBufferSizeMB>
>> <maxThreadStates>10</maxThreadStates>
>> <writeLockTimeout>1000</writeLockTimeout>
>> <commitLockTimeout>10000</commitLockTimeout>
>> <lockType>single</lockType>
>>
>> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> <double name="forceMergeDeletesPctAllowed">0.0</double>
>> <double name="reclaimDeletesWeight">10.0</double>
>> </mergePolicy>
>>
>> <deletionPolicy class="solr.SolrDeletionPolicy">
>> <str name="keepOptimizedOnly">false</str>
>> <str name="maxCommitsToKeep">0</str>
>> </deletionPolicy>
>>
>> </indexDefaults>
>>
>>
>> <updateHandler class="solr.DirectUpdateHandler2">
>> <maxPendingDeletes>1000</maxPendingDeletes>
>> <autoCommit>
>> <maxTime>900000</maxTime>
>> <openSearcher>false</openSearcher>
>> </autoCommit>
>> <autoSoftCommit>
>> <maxTime>${inventory.solr.softcommit.duration:1000}</maxTime>
>> </autoSoftCommit>
>>
>> </updateHandler>
>>
>>
>> Forwarded conversation
>> Subject: Large Index and OutOfMemoryError: Map failed
>> ------------------------
>>
>> From: Gopal Patwa <[EMAIL PROTECTED]>
>> Date: Fri, Mar 30, 2012 at 10:26 PM
>> To: [EMAIL PROTECTED]
>>
>>
>> I need help!!
>>
>>
>>
>>
>>
>> I am using Solr 4.0 nightly build with NRT and I often get this error during
>> auto commit "java.lang.OutOfMemoryError: Map failed". I have search this
>> forum and what I found it is related to OS ulimit setting, please se below
- Mark Miller
lucidimagination.com
-
Re: Large Index and OutOfMemoryError: Map failedGopal Patwa 2012-04-14, 16:39
I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
system data. Is their any existing test case to reproduce this issue? I am
trying understand how I can reproduce this issue with unit/integration test
I will try recent solr trunk build too, if it is some bug in solr or
lucene keeping old searcher open then how to reproduce it?
SYSTEM DATA
==========PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
SYSTEM ID: x86_64
CURRENT CPU SPEED: 1600.000 MHz
CPUS: 8 processor(s)
MEMORY: 49449296 kB
DISTRIBUTION: CentOS release 5.3 (Final)
KERNEL NAME: 2.6.18-128.el5
UPTIME: up 71 days
LOAD AVERAGE: 1.42, 1.45, 1.53
JBOSS Version: Implementation-Version: 4.2.2.GA (build:
SVNTag=JBoss_4_2_2_GA date=20
JAVA Version: java version "1.6.0_24"
On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> Your largest index has 66 segments (690 files) ... biggish but not
> insane. With 64K maps you should be able to have ~47 searchers open
> on each core.
>
> Enabling compound file format (not the opposite!) will mean fewer maps
> ... ie should improve this situation.
>
> I don't understand why Solr defaults to compound file off... that
> seems dangerous.
>
> Really we need a Solr dev here... to answer "how long is a stale
> searcher kept open". Is it somehow possible 46 old searchers are
> being left open...?
>
> I don't see any other reason why you'd run out of maps. Hmm, unless
> MMapDirectory didn't think it could safely invoke unmap in your JVM.
> Which exact JVM are you using? If you can print the
> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>
> Yes, switching away from MMapDir will sidestep the "too many maps"
> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
> there really is a leak here (Solr not closing the old searchers or a
> Lucene bug or something...) then you'll eventually run out of file
> descriptors (ie, same problem, different manifestation).
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
> >
> > I have not change the mergefactor, it was 10. Compound index file is
> disable
> > in my config but I read from below post, that some one had similar issue
> and
> > it was resolved by switching from compound index file format to
> non-compound
> > index file.
> >
> > and some folks resolved by "changing lucene code to disable
> MMapDirectory."
> > Is this best practice to do, if so is this can be done in configuration?
> >
> >
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
> >
> > I have index document of core1 = 5 million, core2=8million and
> > core3=3million and all index are hosted in single Solr instance
> >
> > I am going to use Solr for our site StubHub.com, see attached "ls -l"
> list
> > of index files for all core
> >
> > SolrConfig.xml:
> >
> >
> > <indexDefaults>
> > <useCompoundFile>false</useCompoundFile>
> > <mergeFactor>10</mergeFactor>
> > <maxMergeDocs>2147483647</maxMergeDocs>
> > <maxFieldLength>10000</maxFieldLength-->
> > <ramBufferSizeMB>4096</ramBufferSizeMB>
> > <maxThreadStates>10</maxThreadStates>
> > <writeLockTimeout>1000</writeLockTimeout>
> > <commitLockTimeout>10000</commitLockTimeout>
> > <lockType>single</lockType>
> >
> > <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> > <double name="forceMergeDeletesPctAllowed">0.0</double>
> > <double name="reclaimDeletesWeight">10.0</double>
> > </mergePolicy>
> >
> > <deletionPolicy class="solr.SolrDeletionPolicy">
> > <str name="keepOptimizedOnly">false</str>
> > <str name="maxCommitsToKeep">0</str>
> > </deletionPolicy>
> >
> > </indexDefaults>
> >
> >
> > <updateHandler class="solr.DirectUpdateHandler2"
system data. Is their any existing test case to reproduce this issue? I am
trying understand how I can reproduce this issue with unit/integration test
I will try recent solr trunk build too, if it is some bug in solr or
lucene keeping old searcher open then how to reproduce it?
SYSTEM DATA
==========PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
SYSTEM ID: x86_64
CURRENT CPU SPEED: 1600.000 MHz
CPUS: 8 processor(s)
MEMORY: 49449296 kB
DISTRIBUTION: CentOS release 5.3 (Final)
KERNEL NAME: 2.6.18-128.el5
UPTIME: up 71 days
LOAD AVERAGE: 1.42, 1.45, 1.53
JBOSS Version: Implementation-Version: 4.2.2.GA (build:
SVNTag=JBoss_4_2_2_GA date=20
JAVA Version: java version "1.6.0_24"
On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
[EMAIL PROTECTED]> wrote:
> Your largest index has 66 segments (690 files) ... biggish but not
> insane. With 64K maps you should be able to have ~47 searchers open
> on each core.
>
> Enabling compound file format (not the opposite!) will mean fewer maps
> ... ie should improve this situation.
>
> I don't understand why Solr defaults to compound file off... that
> seems dangerous.
>
> Really we need a Solr dev here... to answer "how long is a stale
> searcher kept open". Is it somehow possible 46 old searchers are
> being left open...?
>
> I don't see any other reason why you'd run out of maps. Hmm, unless
> MMapDirectory didn't think it could safely invoke unmap in your JVM.
> Which exact JVM are you using? If you can print the
> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>
> Yes, switching away from MMapDir will sidestep the "too many maps"
> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
> there really is a leak here (Solr not closing the old searchers or a
> Lucene bug or something...) then you'll eventually run out of file
> descriptors (ie, same problem, different manifestation).
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
> >
> > I have not change the mergefactor, it was 10. Compound index file is
> disable
> > in my config but I read from below post, that some one had similar issue
> and
> > it was resolved by switching from compound index file format to
> non-compound
> > index file.
> >
> > and some folks resolved by "changing lucene code to disable
> MMapDirectory."
> > Is this best practice to do, if so is this can be done in configuration?
> >
> >
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
> >
> > I have index document of core1 = 5 million, core2=8million and
> > core3=3million and all index are hosted in single Solr instance
> >
> > I am going to use Solr for our site StubHub.com, see attached "ls -l"
> list
> > of index files for all core
> >
> > SolrConfig.xml:
> >
> >
> > <indexDefaults>
> > <useCompoundFile>false</useCompoundFile>
> > <mergeFactor>10</mergeFactor>
> > <maxMergeDocs>2147483647</maxMergeDocs>
> > <maxFieldLength>10000</maxFieldLength-->
> > <ramBufferSizeMB>4096</ramBufferSizeMB>
> > <maxThreadStates>10</maxThreadStates>
> > <writeLockTimeout>1000</writeLockTimeout>
> > <commitLockTimeout>10000</commitLockTimeout>
> > <lockType>single</lockType>
> >
> > <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> > <double name="forceMergeDeletesPctAllowed">0.0</double>
> > <double name="reclaimDeletesWeight">10.0</double>
> > </mergePolicy>
> >
> > <deletionPolicy class="solr.SolrDeletionPolicy">
> > <str name="keepOptimizedOnly">false</str>
> > <str name="maxCommitsToKeep">0</str>
> > </deletionPolicy>
> >
> > </indexDefaults>
> >
> >
> > <updateHandler class="solr.DirectUpdateHandler2"
-
Re: Large Index and OutOfMemoryError: Map failedBoon Low 2012-04-19, 14:42
Hi,
Also came across this error recently, while indexing with > 10 DIH processes in parallel + default index setting. The JVM grinds to a halt and throws this error. Checking the index of a core reveals thousands of files! Tuning the default autocommit from 15000ms to 900000ms solved the problem for us. (no 'autosoftcommit').
Boon
-----
Boon Low
Search UX and Engine Developer
brightsolid Online Publishing
On 14 Apr 2012, at 17:40, Gopal Patwa wrote:
> I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
> system data. Is their any existing test case to reproduce this issue? I am
> trying understand how I can reproduce this issue with unit/integration test
>
> I will try recent solr trunk build too, if it is some bug in solr or
> lucene keeping old searcher open then how to reproduce it?
>
> SYSTEM DATA
> ==========> PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
> SYSTEM ID: x86_64
> CURRENT CPU SPEED: 1600.000 MHz
> CPUS: 8 processor(s)
> MEMORY: 49449296 kB
> DISTRIBUTION: CentOS release 5.3 (Final)
> KERNEL NAME: 2.6.18-128.el5
> UPTIME: up 71 days
> LOAD AVERAGE: 1.42, 1.45, 1.53
> JBOSS Version: Implementation-Version: 4.2.2.GA (build:
> SVNTag=JBoss_4_2_2_GA date=20
> JAVA Version: java version "1.6.0_24"
>
>
> On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
> [EMAIL PROTECTED]> wrote:
>
>> Your largest index has 66 segments (690 files) ... biggish but not
>> insane. With 64K maps you should be able to have ~47 searchers open
>> on each core.
>>
>> Enabling compound file format (not the opposite!) will mean fewer maps
>> ... ie should improve this situation.
>>
>> I don't understand why Solr defaults to compound file off... that
>> seems dangerous.
>>
>> Really we need a Solr dev here... to answer "how long is a stale
>> searcher kept open". Is it somehow possible 46 old searchers are
>> being left open...?
>>
>> I don't see any other reason why you'd run out of maps. Hmm, unless
>> MMapDirectory didn't think it could safely invoke unmap in your JVM.
>> Which exact JVM are you using? If you can print the
>> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>>
>> Yes, switching away from MMapDir will sidestep the "too many maps"
>> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
>> there really is a leak here (Solr not closing the old searchers or a
>> Lucene bug or something...) then you'll eventually run out of file
>> descriptors (ie, same problem, different manifestation).
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
>>>
>>> I have not change the mergefactor, it was 10. Compound index file is
>> disable
>>> in my config but I read from below post, that some one had similar issue
>> and
>>> it was resolved by switching from compound index file format to
>> non-compound
>>> index file.
>>>
>>> and some folks resolved by "changing lucene code to disable
>> MMapDirectory."
>>> Is this best practice to do, if so is this can be done in configuration?
>>>
>>>
>> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
>>>
>>> I have index document of core1 = 5 million, core2=8million and
>>> core3=3million and all index are hosted in single Solr instance
>>>
>>> I am going to use Solr for our site StubHub.com, see attached "ls -l"
>> list
>>> of index files for all core
>>>
>>> SolrConfig.xml:
>>>
>>>
>>> <indexDefaults>
>>> <useCompoundFile>false</useCompoundFile>
>>> <mergeFactor>10</mergeFactor>
>>> <maxMergeDocs>2147483647</maxMergeDocs>
>>> <maxFieldLength>10000</maxFieldLength-->
>>> <ramBufferSizeMB>4096</ramBufferSizeMB>
>>> <maxThreadStates>10</maxThreadStates>
>>> <writeLockTimeout>1000</writeLockTimeout>
>>> <commitLockTimeout>10000</commitLockTimeout>
>>> <lockType>single</lockType>
>
______________________________________________________________________
"brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
findmypast.co.uk is a brand of brightsolid online publishing limited.
brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
Email Disclaimer
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
______________________________________________________________________
Also came across this error recently, while indexing with > 10 DIH processes in parallel + default index setting. The JVM grinds to a halt and throws this error. Checking the index of a core reveals thousands of files! Tuning the default autocommit from 15000ms to 900000ms solved the problem for us. (no 'autosoftcommit').
Boon
-----
Boon Low
Search UX and Engine Developer
brightsolid Online Publishing
On 14 Apr 2012, at 17:40, Gopal Patwa wrote:
> I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
> system data. Is their any existing test case to reproduce this issue? I am
> trying understand how I can reproduce this issue with unit/integration test
>
> I will try recent solr trunk build too, if it is some bug in solr or
> lucene keeping old searcher open then how to reproduce it?
>
> SYSTEM DATA
> ==========> PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
> SYSTEM ID: x86_64
> CURRENT CPU SPEED: 1600.000 MHz
> CPUS: 8 processor(s)
> MEMORY: 49449296 kB
> DISTRIBUTION: CentOS release 5.3 (Final)
> KERNEL NAME: 2.6.18-128.el5
> UPTIME: up 71 days
> LOAD AVERAGE: 1.42, 1.45, 1.53
> JBOSS Version: Implementation-Version: 4.2.2.GA (build:
> SVNTag=JBoss_4_2_2_GA date=20
> JAVA Version: java version "1.6.0_24"
>
>
> On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
> [EMAIL PROTECTED]> wrote:
>
>> Your largest index has 66 segments (690 files) ... biggish but not
>> insane. With 64K maps you should be able to have ~47 searchers open
>> on each core.
>>
>> Enabling compound file format (not the opposite!) will mean fewer maps
>> ... ie should improve this situation.
>>
>> I don't understand why Solr defaults to compound file off... that
>> seems dangerous.
>>
>> Really we need a Solr dev here... to answer "how long is a stale
>> searcher kept open". Is it somehow possible 46 old searchers are
>> being left open...?
>>
>> I don't see any other reason why you'd run out of maps. Hmm, unless
>> MMapDirectory didn't think it could safely invoke unmap in your JVM.
>> Which exact JVM are you using? If you can print the
>> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>>
>> Yes, switching away from MMapDir will sidestep the "too many maps"
>> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
>> there really is a leak here (Solr not closing the old searchers or a
>> Lucene bug or something...) then you'll eventually run out of file
>> descriptors (ie, same problem, different manifestation).
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
>>>
>>> I have not change the mergefactor, it was 10. Compound index file is
>> disable
>>> in my config but I read from below post, that some one had similar issue
>> and
>>> it was resolved by switching from compound index file format to
>> non-compound
>>> index file.
>>>
>>> and some folks resolved by "changing lucene code to disable
>> MMapDirectory."
>>> Is this best practice to do, if so is this can be done in configuration?
>>>
>>>
>> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
>>>
>>> I have index document of core1 = 5 million, core2=8million and
>>> core3=3million and all index are hosted in single Solr instance
>>>
>>> I am going to use Solr for our site StubHub.com, see attached "ls -l"
>> list
>>> of index files for all core
>>>
>>> SolrConfig.xml:
>>>
>>>
>>> <indexDefaults>
>>> <useCompoundFile>false</useCompoundFile>
>>> <mergeFactor>10</mergeFactor>
>>> <maxMergeDocs>2147483647</maxMergeDocs>
>>> <maxFieldLength>10000</maxFieldLength-->
>>> <ramBufferSizeMB>4096</ramBufferSizeMB>
>>> <maxThreadStates>10</maxThreadStates>
>>> <writeLockTimeout>1000</writeLockTimeout>
>>> <commitLockTimeout>10000</commitLockTimeout>
>>> <lockType>single</lockType>
>
______________________________________________________________________
"brightsolid" is used in this email to collectively mean brightsolid online innovation limited and its subsidiary companies brightsolid online publishing limited and brightsolid online technology limited.
findmypast.co.uk is a brand of brightsolid online publishing limited.
brightsolid online innovation limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983.
brightsolid online publishing limited, The Glebe, 6 Chapel Place, Rivington Street, London EC2A 3DQ. Registered in England No. 04369607.
brightsolid online technology limited, Gateway House, Luna Place, Dundee Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678.
Email Disclaimer
This message is confidential and may contain privileged information. You should not disclose its contents to any other person. If you are not the intended recipient, please notify the sender named above immediately. It is expressly declared that this e-mail does not constitute nor form part of a contract or unilateral obligation. Opinions, conclusions and other information in this message that do not relate to the official business of brightsolid shall be understood as neither given nor endorsed by it.
______________________________________________________________________
This email has been scanned by the brightsolid Email Security System. Powered by MessageLabs
______________________________________________________________________
-
Re: Large Index and OutOfMemoryError: Map failedGopal Patwa 2012-04-20, 13:33
We cannot avoid auto soft commit, since we need Lucene NRT feature. And I
use StreamingUpdateSolrServer for adding/updating index.
On Thu, Apr 19, 2012 at 7:42 AM, Boon Low <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Also came across this error recently, while indexing with > 10 DIH
> processes in parallel + default index setting. The JVM grinds to a halt and
> throws this error. Checking the index of a core reveals thousands of files!
> Tuning the default autocommit from 15000ms to 900000ms solved the problem
> for us. (no 'autosoftcommit').
>
> Boon
>
> -----
> Boon Low
> Search UX and Engine Developer
> brightsolid Online Publishing
>
> On 14 Apr 2012, at 17:40, Gopal Patwa wrote:
>
> > I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
> > system data. Is their any existing test case to reproduce this issue? I
> am
> > trying understand how I can reproduce this issue with unit/integration
> test
> >
> > I will try recent solr trunk build too, if it is some bug in solr or
> > lucene keeping old searcher open then how to reproduce it?
> >
> > SYSTEM DATA
> > ==========> > PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
> > SYSTEM ID: x86_64
> > CURRENT CPU SPEED: 1600.000 MHz
> > CPUS: 8 processor(s)
> > MEMORY: 49449296 kB
> > DISTRIBUTION: CentOS release 5.3 (Final)
> > KERNEL NAME: 2.6.18-128.el5
> > UPTIME: up 71 days
> > LOAD AVERAGE: 1.42, 1.45, 1.53
> > JBOSS Version: Implementation-Version: 4.2.2.GA (build:
> > SVNTag=JBoss_4_2_2_GA date=20
> > JAVA Version: java version "1.6.0_24"
> >
> >
> > On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Your largest index has 66 segments (690 files) ... biggish but not
> >> insane. With 64K maps you should be able to have ~47 searchers open
> >> on each core.
> >>
> >> Enabling compound file format (not the opposite!) will mean fewer maps
> >> ... ie should improve this situation.
> >>
> >> I don't understand why Solr defaults to compound file off... that
> >> seems dangerous.
> >>
> >> Really we need a Solr dev here... to answer "how long is a stale
> >> searcher kept open". Is it somehow possible 46 old searchers are
> >> being left open...?
> >>
> >> I don't see any other reason why you'd run out of maps. Hmm, unless
> >> MMapDirectory didn't think it could safely invoke unmap in your JVM.
> >> Which exact JVM are you using? If you can print the
> >> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
> >>
> >> Yes, switching away from MMapDir will sidestep the "too many maps"
> >> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
> >> there really is a leak here (Solr not closing the old searchers or a
> >> Lucene bug or something...) then you'll eventually run out of file
> >> descriptors (ie, same problem, different manifestation).
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
> >>>
> >>> I have not change the mergefactor, it was 10. Compound index file is
> >> disable
> >>> in my config but I read from below post, that some one had similar
> issue
> >> and
> >>> it was resolved by switching from compound index file format to
> >> non-compound
> >>> index file.
> >>>
> >>> and some folks resolved by "changing lucene code to disable
> >> MMapDirectory."
> >>> Is this best practice to do, if so is this can be done in
> configuration?
> >>>
> >>>
> >>
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
> >>>
> >>> I have index document of core1 = 5 million, core2=8million and
> >>> core3=3million and all index are hosted in single Solr instance
> >>>
> >>> I am going to use Solr for our site StubHub.com, see attached "ls -l"
> >> list
> >>> of index files for all core
> >>>
> >>> SolrConfig.xml:
> >>>
> >>>
> >>> <indexDefaults>
> >>> <useCompoundFile>false</useCompoundFile>
> >>> <mergeFactor>10</mergeFactor>
use StreamingUpdateSolrServer for adding/updating index.
On Thu, Apr 19, 2012 at 7:42 AM, Boon Low <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Also came across this error recently, while indexing with > 10 DIH
> processes in parallel + default index setting. The JVM grinds to a halt and
> throws this error. Checking the index of a core reveals thousands of files!
> Tuning the default autocommit from 15000ms to 900000ms solved the problem
> for us. (no 'autosoftcommit').
>
> Boon
>
> -----
> Boon Low
> Search UX and Engine Developer
> brightsolid Online Publishing
>
> On 14 Apr 2012, at 17:40, Gopal Patwa wrote:
>
> > I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
> > system data. Is their any existing test case to reproduce this issue? I
> am
> > trying understand how I can reproduce this issue with unit/integration
> test
> >
> > I will try recent solr trunk build too, if it is some bug in solr or
> > lucene keeping old searcher open then how to reproduce it?
> >
> > SYSTEM DATA
> > ==========> > PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
> > SYSTEM ID: x86_64
> > CURRENT CPU SPEED: 1600.000 MHz
> > CPUS: 8 processor(s)
> > MEMORY: 49449296 kB
> > DISTRIBUTION: CentOS release 5.3 (Final)
> > KERNEL NAME: 2.6.18-128.el5
> > UPTIME: up 71 days
> > LOAD AVERAGE: 1.42, 1.45, 1.53
> > JBOSS Version: Implementation-Version: 4.2.2.GA (build:
> > SVNTag=JBoss_4_2_2_GA date=20
> > JAVA Version: java version "1.6.0_24"
> >
> >
> > On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Your largest index has 66 segments (690 files) ... biggish but not
> >> insane. With 64K maps you should be able to have ~47 searchers open
> >> on each core.
> >>
> >> Enabling compound file format (not the opposite!) will mean fewer maps
> >> ... ie should improve this situation.
> >>
> >> I don't understand why Solr defaults to compound file off... that
> >> seems dangerous.
> >>
> >> Really we need a Solr dev here... to answer "how long is a stale
> >> searcher kept open". Is it somehow possible 46 old searchers are
> >> being left open...?
> >>
> >> I don't see any other reason why you'd run out of maps. Hmm, unless
> >> MMapDirectory didn't think it could safely invoke unmap in your JVM.
> >> Which exact JVM are you using? If you can print the
> >> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
> >>
> >> Yes, switching away from MMapDir will sidestep the "too many maps"
> >> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
> >> there really is a leak here (Solr not closing the old searchers or a
> >> Lucene bug or something...) then you'll eventually run out of file
> >> descriptors (ie, same problem, different manifestation).
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
> >>>
> >>> I have not change the mergefactor, it was 10. Compound index file is
> >> disable
> >>> in my config but I read from below post, that some one had similar
> issue
> >> and
> >>> it was resolved by switching from compound index file format to
> >> non-compound
> >>> index file.
> >>>
> >>> and some folks resolved by "changing lucene code to disable
> >> MMapDirectory."
> >>> Is this best practice to do, if so is this can be done in
> configuration?
> >>>
> >>>
> >>
> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
> >>>
> >>> I have index document of core1 = 5 million, core2=8million and
> >>> core3=3million and all index are hosted in single Solr instance
> >>>
> >>> I am going to use Solr for our site StubHub.com, see attached "ls -l"
> >> list
> >>> of index files for all core
> >>>
> >>> SolrConfig.xml:
> >>>
> >>>
> >>> <indexDefaults>
> >>> <useCompoundFile>false</useCompoundFile>
> >>> <mergeFactor>10</mergeFactor>
-
Re: Large Index and OutOfMemoryError: Map failedMichael McCandless 2012-04-22, 12:29
Is it possible you are hitting this (just opened) Solr issue?:
https://issues.apache.org/jira/browse/SOLR-3392
Mike McCandless
http://blog.mikemccandless.com
On Fri, Apr 20, 2012 at 9:33 AM, Gopal Patwa <[EMAIL PROTECTED]> wrote:
> We cannot avoid auto soft commit, since we need Lucene NRT feature. And I
> use StreamingUpdateSolrServer for adding/updating index.
>
> On Thu, Apr 19, 2012 at 7:42 AM, Boon Low <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> Also came across this error recently, while indexing with > 10 DIH
>> processes in parallel + default index setting. The JVM grinds to a halt and
>> throws this error. Checking the index of a core reveals thousands of files!
>> Tuning the default autocommit from 15000ms to 900000ms solved the problem
>> for us. (no 'autosoftcommit').
>>
>> Boon
>>
>> -----
>> Boon Low
>> Search UX and Engine Developer
>> brightsolid Online Publishing
>>
>> On 14 Apr 2012, at 17:40, Gopal Patwa wrote:
>>
>> > I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
>> > system data. Is their any existing test case to reproduce this issue? I
>> am
>> > trying understand how I can reproduce this issue with unit/integration
>> test
>> >
>> > I will try recent solr trunk build too, if it is some bug in solr or
>> > lucene keeping old searcher open then how to reproduce it?
>> >
>> > SYSTEM DATA
>> > ==========>> > PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
>> > SYSTEM ID: x86_64
>> > CURRENT CPU SPEED: 1600.000 MHz
>> > CPUS: 8 processor(s)
>> > MEMORY: 49449296 kB
>> > DISTRIBUTION: CentOS release 5.3 (Final)
>> > KERNEL NAME: 2.6.18-128.el5
>> > UPTIME: up 71 days
>> > LOAD AVERAGE: 1.42, 1.45, 1.53
>> > JBOSS Version: Implementation-Version: 4.2.2.GA (build:
>> > SVNTag=JBoss_4_2_2_GA date=20
>> > JAVA Version: java version "1.6.0_24"
>> >
>> >
>> > On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> Your largest index has 66 segments (690 files) ... biggish but not
>> >> insane. With 64K maps you should be able to have ~47 searchers open
>> >> on each core.
>> >>
>> >> Enabling compound file format (not the opposite!) will mean fewer maps
>> >> ... ie should improve this situation.
>> >>
>> >> I don't understand why Solr defaults to compound file off... that
>> >> seems dangerous.
>> >>
>> >> Really we need a Solr dev here... to answer "how long is a stale
>> >> searcher kept open". Is it somehow possible 46 old searchers are
>> >> being left open...?
>> >>
>> >> I don't see any other reason why you'd run out of maps. Hmm, unless
>> >> MMapDirectory didn't think it could safely invoke unmap in your JVM.
>> >> Which exact JVM are you using? If you can print the
>> >> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>> >>
>> >> Yes, switching away from MMapDir will sidestep the "too many maps"
>> >> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
>> >> there really is a leak here (Solr not closing the old searchers or a
>> >> Lucene bug or something...) then you'll eventually run out of file
>> >> descriptors (ie, same problem, different manifestation).
>> >>
>> >> Mike McCandless
>> >>
>> >> http://blog.mikemccandless.com
>> >>
>> >> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
>> >>>
>> >>> I have not change the mergefactor, it was 10. Compound index file is
>> >> disable
>> >>> in my config but I read from below post, that some one had similar
>> issue
>> >> and
>> >>> it was resolved by switching from compound index file format to
>> >> non-compound
>> >>> index file.
>> >>>
>> >>> and some folks resolved by "changing lucene code to disable
>> >> MMapDirectory."
>> >>> Is this best practice to do, if so is this can be done in
>> configuration?
>> >>>
>> >>>
>> >>
>> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
>> >>>
>> >>> I have index document of core1 = 5 million, core2=8million and
>> >>> core3=3million and all index are hosted in single Solr instance
https://issues.apache.org/jira/browse/SOLR-3392
Mike McCandless
http://blog.mikemccandless.com
On Fri, Apr 20, 2012 at 9:33 AM, Gopal Patwa <[EMAIL PROTECTED]> wrote:
> We cannot avoid auto soft commit, since we need Lucene NRT feature. And I
> use StreamingUpdateSolrServer for adding/updating index.
>
> On Thu, Apr 19, 2012 at 7:42 AM, Boon Low <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> Also came across this error recently, while indexing with > 10 DIH
>> processes in parallel + default index setting. The JVM grinds to a halt and
>> throws this error. Checking the index of a core reveals thousands of files!
>> Tuning the default autocommit from 15000ms to 900000ms solved the problem
>> for us. (no 'autosoftcommit').
>>
>> Boon
>>
>> -----
>> Boon Low
>> Search UX and Engine Developer
>> brightsolid Online Publishing
>>
>> On 14 Apr 2012, at 17:40, Gopal Patwa wrote:
>>
>> > I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my
>> > system data. Is their any existing test case to reproduce this issue? I
>> am
>> > trying understand how I can reproduce this issue with unit/integration
>> test
>> >
>> > I will try recent solr trunk build too, if it is some bug in solr or
>> > lucene keeping old searcher open then how to reproduce it?
>> >
>> > SYSTEM DATA
>> > ==========>> > PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
>> > SYSTEM ID: x86_64
>> > CURRENT CPU SPEED: 1600.000 MHz
>> > CPUS: 8 processor(s)
>> > MEMORY: 49449296 kB
>> > DISTRIBUTION: CentOS release 5.3 (Final)
>> > KERNEL NAME: 2.6.18-128.el5
>> > UPTIME: up 71 days
>> > LOAD AVERAGE: 1.42, 1.45, 1.53
>> > JBOSS Version: Implementation-Version: 4.2.2.GA (build:
>> > SVNTag=JBoss_4_2_2_GA date=20
>> > JAVA Version: java version "1.6.0_24"
>> >
>> >
>> > On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> Your largest index has 66 segments (690 files) ... biggish but not
>> >> insane. With 64K maps you should be able to have ~47 searchers open
>> >> on each core.
>> >>
>> >> Enabling compound file format (not the opposite!) will mean fewer maps
>> >> ... ie should improve this situation.
>> >>
>> >> I don't understand why Solr defaults to compound file off... that
>> >> seems dangerous.
>> >>
>> >> Really we need a Solr dev here... to answer "how long is a stale
>> >> searcher kept open". Is it somehow possible 46 old searchers are
>> >> being left open...?
>> >>
>> >> I don't see any other reason why you'd run out of maps. Hmm, unless
>> >> MMapDirectory didn't think it could safely invoke unmap in your JVM.
>> >> Which exact JVM are you using? If you can print the
>> >> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure.
>> >>
>> >> Yes, switching away from MMapDir will sidestep the "too many maps"
>> >> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if
>> >> there really is a leak here (Solr not closing the old searchers or a
>> >> Lucene bug or something...) then you'll eventually run out of file
>> >> descriptors (ie, same problem, different manifestation).
>> >>
>> >> Mike McCandless
>> >>
>> >> http://blog.mikemccandless.com
>> >>
>> >> 2012/4/11 Gopal Patwa <[EMAIL PROTECTED]>:
>> >>>
>> >>> I have not change the mergefactor, it was 10. Compound index file is
>> >> disable
>> >>> in my config but I read from below post, that some one had similar
>> issue
>> >> and
>> >>> it was resolved by switching from compound index file format to
>> >> non-compound
>> >>> index file.
>> >>>
>> >>> and some folks resolved by "changing lucene code to disable
>> >> MMapDirectory."
>> >>> Is this best practice to do, if so is this can be done in
>> configuration?
>> >>>
>> >>>
>> >>
>> http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html
>> >>>
>> >>> I have index document of core1 = 5 million, core2=8million and
>> >>> core3=3million and all index are hosted in single Solr instance
