Hi folks,
my Test-Server with Solr 4.0 from trunk(version 1292064 from late
february) throws this exception...
auto commit error...:java.lang.IllegalStateException: this writer hit
an OutOfMemoryError; cannot commit
        at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2650)
        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2804)
        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2786)
        at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:391)
        at org.apache.solr.update.CommitTracker.run(CommitTracker.java:197)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
My Server has 24GB RAM, 8GB for JVM. I index round about 20 docs per
seconds, my index is small with 10Mio docs. It runs
about a couple of weeks and then suddenly i get this errors..
I can't see any problems in VisualVM with my GC. It's all ok, memory
consumption is about 6GB, no swapping, no i/o problems..it's all
green:)
What's going on on this machine?:)  My uncommitted docs are gone, right?

Best regards
Vadim

On Tue, Jul 10, 2012 at 3:11 AM, Vadim Kisselmann
<[EMAIL PROTECTED]> wrote:
> Hi folks,
> my Test-Server with Solr 4.0 from trunk(version 1292064 from late
> february) throws this exception...

Can you run Lucene's checkIndex tool on your index?

If that is clean, can you try a newer version?
This could be a number of things, including something already fixed.

>
>
> auto commit error...:java.lang.IllegalStateException: this writer hit
> an OutOfMemoryError; cannot commit
>         at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2650)
>         at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2804)
>         at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2786)
>         at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:391)
>         at org.apache.solr.update.CommitTracker.run(CommitTracker.java:197)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>

do you have another exception in your logs? To my knowledge, in all
cases that IndexWriter throws an OutOfMemoryError, the original
OutOfMemoryError is also rethrown (not just this IllegalStateException
noting that at some point, it hit OOM.

>
> My Server has 24GB RAM, 8GB for JVM. I index round about 20 docs per
> seconds, my index is small with 10Mio docs. It runs
> about a couple of weeks and then suddenly i get this errors..
> I can't see any problems in VisualVM with my GC. It's all ok, memory
> consumption is about 6GB, no swapping, no i/o problems..it's all
> green:)
> What's going on on this machine?:)  My uncommitted docs are gone, right?

Yes, your commit failed.

--
lucidimagination.com

Hi Robert,

> Can you run Lucene's checkIndex tool on your index?

No, unfortunately not. This Solr should run without stoppage, an
tomcat-restart is ok, but not more:)
I tested newer trunk-versions a couple of months ago, but they fail
all with tomcat.
i would test 4.0-alpha in next days with tomcat and open an jira-issue
if it doesn't work with it.

> do you have another exception in your logs? To my knowledge, in all
> cases that IndexWriter throws an OutOfMemoryError, the original
> OutOfMemoryError is also rethrown (not just this IllegalStateException
> noting that at some point, it hit OOM.

Hmm, i checked older logs and found something new, what i have not
seen in VisualVM. "Java heap space"-Problems, just before OOM.
My JVM has 8GB -Xmx/-Xms, 16GB for OS, nothing else on this machine.
This Errors pop up's during normal run according logs, no optimizes,
high loads(max. 30 queries per minute) or something special at this time.

SCHWERWIEGEND: null:ClientAbortException:  java.net.SocketException: Broken pipe
SCHWERWIEGEND: null:java.lang.OutOfMemoryError: Java heap space
SCHWERWIEGEND: auto commit error...:java.lang.IllegalStateException:
this writer hit an OutOfMemoryError; cannot commit
SCHWERWIEGEND: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@7cba935e:java.lang.OutOfMemoryError:
Java heap space
SCHWERWIEGEND: org.apache.solr.common.SolrException: Internal Server Error
SCHWERWIEGEND: null:org.apache.solr.common.SolrException: Internal Server Error

I knew this failures when i work on virtual machines with solr 1.4,
big indexes and ridiculous small -Xmx sizes.
But on real hardware, with enough RAM, fast disks/cpu's it's new for me:)

Best regards
Vadim

it really seems that you are hitting an OOM during auto warming. can
this be the case for your failure.
Can you raise the JVM memory and see if you still hit the spike and go
OOM? this is very unlikely a IndexWriter problem. I'd rather look at
your warmup queries ie. fieldcache, FieldValueCache usage. Are you
sorting / facet on anything?

simon

On Tue, Jul 10, 2012 at 4:49 PM, Vadim Kisselmann
<[EMAIL PROTECTED]> wrote:
> Hi Robert,
>
>> Can you run Lucene's checkIndex tool on your index?
>
> No, unfortunately not. This Solr should run without stoppage, an
> tomcat-restart is ok, but not more:)
> I tested newer trunk-versions a couple of months ago, but they fail
> all with tomcat.
> i would test 4.0-alpha in next days with tomcat and open an jira-issue
> if it doesn't work with it.
>
>> do you have another exception in your logs? To my knowledge, in all
>> cases that IndexWriter throws an OutOfMemoryError, the original
>> OutOfMemoryError is also rethrown (not just this IllegalStateException
>> noting that at some point, it hit OOM.
>
> Hmm, i checked older logs and found something new, what i have not
> seen in VisualVM. "Java heap space"-Problems, just before OOM.
> My JVM has 8GB -Xmx/-Xms, 16GB for OS, nothing else on this machine.
> This Errors pop up's during normal run according logs, no optimizes,
> high loads(max. 30 queries per minute) or something special at this time.
>
> SCHWERWIEGEND: null:ClientAbortException:  java.net.SocketException: Broken pipe
> SCHWERWIEGEND: null:java.lang.OutOfMemoryError: Java heap space
> SCHWERWIEGEND: auto commit error...:java.lang.IllegalStateException:
> this writer hit an OutOfMemoryError; cannot commit
> SCHWERWIEGEND: Error during auto-warming of
> key:org.apache.solr.search.QueryResultKey@7cba935e:java.lang.OutOfMemoryError:
> Java heap space
> SCHWERWIEGEND: org.apache.solr.common.SolrException: Internal Server Error
> SCHWERWIEGEND: null:org.apache.solr.common.SolrException: Internal Server Error
>
> I knew this failures when i work on virtual machines with solr 1.4,
> big indexes and ridiculous small -Xmx sizes.
> But on real hardware, with enough RAM, fast disks/cpu's it's new for me:)
>
> Best regards
> Vadim

Hi Simon,
i checked my log files one more time to get the error timestamps.
I get the first Error at 14:37:

06.07.2012 14:37:52 org.apache.solr.common.SolrException log
SCHWERWIEGEND: null:ClientAbortException:  java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:323)

Next one, and the first Java heap Space error at 17:35:
06.07.2012 17:35:36 org.apache.solr.common.SolrException log
SCHWERWIEGEND: null:java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.<init>(FreqProxTermsWriterPerField.java:248)
at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:269)
at org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)
at org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray.grow(TermsHashPerField.java:307)
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:330)

Commit failure a couple of seconds later:
06.07.2012 17:35:38 org.apache.solr.common.SolrException log
SCHWERWIEGEND: auto commit error...:java.lang.IllegalStateException:
this writer hit an OutOfMemoryError; cannot commit
at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2650)

follow by 10 Java heap space Exceptions, and one minute later at 17:36
the first auto-warming Exception:
06.07.2012 17:36:26 org.apache.solr.common.SolrException log
SCHWERWIEGEND: Error during auto-warming of key:pubDate:[1340971496000
TO 1341576296000]:java.lang.OutOfMemoryError: Java heap space
06.07.2012 17:36:28 org.apache.solr.common.SolrException log
SCHWERWIEGEND: Error during auto-warming of key:pubDate:[1340971495000
TO 1341576295000]:java.lang.OutOfMemoryError: Java heap space

> it really seems that you are hitting an OOM during auto warming. can
> this be the case for your failure.
> Can you raise the JVM memory and see if you still hit the spike and go
> OOM? this is very unlikely a IndexWriter problem. I'd rather look at
> your warmup queries ie. fieldcache, FieldValueCache usage. Are you
> sorting / facet on anything?

Auto warming problems began one minute after the "java
heap"-exceptions, so i think this are subsequent problems.
I configured very small caches(max. sizes between 512 and 2048) for my use case.
Warming queries looks like this, with sorting, but without facetting:
<lst>
           <str name="q">ag</str>
           <str name="fq">pubDate:[NOW-1DAY TO *]</str>
           <str name="sort">pubDate desc</str>
        </lst>

Du you think that 8GB for JVM are not enough? To raise the JVM memory
can solve the problem..
As mentioned, this server runs a long time with the same config
without problems, i am surprised that this problem was there at one
time
without heavy usage...now it's running smoothly again after restart
yesterday, so i don't know whet the problem appears again.

I try to update to 4.0 alpha today and run it with tomcat and report:)

Best regards
Vadim

2012/7/10 Simon Willnauer <[EMAIL PROTECTED]>:
> it really seems that you are hitting an OOM during auto warming. can
> this be the case for your failure.
> Can you raise the JVM memory and see if you still hit the spike and go
> OOM? this is very unlikely a IndexWriter problem. I'd rather look at
> your warmup queries ie. fieldcache, FieldValueCache usage. Are you
> sorting / facet on anything?
>
> simon
>
> On Tue, Jul 10, 2012 at 4:49 PM, Vadim Kisselmann
> <[EMAIL PROTECTED]> wrote:
>> Hi Robert,
>>
>>> Can you run Lucene's checkIndex tool on your index?
>>
>> No, unfortunately not. This Solr should run without stoppage, an
>> tomcat-restart is ok, but not more:)
>> I tested newer trunk-versions a couple of months ago, but they fail
>> all with tomcat.
>> i would test 4.0-alpha in next days with tomcat and open an jira-issue
>> if it doesn't work with it.