We are getting large Solr pauses on Java garbage collection in 1.6 Java.

We have tried CMS. But we still have. 4 second wait on GC.

What works well for Solr when using 16 GB of RAM?

I have read lots of articles and now just looking for practical advise and examples.

Sent from my Mobile device
720-256-8076
---------------------------------------------------------------------

Nothing?

Bill Bell
Sent from mobile
On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:

> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>
> We have tried CMS. But we still have. 4 second wait on GC.
>
> What works well for Solr when using 16 GB of RAM?
>
> I have read lots of articles and now just looking for practical advise and examples.
>
> Sent from my Mobile device
> 720-256-8076

---------------------------------------------------------------------

Out of curiosity, how long does your Solr instance run before you hit a GC
issue? Minutes, hours, days? If reasonably long, maybe you simply need to
periodically take each Solr instance out of rotation and shut it down and
restart the JVM before any of them hit the GC issue.

The first question is whether the GC delay is being caused because your JVM
heap is too small (thrashing) or too large (long GC elapsed time.)

-- Jack Krupansky

-----Original Message-----
From: Bill Bell
Sent: Saturday, June 30, 2012 8:48 PM
To: Bill Bell
Cc: [EMAIL PROTECTED]
Subject: Re: Low pause GC for java 1.6

Nothing?

Bill Bell
Sent from mobile
On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:

> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>
> We have tried CMS. But we still have. 4 second wait on GC.
>
> What works well for Solr when using 16 GB of RAM?
>
> I have read lots of articles and now just looking for practical advise and
> examples.
>
> Sent from my Mobile device
> 720-256-8076

---------------------------------------------------------------------
---------------------------------------------------------------------

It GC after about 10 minutes.

Bill Bell
Sent from mobile
On Jun 30, 2012, at 10:48 PM, "Jack Krupansky" <[EMAIL PROTECTED]> wrote:

> Out of curiosity, how long does your Solr instance run before you hit a GC issue? Minutes, hours, days? If reasonably long, maybe you simply need to periodically take each Solr instance out of rotation and shut it down and restart the JVM before any of them hit the GC issue.
>
> The first question is whether the GC delay is being caused because your JVM heap is too small (thrashing) or too large (long GC elapsed time.)
>
> -- Jack Krupansky
>
> -----Original Message----- From: Bill Bell
> Sent: Saturday, June 30, 2012 8:48 PM
> To: Bill Bell
> Cc: [EMAIL PROTECTED]
> Subject: Re: Low pause GC for java 1.6
>
> Nothing?
>
> Bill Bell
> Sent from mobile
>
>
> On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:
>
>> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>>
>> We have tried CMS. But we still have. 4 second wait on GC.
>>
>> What works well for Solr when using 16 GB of RAM?
>>
>> I have read lots of articles and now just looking for practical advise and examples.
>>
>> Sent from my Mobile device
>> 720-256-8076
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

To me, that suggests that you simply don't have enough JVM heap memory from
the start. Maybe the nature of your schema and data results in a larger than
expected use of heap from the start. Check JVM available memory/heap right
after starting Solr, before any user queries, and then after doing a few
queries. What are you actually doing during those 10 minutes? Are you doing
lots of queries? Lots of document updates? Or... absolutely nothing? Maybe
the warming queries are consuming the available heap. Maybe the field caches
are consuming the heap? Do you have lots of facets with lots of unique
values? Roughly how many documents and how many fields and how many facets
are we talking about?

It would probably be more appropriate to respond to my inquiry over on the
Solr user list and continue discussion there since there doesn't yet appear
to be any evidence of a problem with Lucene/Solr itself.

-- Jack Krupansky

-----Original Message-----
From: Bill Bell
Sent: Sunday, July 01, 2012 1:28 AM
To: [EMAIL PROTECTED]
Cc: <[EMAIL PROTECTED]>
Subject: Re: Low pause GC for java 1.6

It GC after about 10 minutes.

Bill Bell
Sent from mobile
On Jun 30, 2012, at 10:48 PM, "Jack Krupansky" <[EMAIL PROTECTED]>
wrote:

> Out of curiosity, how long does your Solr instance run before you hit a GC
> issue? Minutes, hours, days? If reasonably long, maybe you simply need to
> periodically take each Solr instance out of rotation and shut it down and
> restart the JVM before any of them hit the GC issue.
>
> The first question is whether the GC delay is being caused because your
> JVM heap is too small (thrashing) or too large (long GC elapsed time.)
>
> -- Jack Krupansky
>
> -----Original Message----- From: Bill Bell
> Sent: Saturday, June 30, 2012 8:48 PM
> To: Bill Bell
> Cc: [EMAIL PROTECTED]
> Subject: Re: Low pause GC for java 1.6
>
> Nothing?
>
> Bill Bell
> Sent from mobile
>
>
> On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:
>
>> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>>
>> We have tried CMS. But we still have. 4 second wait on GC.
>>
>> What works well for Solr when using 16 GB of RAM?
>>
>> I have read lots of articles and now just looking for practical advise
>> and examples.
>>
>> Sent from my Mobile device
>> 720-256-8076
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------
---------------------------------------------------------------------

Is that a 4 second or 0.4 second wait?

You can benchmark the different collectors yourself.

CMS is probably the best choice.

If you have more heap than you need, then the collections will be longer than necessary. At some point, the collector has to look at all of the heap and that is slow. Try reducing the heap size.

You really should use a tool that graphs the GC behavior. Make sure that you have a big enough nursery (or whatever Sun calls it) to handle all of the temporary storage for for each HTTP request. You want all of that to be reclaimed in the minor collections. If per-request storage is allocated in tenured space, that means the next major GC will be sooner than necessary.

wunder

On Jun 30, 2012, at 6:48 PM, Bill Bell wrote:

> Nothing?
>
> Bill Bell
> Sent from mobile
>
>
> On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:
>
>> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>>
>> We have tried CMS. But we still have. 4 second wait on GC.
>>
>> What works well for Solr when using 16 GB of RAM?
>>
>> I have read lots of articles and now just looking for practical advise and examples.
>>
>> Sent from my Mobile device
>> 720-256-8076
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

Bill,

As you know, it really depends on the size of your index combined with which features you're using.  There is really no substitute for having a good load test and monitoring tool and to run multiple tests while trying different settings.

My guess is that you're experiencing "full" gc's, even with CMS enabled.  This means either your tenured ("old") generation is too small or you have the "-XX:CMSInitiatingOccupancyFraction" set too high (it starts the CMS too late and runs out of memory before it can finish).  We've found that some of the defaults the JVM picks and/or the general advice out there doesn't apply to an app like Solr, which is just a different kind of animal than the typical web frontend you might run in a J2ee container.

Below are the settings i am using as a starting point for our development Solr 4.0 app.  These may or may not work for you but at least should give you a basic idea of how 1 other installation is configured.

Also, if you're using older grouping patches (I remember you worked on some of these), perhaps you're hitting some of the scalibility problems that were predicted for some of these?  I'm pretty sure the GA grouping features in 3.x solved these problems though.

Finally, you probably will get better responses on the users list than the dev list.  Also, other users might benefit from other answers you get, so perhaps you could cross-post your question.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311

# Basic JVM settings.  I thing the 3g new generation size is bigger than you'd normally have with a typical web app but for us it makes the old gen fill up slower and have fewer CMS gc's.  Minor ("parnew") gc's are still fast enough for us, even with a biggish new gen.
-XX:MaxNewSize=3000m
-XX:NewSize=3000m
-Xms20g
-Xmx20g
-XX:MaxPermSize=256m

# These are our CMS settings
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=85
-XX:+CMSParallelRemarkEnabled
-XX:CMSMaxAbortablePrecleanTime=15000

# trial and error found this to be the sweet spot for out 16-way machines.
-XX:ParallelGCThreads=8

# YOu want these so you can see in your logs what is going on.  There are some tutorials on the web on how to make sense of verbose garbage collection.  There's no problem using these in Production.
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps

# Use this on a 64-bit machine unless your jvm is too old to support it (on by default on newer jvms, I think)
-XX:+UseCompressedOops

# we found these save a little memory
-XX:+UseStringCache
-XX:+UseCompressedStrings

-----Original Message-----
From: Bill Bell [mailto:[EMAIL PROTECTED]]
Sent: Saturday, June 30, 2012 8:49 PM
To: Bill Bell
Cc: [EMAIL PROTECTED]
Subject: Re: Low pause GC for java 1.6

Nothing?

Bill Bell
Sent from mobile
On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:

> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>
> We have tried CMS. But we still have. 4 second wait on GC.
>
> What works well for Solr when using 16 GB of RAM?
>
> I have read lots of articles and now just looking for practical advise and examples.
>
> Sent from my Mobile device
> 720-256-8076

---------------------------------------------------------------------
---------------------------------------------------------------------

On 6/30/2012 7:48 PM, Bill Bell wrote:
> Nothing?
>
> Bill Bell
> Sent from mobile
>
>
> On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:
>
>> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>>
>> We have tried CMS. But we still have. 4 second wait on GC.
>>
>> What works well for Solr when using 16 GB of RAM?
>>
>> I have read lots of articles and now just looking for practical advise and examples.

Bill, this is what I am using to minimize GC impact on Solr:

-Xms4096M
-Xmx8192M
-XX:NewRatio=1
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled

The key seems to be having a large chunk of memory for the new
generation (eden), so that it's not forced to heavily use the old
generation.  With a NewRatio of 1, half the memory gets used for the old
generation, the other half is divided up among eden and survivor spaces,
with eden getting most of it.  I don't know what the default is, but
whatever it is resulted in a pretty small eden space, and I was doing a
lot of full GCs.

As of this moment, Solr (3.5.0) and Jetty are using a little less than
5GB of RAM.  At an uptime of 53190.8 seconds (14 hours), there were 229
young garbage collections that took a total of 30.522 seconds.  There
were 14 full garbage collections that took a total of 9.351 seconds.  
These numbers come from jstat with a 5000 millisecond interval.

jstat -gc -t -h12 <pid> 5000

I don't think you can specify a NewRatio below 1, so if you end up
needing more eden space than that can give you, you can set an explicit
NewSize instead.

Thanks,
Shawn
---------------------------------------------------------------------

Which params do you use now?
Which ones have you tried?

Please move to user@

Otis 
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 

----- Original Message -----
> From: Bill Bell <[EMAIL PROTECTED]>
> To: Bill Bell <[EMAIL PROTECTED]>
> Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Saturday, June 30, 2012 9:48 PM
> Subject: Re: Low pause GC for java 1.6
>
> Nothing?
>
> Bill Bell
> Sent from mobile
>
>
> On Jun 29, 2012, at 9:09 PM, Bill Bell <[EMAIL PROTECTED]> wrote:
>
>>  We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>>
>>  We have tried CMS. But we still have. 4 second wait on GC.
>>
>>  What works well for Solr when using 16 GB of RAM?
>>
>>  I have read lots of articles and now just looking for practical advise and
> examples.
>>
>>  Sent from my Mobile device
>>  720-256-8076
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

My first question would be, do you need 16GB of heap, thats a lot and
will contribute to long pauses, what happens if you try it out with 8GB,
are you trying to be too greedy with the caches ?

You question is very open ended, tuning a GC is an art from, what works
for one workload is often quite wrong for another, are you using any JVM
flags what are they ?

If you put the following on your JVM arguments, and then put the log
somewhere on the internet it will make it easietr to fgure out what
might be wrong

-Xloggc:gc.log -XX:+NUMAStats -XX:+PrintCMSInitiationStatistics
-XX:+PrintFlagsFinal -XX:+PrintGCApplicationConcurrentTime
-XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps -XX:+PrintJNIGCStalls -XX:+PrintReferenceGC
-XX:+PrintTenuringDistribution

-- Greg

On 29/06/2012 20:09, Bill Bell wrote:
> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>
> We have tried CMS. But we still have. 4 second wait on GC.
>
> What works well for Solr when using 16 GB of RAM?
>
> I have read lots of articles and now just looking for practical advise and examples.
>
> Sent from my Mobile device
> 720-256-8076
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------