It's a small indexing job coming from nutch.

2011-10-26 15:07:29,039 WARN  mapred.LocalJobRunner - job_local_0011
java.io.IOException: org.apache.solr.client.solrj.SolrServerException: Error
executi$
        at
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: org.apache.solr.client.solrj.SolrServerException: Error executing
query
        at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja$
        at
org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
        at
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$
        ... 3 more
Caused by: org.apache.solr.common.SolrException: Java heap space
 java.lang.OutOfMem$

Java heap space  java.lang.OutOfMemoryError: Java heap space    at
org.apache.lucene$

request: localhost/solr/select?q=id:[* TO *]&fl=id,boost,tstamp,$
        at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHt$
        at
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHt$
        at
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja$
        ... 5 more

More on what's happening. It seems to be timing out during the commit.

The new documents are small, but the existing index is large (11 million
records).

INFO: Closing Searcher@4a7df6 main
>
> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>         ...
>
> Oct 26, 2011 4:51:17 PM org.apache.solr.update.processor.LogUpdateProcessor
> finish
> *INFO: {commit=} 0 2453
> **Oct 26, 2011 4:51:17 PM org.apache.solr.core.SolrCore execute
> **INFO: [] webapp=/solr path=/update
> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2}
> status=0 QTime=2453
> *Oct 26, 2011 4:51:52 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=11576871 *status=0
> QTime=35298*
> Oct 26, 2011 4:51:53 PM org.apache.solr.core.SolrCore execute
> INFO: [] webapp=/solr path=/select
> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=11576871
> status=0 QTime=1
> *java.lang.OutOfMemoryError: Java heap space*
> Dumping heap to /home/bitnami/apache-solr-3.4.0/example/heaplog ...
> Heap dump file created [306866344 bytes in 32.376 secs]

On Wed, Oct 26, 2011 at 11:09 AM, Fred Zimmerman <[EMAIL PROTECTED]>wrote:

> It's a small indexing job coming from nutch.
>
> 2011-10-26 15:07:29,039 WARN  mapred.LocalJobRunner - job_local_0011
> java.io.IOException: org.apache.solr.client.solrj.SolrServerException:
> Error executi$
>         at
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
> Caused by: org.apache.solr.client.solrj.SolrServerException: Error
> executing query
>         at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja$
>         at
> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
>         at
> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$
>         ... 3 more
> Caused by: org.apache.solr.common.SolrException: Java heap space
>  java.lang.OutOfMem$
>
> Java heap space  java.lang.OutOfMemoryError: Java heap space    at
> org.apache.lucene$
>
> request: localhost/solr/select?q=id:[* TO *]&fl=id,boost,tstamp,$
>         at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHt$
>         at
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHt$
>         at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja$
>         ... 5 more
>
>

http://wiki.apache.org/solr/SolrPerformanceFactors#Schema_Design_Considerations

The number of indexed fields greatly increases the following:
>
>    - Memory usage during indexing
>
>
>    - Segment merge time
>
>
>    - Optimization times
>
>
>    - Index size
>
> These impacts can be reduced by the use of omitNorms="true"
http://lucene.472066.n3.nabble.com/What-is-omitNorms-td2987547.html

1. length normalization will not work on the specific field--
> Which means matching documents with shorter length will not be
> preferred/boost over matching documents with greater length for the specific
> field, at search time.
> For my application, I actually prefer documents with greater length.
> 2. Index time boosting will not be available on the field.
> If, both the above cases are not required by you, then, you can set
> "omitNorms=true" for the specific fields.
> This has an added advantage, it will save you some(or a lot of) RAM also,
> since, with "omitNorms=false" on total "N" fields in the index will require
> RAM of size:
>  Total docs in index * 1 byte * N
> I have a lot of fields: I count 31 without omitNorms values, which means
> false by default.
Gak!  11,000,000 * 1 * 31 = 31 x 10M = 310MB RAM all by itself.

On Wed, Oct 26, 2011 at 1:01 PM, Fred Zimmerman <[EMAIL PROTECTED]>wrote:

> More on what's happening. It seems to be timing out during the commit.
>
> The new documents are small, but the existing index is large (11 million
> records).
>
> INFO: Closing Searcher@4a7df6 main
>>
>> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
>>         ...
>>
>
>
>> Oct 26, 2011 4:51:17 PM
>> org.apache.solr.update.processor.LogUpdateProcessor finish
>> *INFO: {commit=} 0 2453
>> **Oct 26, 2011 4:51:17 PM org.apache.solr.core.SolrCore execute
>> **INFO: [] webapp=/solr path=/update
>> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2}
>> status=0 QTime=2453
>> *Oct 26, 2011 4:51:52 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select
>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=11576871 *status=0
>> QTime=35298*
>> Oct 26, 2011 4:51:53 PM org.apache.solr.core.SolrCore execute
>> INFO: [] webapp=/solr path=/select
>> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=11576871
>> status=0 QTime=1
>> *java.lang.OutOfMemoryError: Java heap space*
>> Dumping heap to /home/bitnami/apache-solr-3.4.0/example/heaplog ...
>> Heap dump file created [306866344 bytes in 32.376 secs]
>
>
>
> On Wed, Oct 26, 2011 at 11:09 AM, Fred Zimmerman <[EMAIL PROTECTED]>wrote:
>
>> It's a small indexing job coming from nutch.
>>
>> 2011-10-26 15:07:29,039 WARN  mapred.LocalJobRunner - job_local_0011
>> java.io.IOException: org.apache.solr.client.solrj.SolrServerException:
>> Error executi$
>>         at
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$
>>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>>         at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
>> Caused by: org.apache.solr.client.solrj.SolrServerException: Error
>> executing query
>>         at
>> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja$
>>         at
>> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
>>         at
>> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$
>>         ... 3 more
>> Caused by: org.apache.solr.common.SolrException: Java heap space
>>  java.lang.OutOfMem$
>>
>> Java heap space  java.lang.OutOfMemoryError: Java heap space    at
>> org.apache.lucene$
>>
>> request: localhost/solr/select?q=id:[* TO *]&fl=id,boost,tstamp,$
>>         at
>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHt$