Solr, mail # user - seeing errors during replication process on slave boxes - read past EOF

seeing errors during replication process on slave boxes - read past EOF

geeky2 2012-06-04, 01:40

hello all,

environment: solr 3.5

1 - master
2 - slave

slaves are set to poll master every 10 minutes.

i have had replication running on one master and two slaves - for a few
weeks now. these boxes are not production boxes - just QA/test boxes.

right after i started a re-index on the master - i started to see the
following errors on both of the slave boxes.

in previous test runs - i have not noticed any errors.

can someone help me understand what is causing these errors?

thank you,

2012-06-03 19:30:23,104 INFO [org.apache.solr.update.UpdateHandler]
(pool-16-thread-1) start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
2012-06-03 19:30:23,164 SEVERE [org.apache.solr.handler.ReplicationHandler]
(pool-16-thread-1) SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
   at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331)
   at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
   at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
   at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
   at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1103)
   at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
   at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:470)
   at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:321)
   ... 11 more
Caused by: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
   at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:279)
   at org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
   at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readInt(MMapDirectory.java:315)
   at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:138)
   at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212)
   at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
   at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
   at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:235)
   at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:34)
   at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:506)
   at
org.apache.lucene.index.DirectoryReader.access$000(DirectoryReader.java:45)
   at
org.apache.lucene.index.DirectoryReader$2.doBody(DirectoryReader.java:498)
   at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
   at
org.apache.lucene.index.DirectoryReader.doOpenNoWriter(DirectoryReader.java:493)
   at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:450)
   at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:396)
   at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:520)
   at org.apache.lucene.index.IndexReader.reopen(IndexReader.java:697)
   at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:414)
   at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:425)
   at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:35)
   at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:501)
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1083)
   ... 14 more
2012-06-03 19:30:23,197 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tis
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kit.tis
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdt
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5jra.prx
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/segments_50qw
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tii
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Total time taken for download : 23 secs
2012-06-03 19:30:23,201 SEVERE [org.apache.solr.handler.ReplicationHandler]
(pool-12-thread-1) SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
   at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331)
   at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
   at org.

Re: seeing errors during replication process on slave boxes - read past EOF

Jack Krupansky 2012-06-04, 16:35

"started a re-index on the master "

Out of curiosity, how did you do that? Did you delete files or do a
deleteQuery for "*:*" or what?

Did the delete step have time to propagate to the slaves before documents
were added?

I am just curious - I don't know what the proper technique is for initiating
a re-index for a master-slave configuration.

-- Jack Krupansky

-----Original Message-----
From: geeky2
Sent: Sunday, June 03, 2012 9:40 PM
To: [EMAIL PROTECTED]
Subject: seeing errors during replication process on slave boxes - read past
EOF

hello all,

environment: solr 3.5

1 - master
2 - slave

slaves are set to poll master every 10 minutes.

i have had replication running on one master and two slaves - for a few
weeks now. these boxes are not production boxes - just QA/test boxes.

right after i started a re-index on the master - i started to see the
following errors on both of the slave boxes.

in previous test runs - i have not noticed any errors.

can someone help me understand what is causing these errors?

thank you,

2012-06-03 19:30:23,104 INFO [org.apache.solr.update.UpdateHandler]
(pool-16-thread-1) start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
2012-06-03 19:30:23,164 SEVERE [org.apache.solr.handler.ReplicationHandler]
(pool-16-thread-1) SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
   at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331)
   at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
   at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
   at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
   at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1103)
   at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
   at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:470)
   at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:321)
   ... 11 more
Caused by: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
   at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:279)
   at org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
   at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readInt(MMapDirectory.java:315)
   at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:138)
   at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212)
   at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
   at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
   at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:235)
   at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:34)
   at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:506)
   at
org.apache.lucene.index.DirectoryReader.access$000(DirectoryReader.java:45)
   at
org.apache.lucene.index.DirectoryReader$2.doBody(DirectoryReader.java:498)
   at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
   at
org.apache.lucene.index.DirectoryReader.doOpenNoWriter(DirectoryReader.java:493)
   at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:450)
   at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:396)
   at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:520)
   at org.apache.lucene.index.IndexReader.reopen(IndexReader.java:697)
   at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:414)
   at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:425)
   at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:35)
   at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:501)
   at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1083)
   ... 14 more
2012-06-03 19:30:23,197 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tis
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kit.tis
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdt
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5jra.prx
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/segments_50qw
2012-06-03 19:30:23,198 INFO [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping