hello all,

environment: solr 3.5

1 - master
2 - slave

slaves are set to poll master every 10 minutes.

i have had replication running on one master and two slaves - for a few
weeks now.  these boxes are not production boxes - just QA/test boxes.

right after i started a re-index on the master - i started to see the
following errors on both of the slave boxes.

in previous test runs - i have not noticed any errors.

can someone help me understand what is causing these errors?

thank you,

2012-06-03 19:30:23,104 INFO  [org.apache.solr.update.UpdateHandler]
(pool-16-thread-1) start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
2012-06-03 19:30:23,164 SEVERE [org.apache.solr.handler.ReplicationHandler]
(pool-16-thread-1) SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
    at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331)
    at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
    at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
    at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1103)
    at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
    at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:470)
    at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:321)
    ... 11 more
Caused by: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
    at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:279)
    at org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
    at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readInt(MMapDirectory.java:315)
    at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:138)
    at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
    at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:235)
    at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:34)
    at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:506)
    at
org.apache.lucene.index.DirectoryReader.access$000(DirectoryReader.java:45)
    at
org.apache.lucene.index.DirectoryReader$2.doBody(DirectoryReader.java:498)
    at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
    at
org.apache.lucene.index.DirectoryReader.doOpenNoWriter(DirectoryReader.java:493)
    at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:450)
    at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:396)
    at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:520)
    at org.apache.lucene.index.IndexReader.reopen(IndexReader.java:697)
    at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:414)
    at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:425)
    at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:35)
    at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:501)
    at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1083)
    ... 14 more
2012-06-03 19:30:23,197 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tis
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kit.tis
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdt
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5jra.prx
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/segments_50qw
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tii
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Total time taken for download : 23 secs
2012-06-03 19:30:23,201 SEVERE [org.apache.solr.handler.ReplicationHandler]
(pool-12-thread-1) SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
    at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331)
    at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
    at org.

"started a re-index on the master "

Out of curiosity, how did you do that? Did you delete files or do a
deleteQuery for "*:*" or what?

Did the delete step have time to propagate to the slaves before documents
were added?

I am just curious - I don't know what the proper technique is for initiating
a re-index for a master-slave configuration.

-- Jack Krupansky

-----Original Message-----
From: geeky2
Sent: Sunday, June 03, 2012 9:40 PM
To: [EMAIL PROTECTED]
Subject: seeing errors during replication process on slave boxes - read past
EOF

hello all,

environment: solr 3.5

1 - master
2 - slave

slaves are set to poll master every 10 minutes.

i have had replication running on one master and two slaves - for a few
weeks now.  these boxes are not production boxes - just QA/test boxes.

right after i started a re-index on the master - i started to see the
following errors on both of the slave boxes.

in previous test runs - i have not noticed any errors.

can someone help me understand what is causing these errors?

thank you,

2012-06-03 19:30:23,104 INFO  [org.apache.solr.update.UpdateHandler]
(pool-16-thread-1) start
commit(optimize=false,waitFlush=true,waitSearcher=true,expungeDeletes=false)
2012-06-03 19:30:23,164 SEVERE [org.apache.solr.handler.ReplicationHandler]
(pool-16-thread-1) SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
    at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:331)
    at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:268)
    at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
    at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
    at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1103)
    at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
    at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:470)
    at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:321)
    ... 11 more
Caused by: java.io.IOException: read past EOF:
MMapIndexInput(path="/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdx")
    at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte(MMapDirectory.java:279)
    at org.apache.lucene.store.DataInput.readInt(DataInput.java:84)
    at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readInt(MMapDirectory.java:315)
    at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:138)
    at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:93)
    at
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:235)
    at
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:34)
    at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:506)
    at
org.apache.lucene.index.DirectoryReader.access$000(DirectoryReader.java:45)
    at
org.apache.lucene.index.DirectoryReader$2.doBody(DirectoryReader.java:498)
    at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754)
    at
org.apache.lucene.index.DirectoryReader.doOpenNoWriter(DirectoryReader.java:493)
    at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:450)
    at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:396)
    at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:520)
    at org.apache.lucene.index.IndexReader.reopen(IndexReader.java:697)
    at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:414)
    at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:425)
    at
org.apache.solr.search.SolrIndexReader.reopen(SolrIndexReader.java:35)
    at
org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:501)
    at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1083)
    ... 14 more
2012-06-03 19:30:23,197 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kiq.tis
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kit.tis
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5kgm.fdt
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/_5jra.prx
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping download for
/appl/solr/stress/partcatalog/index/core1/index.20120514101522/segments_50qw
2012-06-03 19:30:23,198 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Skipping

hello,

i have shell scripts that handle all of the operational tasks.  

example:

curl -v http://${SERVER}.bogus.com:${PORT}/somecore/dataimport -F
command=full-import -F clean=${CLEAN} -F commit=${COMMIT} -F
optimize=${OPTIMIZE}
--
View this message in context: http://lucene.472066.n3.nabble.com/seeing-errors-during-replication-process-on-slave-boxes-read-past-EOF-tp3987489p3987617.html
Sent from the Solr - User mailing list archive at Nabble.com.