Hi,

I am trying to index documents and when all is complete and optimize is
called I get

IFD [main]: setInfoStream
deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
IW 0 [main]: setInfoStream:
dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
autoCommit=false mergePolicy=org
.apache.lucene.index.LogByteSizeMergePolicy@92cb49
mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
ramBufferS
izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1 maxFieldLength=10000
index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
flushDocs=false flushDeletes=true flushDocStores=false numD
ocs=0 numBufDelTerms=0
IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
_ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
IW 0 [main]: CMS: now merge
IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0 _ay:c225->_0
_az:c5->_0 _b0:c5->_0 into _b5 [optimize]
IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
IW 0 [main]: CMS:   no more merges pending; now return
IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
IW 0 [Lucene Merge Thread #73]: now merge
  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
_b5 [optimize]
  merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
  index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
_az:c5->_0 _b0:c5->_0
IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
_ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
IW 0 [Lucene Merge Thread #73]: hit exception during merge
Exception in thread "Lucene Merge Thread #73"
org.apache.lucene.index.MergePolicy$MergeException:
java.io.FileNotFoundException: /SY
S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
        at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
        at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
Caused by: java.io.FileNotFoundException:
/SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
        at
org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
        at
org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
        at
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
        at
org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
        at
org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
        at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
        at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
        at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP jspMain
SEVERE: Indexing not completebackground merge hit exception: _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
optimize]
java.io.IOException: background merge hit exception: _61:c245->_0
_9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
]
        at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
        at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
        at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
        at org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
Source)
        at org.hp.openvms.lucene.index.FileIndexer.main(Unknown Source)
Caused by: java.io.FileNotFoundException:
/SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
        at
org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
        at
org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
        at
org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
        at
org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
        at
org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
        at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
        at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
        at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
IW 0 [main]: now flush at close
IW 0 [main]:   flush: segment=null d

Is there any chance that two writers are open on this directory?  Or,
something external to Lucene is removing files from the directory.

It looks like there were at least two missing files (_37

On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<[EMAIL PROTECTED]> wrote:
>
> Hi,
>
> I am trying to index documents and when all is complete and optimize is
> called I get
>
> IFD [main]: setInfoStream
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
> IW 0 [main]: setInfoStream:
> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
> autoCommit=false mergePolicy=org
> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
> ramBufferS
> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1 maxFieldLength=10000
> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
> flushDocs=false flushDeletes=true flushDocStores=false numD
> ocs=0 numBufDelTerms=0
> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
> IW 0 [main]: CMS: now merge
> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0 _ay:c225->_0
> _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
> IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
> IW 0 [main]: CMS:   no more merges pending; now return
> IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
> IW 0 [Lucene Merge Thread #73]: now merge
>  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
> _b5 [optimize]
>  merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
>  index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
> _az:c5->_0 _b0:c5->_0
> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
> IW 0 [Lucene Merge Thread #73]: hit exception during merge
> Exception in thread "Lucene Merge Thread #73"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.io.FileNotFoundException: /SY
> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
> Caused by: java.io.FileNotFoundException:
> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>        at
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
>        at
> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
>        at
> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
>        at
> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
>        at
> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
>        at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)

>>Is there any chance that two writers are open on this directory?  
No, thats not true.

>>something external to Lucene is removing files from the directory.
No this also has rare chances as I am the owner of these files and other
then me no one can delete the, :-)

Here are all the files in the index directory:
segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
_0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
_9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
_b2.tis;1           _b3.fdt;1           _b4.fdt;1

I see _37 present.

Regards,
Rishi
Michael McCandless-2 wrote:
>
> Is there any chance that two writers are open on this directory?  Or,
> something external to Lucene is removing files from the directory.
>
> It looks like there were at least two missing files (_37
>
> On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<[EMAIL PROTECTED]>
> wrote:
>>
>> Hi,
>>
>> I am trying to index documents and when all is complete and optimize is
>> called I get
>>
>> IFD [main]: setInfoStream
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
>> IW 0 [main]: setInfoStream:
>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
>> autoCommit=false mergePolicy=org
>> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
>> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
>> ramBufferS
>> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
>> maxFieldLength=10000
>> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
>> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
>> flushDocs=false flushDeletes=true flushDocStores=false numD
>> ocs=0 numBufDelTerms=0
>> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
>> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
>> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
>> IW 0 [main]: CMS: now merge
>> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
>> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0 _ay:c225->_0
>> _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>> IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
>> IW 0 [main]: CMS:   no more merges pending; now return
>> IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
>> IW 0 [Lucene Merge Thread #73]: now merge
>>  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
>> _b5 [optimize]
>>  merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
>>  index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
>> _az:c5->_0 _b0:c5->_0
>> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
>> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>> IW 0 [Lucene Merge Thread #73]: hit exception during merge
>> Exception in thread "Lucene Merge Thread #73"
>> org.apache.lucene.index.MergePolicy$MergeException:
>> java.io.FileNotFoundException: /SY
>> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
>> Caused by: java.io.FileNotFoundException:
>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>>        at java.io.RandomAccessFile.open(Native Method)
>>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

Is that a local file system, or a network share?

On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal <[EMAIL PROTECTED]>wrote:

>
> >>Is there any chance that two writers are open on this directory?
> No, thats not true.
>
> >>something external to Lucene is removing files from the directory.
> No this also has rare chances as I am the owner of these files and other
> then me no one can delete the, :-)
>
> Here are all the files in the index directory:
> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
> _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
> _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
>
> I see _37 present.
>
> Regards,
> Rishi
>
>
> Michael McCandless-2 wrote:
> >
> > Is there any chance that two writers are open on this directory?  Or,
> > something external to Lucene is removing files from the directory.
> >
> > It looks like there were at least two missing files (_37
> >
> > On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<[EMAIL PROTECTED]>
> > wrote:
> >>
> >> Hi,
> >>
> >> I am trying to index documents and when all is complete and optimize is
> >> called I get
> >>
> >> IFD [main]: setInfoStream
> >>
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
> >> IW 0 [main]: setInfoStream:
> >> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
> >> autoCommit=false mergePolicy=org
> >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
> >> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
> >> ramBufferS
> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
> >> maxFieldLength=10000
> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
> >>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 _61:c245->_0
> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> IW 0 [main]:   flush: segment=null docStoreSegment=null docStoreOffset=0
> >> flushDocs=false flushDeletes=true flushDocStores=false numD
> >> ocs=0 numBufDelTerms=0
> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0 _61:c245->_0
> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
> >> IW 0 [main]: CMS: now merge
> >> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
> _ay:c225->_0
> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
> >> IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
> >> IW 0 [main]: CMS:   no more merges pending; now return
> >> IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
> >> IW 0 [Lucene Merge Thread #73]: now merge
> >>  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into
> >> _b5 [optimize]
> >>  merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
> >>  index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 _ay:c225->_0
> >> _az:c5->_0 _b0:c5->_0
> >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
> >> IW 0 [Lucene Merge Thread #73]: hit exception during merge
> >> Exception in thread "Lucene Merge Thread #73"
> >> org.apache.lucene.index.MergePolicy$MergeException:
> >> java.io.FileNotFoundException: /SY
> >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
> >>        at
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
> >>        at
> >>
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)

It is a local file system.

We are using lucene 2.4 and java 1.5

Regards,
Rishi
Shai Erera wrote:
>
> Is that a local file system, or a network share?
>
> On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
> <[EMAIL PROTECTED]>wrote:
>
>>
>> >>Is there any chance that two writers are open on this directory?
>> No, thats not true.
>>
>> >>something external to Lucene is removing files from the directory.
>> No this also has rare chances as I am the owner of these files and other
>> then me no one can delete the, :-)
>>
>> Here are all the files in the index directory:
>> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
>> _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
>> _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
>> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
>>
>> I see _37 present.
>>
>> Regards,
>> Rishi
>>
>>
>> Michael McCandless-2 wrote:
>> >
>> > Is there any chance that two writers are open on this directory?  Or,
>> > something external to Lucene is removing files from the directory.
>> >
>> > It looks like there were at least two missing files (_37
>> >
>> > On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I am trying to index documents and when all is complete and optimize
>> is
>> >> called I get
>> >>
>> >> IFD [main]: setInfoStream
>> >>
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
>> >> IW 0 [main]: setInfoStream:
>> >>
>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
>> >> autoCommit=false mergePolicy=org
>> >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
>> >> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
>> >> ramBufferS
>> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
>> >> maxFieldLength=10000
>> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>> >>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
>> _61:c245->_0
>> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> IW 0 [main]:   flush: segment=null docStoreSegment=null
>> docStoreOffset=0
>> >> flushDocs=false flushDeletes=true flushDocStores=false numD
>> >> ocs=0 numBufDelTerms=0
>> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
>> _61:c245->_0
>> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
>> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
>> >> IW 0 [main]: CMS: now merge
>> >> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
>> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
>> _ay:c225->_0
>> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>> >> IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
>> >> IW 0 [main]: CMS:   no more merges pending; now return
>> >> IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
>> >> IW 0 [Lucene Merge Thread #73]: now merge
>> >>  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> into
>> >> _b5 [optimize]
>> >>  merge=org.apache.lucene.index.MergePolicy$OneMerge@3725db
>> >>  index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>> _ay:c225->_0
>> >> _az:c5->_0 _b0:c5->_0
>> >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
>> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>> >> IW 0 [Lucene Merge Thread #73]: hit exception during merge
>> >> Exception in thread "Lucene Merge Thread #73"
>> >> org.apache.lucene.index.MergePolicy$MergeException:
>> >> java.io.FileNotFoundException: /SY
>> >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

I noticed the exception is "Caused by: java.io.FileNotFoundException:
/SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"

I searched for i/o error (errno:5) and found some information which
associates it w/ a more native IO problem, like corrupt file due to system
crash etc.

Did you experience that? Can you try to create the index somewhere else, or
on another drive?

Shai

On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal <[EMAIL PROTECTED]>wrote:

>
> It is a local file system.
>
> We are using lucene 2.4 and java 1.5
>
> Regards,
> Rishi
>
>
> Shai Erera wrote:
> >
> > Is that a local file system, or a network share?
> >
> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
> > <[EMAIL PROTECTED]>wrote:
> >
> >>
> >> >>Is there any chance that two writers are open on this directory?
> >> No, thats not true.
> >>
> >> >>something external to Lucene is removing files from the directory.
> >> No this also has rare chances as I am the owner of these files and other
> >> then me no one can delete the, :-)
> >>
> >> Here are all the files in the index directory:
> >> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
> >>
> >> I see _37 present.
> >>
> >> Regards,
> >> Rishi
> >>
> >>
> >> Michael McCandless-2 wrote:
> >> >
> >> > Is there any chance that two writers are open on this directory?  Or,
> >> > something external to Lucene is removing files from the directory.
> >> >
> >> > It looks like there were at least two missing files (_37
> >> >
> >> > On Thu, Aug 13, 2009 at 5:19 AM, rishisinghal<[EMAIL PROTECTED]
> >
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I am trying to index documents and when all is complete and optimize
> >> is
> >> >> called I get
> >> >>
> >> >> IFD [main]: setInfoStream
> >> >>
> >>
> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
> >> >> IW 0 [main]: setInfoStream:
> >> >>
> >> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
> >> >> autoCommit=false mergePolicy=org
> >> >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
> >> >>
> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
> >> >> ramBufferS
> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
> >> >> maxFieldLength=10000
> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
> >> >>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
> >> _61:c245->_0
> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> >> IW 0 [main]:   flush: segment=null docStoreSegment=null
> >> docStoreOffset=0
> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD
> >> >> ocs=0 numBufDelTerms=0
> >> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
> >> _61:c245->_0
> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
> >> >> IW 0 [main]: CMS: now merge
> >> >> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> >> IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
> >> _ay:c225->_0
> >> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
> >> >> IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
> >> >> IW 0 [main]: CMS:   no more merges pending; now return
> >> >> IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
> >> >> IW 0 [Lucene Merge Thread #73]: now merge
> >> >>  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
> >> into
> >> >> _b5 [optimize]

I tried creating the index in different disks but still i see the issue :-(

I tried to index documents in other disks also and got the same exception.

I also tried
$ java org.apache.lucene.index.CheckIndex /SYS$SYSDEVICE/RISHI/melon_1600/
-segment _61
NOTE: testing will be more thorough if you run java with
'-ea:org.apache.lucene...', so assertions are enabled
Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene 2.4]

Checking only these segments: _61:
No problems were detected with this index.

Regards,
Rishi
Shai Erera wrote:
>
> I noticed the exception is "Caused by: java.io.FileNotFoundException:
> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
>
> I searched for i/o error (errno:5) and found some information which
> associates it w/ a more native IO problem, like corrupt file due to system
> crash etc.
>
> Did you experience that? Can you try to create the index somewhere else,
> or
> on another drive?
>
> Shai
>
> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
> <[EMAIL PROTECTED]>wrote:
>
>>
>> It is a local file system.
>>
>> We are using lucene 2.4 and java 1.5
>>
>> Regards,
>> Rishi
>>
>>
>> Shai Erera wrote:
>> >
>> > Is that a local file system, or a network share?
>> >
>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
>> > <[EMAIL PROTECTED]>wrote:
>> >
>> >>
>> >> >>Is there any chance that two writers are open on this directory?
>> >> No, thats not true.
>> >>
>> >> >>something external to Lucene is removing files from the directory.
>> >> No this also has rare chances as I am the owner of these files and
>> other
>> >> then me no one can delete the, :-)
>> >>
>> >> Here are all the files in the index directory:
>> >> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
>> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
>> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
>> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
>> >>
>> >> I see _37 present.
>> >>
>> >> Regards,
>> >> Rishi
>> >>
>> >>
>> >> Michael McCandless-2 wrote:
>> >> >
>> >> > Is there any chance that two writers are open on this directory?
>> Or,
>> >> > something external to Lucene is removing files from the directory.
>> >> >
>> >> > It looks like there were at least two missing files (_37
>> >> >
>> >> > On Thu, Aug 13, 2009 at 5:19 AM,
>> rishisinghal<[EMAIL PROTECTED]
>> >
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> I am trying to index documents and when all is complete and
>> optimize
>> >> is
>> >> >> called I get
>> >> >>
>> >> >> IFD [main]: setInfoStream
>> >> >>
>> >>
>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
>> >> >> IW 0 [main]: setInfoStream:
>> >> >>
>> >>
>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
>> >> >> autoCommit=false mergePolicy=org
>> >> >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
>> >> >>
>> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
>> >> >> ramBufferS
>> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
>> >> >> maxFieldLength=10000
>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>> >> >>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
>> >> _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]:   flush: segment=null docStoreSegment=null
>> >> docStoreOffset=0
>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD
>> >> >> ocs=0 numBufDelTerms=0
>> >> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
>> >> _61:c245->_0
>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>> >> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

It's very odd that CheckIndex has no trouble opening the segment's
files, yet when you run optimize the OS reports a "file not found"
exception (errno 5).  Something odd is happening at the OS/filesystem
level.

What OS are you running on?

Can you boil this down to a smallish standalone test that shows the problem?

Mike

On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<[EMAIL PROTECTED]> wrote:
>
> I tried creating the index in different disks but still i see the issue :-(
>
> I tried to index documents in other disks also and got the same exception.
>
> I also tried
> $ java org.apache.lucene.index.CheckIndex /SYS$SYSDEVICE/RISHI/melon_1600/
> -segment _61
> NOTE: testing will be more thorough if you run java with
> '-ea:org.apache.lucene...', so assertions are enabled
> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene 2.4]
>
> Checking only these segments: _61:
> No problems were detected with this index.
>
> Regards,
> Rishi
>
>
> Shai Erera wrote:
>>
>> I noticed the exception is "Caused by: java.io.FileNotFoundException:
>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
>>
>> I searched for i/o error (errno:5) and found some information which
>> associates it w/ a more native IO problem, like corrupt file due to system
>> crash etc.
>>
>> Did you experience that? Can you try to create the index somewhere else,
>> or
>> on another drive?
>>
>> Shai
>>
>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
>> <[EMAIL PROTECTED]>wrote:
>>
>>>
>>> It is a local file system.
>>>
>>> We are using lucene 2.4 and java 1.5
>>>
>>> Regards,
>>> Rishi
>>>
>>>
>>> Shai Erera wrote:
>>> >
>>> > Is that a local file system, or a network share?
>>> >
>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
>>> > <[EMAIL PROTECTED]>wrote:
>>> >
>>> >>
>>> >> >>Is there any chance that two writers are open on this directory?
>>> >> No, thats not true.
>>> >>
>>> >> >>something external to Lucene is removing files from the directory.
>>> >> No this also has rare chances as I am the owner of these files and
>>> other
>>> >> then me no one can delete the, :-)
>>> >>
>>> >> Here are all the files in the index directory:
>>> >> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
>>> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
>>> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
>>> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
>>> >>
>>> >> I see _37 present.
>>> >>
>>> >> Regards,
>>> >> Rishi
>>> >>
>>> >>
>>> >> Michael McCandless-2 wrote:
>>> >> >
>>> >> > Is there any chance that two writers are open on this directory?
>>> Or,
>>> >> > something external to Lucene is removing files from the directory.
>>> >> >
>>> >> > It looks like there were at least two missing files (_37
>>> >> >
>>> >> > On Thu, Aug 13, 2009 at 5:19 AM,
>>> rishisinghal<[EMAIL PROTECTED]
>>> >
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> I am trying to index documents and when all is complete and
>>> optimize
>>> >> is
>>> >> >> called I get
>>> >> >>
>>> >> >> IFD [main]: setInfoStream
>>> >> >>
>>> >>
>>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
>>> >> >> IW 0 [main]: setInfoStream:
>>> >> >>
>>> >>
>>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
>>> >> >> autoCommit=false mergePolicy=org
>>> >> >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
>>> >> >>
>>> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4
>>> >> >> ramBufferS
>>> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
>>> >> >> maxFieldLength=10000
>>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>>> >> >>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
>>> >> _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0

I am running this on OpenVMS V8.2-1 on IA64. For a small number of files this
works all fine.

I checked the resources part and i have enough disk and ram available.

Regards,
Rishi
Michael McCandless-2 wrote:
>
> It's very odd that CheckIndex has no trouble opening the segment's
> files, yet when you run optimize the OS reports a "file not found"
> exception (errno 5).  Something odd is happening at the OS/filesystem
> level.
>
> What OS are you running on?
>
> Can you boil this down to a smallish standalone test that shows the
> problem?
>
> Mike
>
> On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<[EMAIL PROTECTED]>
> wrote:
>>
>> I tried creating the index in different disks but still i see the issue
>> :-(
>>
>> I tried to index documents in other disks also and got the same
>> exception.
>>
>> I also tried
>> $ java org.apache.lucene.index.CheckIndex
>> /SYS$SYSDEVICE/RISHI/melon_1600/
>> -segment _61
>> NOTE: testing will be more thorough if you run java with
>> '-ea:org.apache.lucene...', so assertions are enabled
>> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
>> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
>> 2.4]
>>
>> Checking only these segments: _61:
>> No problems were detected with this index.
>>
>> Regards,
>> Rishi
>>
>>
>> Shai Erera wrote:
>>>
>>> I noticed the exception is "Caused by: java.io.FileNotFoundException:
>>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
>>>
>>> I searched for i/o error (errno:5) and found some information which
>>> associates it w/ a more native IO problem, like corrupt file due to
>>> system
>>> crash etc.
>>>
>>> Did you experience that? Can you try to create the index somewhere else,
>>> or
>>> on another drive?
>>>
>>> Shai
>>>
>>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>>
>>>> It is a local file system.
>>>>
>>>> We are using lucene 2.4 and java 1.5
>>>>
>>>> Regards,
>>>> Rishi
>>>>
>>>>
>>>> Shai Erera wrote:
>>>> >
>>>> > Is that a local file system, or a network share?
>>>> >
>>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
>>>> > <[EMAIL PROTECTED]>wrote:
>>>> >
>>>> >>
>>>> >> >>Is there any chance that two writers are open on this directory?
>>>> >> No, thats not true.
>>>> >>
>>>> >> >>something external to Lucene is removing files from the directory.
>>>> >> No this also has rare chances as I am the owner of these files and
>>>> other
>>>> >> then me no one can delete the, :-)
>>>> >>
>>>> >> Here are all the files in the index directory:
>>>> >> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
>>>> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1          
>>>> _61.cfs;1
>>>> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1          
>>>> _b0.cfs;1
>>>> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
>>>> >>
>>>> >> I see _37 present.
>>>> >>
>>>> >> Regards,
>>>> >> Rishi
>>>> >>
>>>> >>
>>>> >> Michael McCandless-2 wrote:
>>>> >> >
>>>> >> > Is there any chance that two writers are open on this directory?
>>>> Or,
>>>> >> > something external to Lucene is removing files from the directory.
>>>> >> >
>>>> >> > It looks like there were at least two missing files (_37
>>>> >> >
>>>> >> > On Thu, Aug 13, 2009 at 5:19 AM,
>>>> rishisinghal<[EMAIL PROTECTED]
>>>> >
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> I am trying to index documents and when all is complete and
>>>> optimize
>>>> >> is
>>>> >> >> called I get
>>>> >> >>
>>>> >> >> IFD [main]: setInfoStream
>>>> >> >>
>>>> >>
>>>> deletionPolicy=org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy@4fced0
>>>> >> >> IW 0 [main]: setInfoStream:
>>>> >> >>
>>>> >>
>>>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
>>>> >> >> autoCommit=false mergePolicy=org
>>>> >> >> .apache.lucene.index.LogByteSizeMergePolicy@92cb49
>>>> >> >>
>>>> mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler@6038e4

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

If it's easily reproducible, can you please provide a test that reproduces
it, even if on just your FS? And also, can you turn on infoStream on
IndexWriter and post the output?

If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.

Shai

On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal <[EMAIL PROTECTED]>wrote:

>
> I am running this on OpenVMS V8.2-1 on IA64. For a small number of files
> this
> works all fine.
>
> I checked the resources part and i have enough disk and ram available.
>
> Regards,
> Rishi
>
>
> Michael McCandless-2 wrote:
> >
> > It's very odd that CheckIndex has no trouble opening the segment's
> > files, yet when you run optimize the OS reports a "file not found"
> > exception (errno 5).  Something odd is happening at the OS/filesystem
> > level.
> >
> > What OS are you running on?
> >
> > Can you boil this down to a smallish standalone test that shows the
> > problem?
> >
> > Mike
> >
> > On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<[EMAIL PROTECTED]>
> > wrote:
> >>
> >> I tried creating the index in different disks but still i see the issue
> >> :-(
> >>
> >> I tried to index documents in other disks also and got the same
> >> exception.
> >>
> >> I also tried
> >> $ java org.apache.lucene.index.CheckIndex
> >> /SYS$SYSDEVICE/RISHI/melon_1600/
> >> -segment _61
> >> NOTE: testing will be more thorough if you run java with
> >> '-ea:org.apache.lucene...', so assertions are enabled
> >> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
> >> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
> >> 2.4]
> >>
> >> Checking only these segments: _61:
> >> No problems were detected with this index.
> >>
> >> Regards,
> >> Rishi
> >>
> >>
> >> Shai Erera wrote:
> >>>
> >>> I noticed the exception is "Caused by: java.io.FileNotFoundException:
> >>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
> >>>
> >>> I searched for i/o error (errno:5) and found some information which
> >>> associates it w/ a more native IO problem, like corrupt file due to
> >>> system
> >>> crash etc.
> >>>
> >>> Did you experience that? Can you try to create the index somewhere
> else,
> >>> or
> >>> on another drive?
> >>>
> >>> Shai
> >>>
> >>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
> >>> <[EMAIL PROTECTED]>wrote:
> >>>
> >>>>
> >>>> It is a local file system.
> >>>>
> >>>> We are using lucene 2.4 and java 1.5
> >>>>
> >>>> Regards,
> >>>> Rishi
> >>>>
> >>>>
> >>>> Shai Erera wrote:
> >>>> >
> >>>> > Is that a local file system, or a network share?
> >>>> >
> >>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
> >>>> > <[EMAIL PROTECTED]>wrote:
> >>>> >
> >>>> >>
> >>>> >> >>Is there any chance that two writers are open on this directory?
> >>>> >> No, thats not true.
> >>>> >>
> >>>> >> >>something external to Lucene is removing files from the
> directory.
> >>>> >> No this also has rare chances as I am the owner of these files and
> >>>> other
> >>>> >> then me no one can delete the, :-)
> >>>> >>
> >>>> >> Here are all the files in the index directory:
> >>>> >> segments.gen;1      segments_1.;1       segments_2.;1
> _0.fdt;1
> >>>> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1
> >>>> _61.cfs;1
> >>>> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1
> >>>> _b0.cfs;1
> >>>> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
> >>>> >>
> >>>> >> I see _37 present.
> >>>> >>
> >>>> >> Regards,
> >>>> >> Rishi
> >>>> >>
> >>>> >>
> >>>> >> Michael McCandless-2 wrote:
> >>>> >> >
> >>>> >> > Is there any chance that two writers are open on this directory?
> >>>> Or,
> >>>> >> > something external to Lucene is removing files from the
> directory.
> >>>> >> >
> >>>> >> > It looks like there were at least two missing files (_37
> >>>> >> >
> >>>> >> > On Thu, Aug 13, 2009 at 5:19 AM,
> >>>> rishisinghal<[EMAIL PROTECTED]
> >>>> >
> >>>> >> > wrote:
> >>>> >> >>
> >>>> >> >> Hi,
> >>>> >> >>
> >>>> >> >> I am trying to index documents and when all is complete and

It is reproducible in my system. If the number of files are small I am not
able to see the crash.

>>can you turn on infoStream on IndexWriter and post the output?
I am not clear on this. If you are talking of
writer.setInfoStream(System.out);
I already have posted the output while raising the issue.
>>If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.
I am not clear on this too. As I know CheckIndex program has been run after
I get the error and it shows no issues.

Regards,
Rishi

Shai Erera wrote:
>
> If it's easily reproducible, can you please provide a test that reproduces
> it, even if on just your FS? And also, can you turn on infoStream on
> IndexWriter and post the output?
>
> If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.
>
> Shai
>
> On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
> <[EMAIL PROTECTED]>wrote:
>
>>
>> I am running this on OpenVMS V8.2-1 on IA64. For a small number of files
>> this
>> works all fine.
>>
>> I checked the resources part and i have enough disk and ram available.
>>
>> Regards,
>> Rishi
>>
>>
>> Michael McCandless-2 wrote:
>> >
>> > It's very odd that CheckIndex has no trouble opening the segment's
>> > files, yet when you run optimize the OS reports a "file not found"
>> > exception (errno 5).  Something odd is happening at the OS/filesystem
>> > level.
>> >
>> > What OS are you running on?
>> >
>> > Can you boil this down to a smallish standalone test that shows the
>> > problem?
>> >
>> > Mike
>> >
>> > On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<[EMAIL PROTECTED]>
>> > wrote:
>> >>
>> >> I tried creating the index in different disks but still i see the
>> issue
>> >> :-(
>> >>
>> >> I tried to index documents in other disks also and got the same
>> >> exception.
>> >>
>> >> I also tried
>> >> $ java org.apache.lucene.index.CheckIndex
>> >> /SYS$SYSDEVICE/RISHI/melon_1600/
>> >> -segment _61
>> >> NOTE: testing will be more thorough if you run java with
>> >> '-ea:org.apache.lucene...', so assertions are enabled
>> >> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
>> >> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
>> >> 2.4]
>> >>
>> >> Checking only these segments: _61:
>> >> No problems were detected with this index.
>> >>
>> >> Regards,
>> >> Rishi
>> >>
>> >>
>> >> Shai Erera wrote:
>> >>>
>> >>> I noticed the exception is "Caused by: java.io.FileNotFoundException:
>> >>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
>> >>>
>> >>> I searched for i/o error (errno:5) and found some information which
>> >>> associates it w/ a more native IO problem, like corrupt file due to
>> >>> system
>> >>> crash etc.
>> >>>
>> >>> Did you experience that? Can you try to create the index somewhere
>> else,
>> >>> or
>> >>> on another drive?
>> >>>
>> >>> Shai
>> >>>
>> >>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
>> >>> <[EMAIL PROTECTED]>wrote:
>> >>>
>> >>>>
>> >>>> It is a local file system.
>> >>>>
>> >>>> We are using lucene 2.4 and java 1.5
>> >>>>
>> >>>> Regards,
>> >>>> Rishi
>> >>>>
>> >>>>
>> >>>> Shai Erera wrote:
>> >>>> >
>> >>>> > Is that a local file system, or a network share?
>> >>>> >
>> >>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
>> >>>> > <[EMAIL PROTECTED]>wrote:
>> >>>> >
>> >>>> >>
>> >>>> >> >>Is there any chance that two writers are open on this
>> directory?
>> >>>> >> No, thats not true.
>> >>>> >>
>> >>>> >> >>something external to Lucene is removing files from the
>> directory.
>> >>>> >> No this also has rare chances as I am the owner of these files
>> and
>> >>>> other
>> >>>> >> then me no one can delete the, :-)
>> >>>> >>
>> >>>> >> Here are all the files in the index directory:
>> >>>> >> segments.gen;1      segments_1.;1       segments_2.;1
>> _0.fdt;1
>> >>>> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1
>> >>>> _61.cfs;1
>> >>>> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1
>> >>>> _b0.cfs;1
>> >>>> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970274.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

Can you post the full sources for the test that hits the exception on
your system?

Mike

On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<[EMAIL PROTECTED]> wrote:
>
> It is reproducible in my system. If the number of files are small I am not
> able to see the crash.
>
>>>can you turn on infoStream on IndexWriter and post the output?
> I am not clear on this. If you are talking of
> writer.setInfoStream(System.out);
> I already have posted the output while raising the issue.
>
>
>>>If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.
> I am not clear on this too. As I know CheckIndex program has been run after
> I get the error and it shows no issues.
>
> Regards,
> Rishi
>
>
>
> Shai Erera wrote:
>>
>> If it's easily reproducible, can you please provide a test that reproduces
>> it, even if on just your FS? And also, can you turn on infoStream on
>> IndexWriter and post the output?
>>
>> If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex.
>>
>> Shai
>>
>> On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
>> <[EMAIL PROTECTED]>wrote:
>>
>>>
>>> I am running this on OpenVMS V8.2-1 on IA64. For a small number of files
>>> this
>>> works all fine.
>>>
>>> I checked the resources part and i have enough disk and ram available.
>>>
>>> Regards,
>>> Rishi
>>>
>>>
>>> Michael McCandless-2 wrote:
>>> >
>>> > It's very odd that CheckIndex has no trouble opening the segment's
>>> > files, yet when you run optimize the OS reports a "file not found"
>>> > exception (errno 5).  Something odd is happening at the OS/filesystem
>>> > level.
>>> >
>>> > What OS are you running on?
>>> >
>>> > Can you boil this down to a smallish standalone test that shows the
>>> > problem?
>>> >
>>> > Mike
>>> >
>>> > On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<[EMAIL PROTECTED]>
>>> > wrote:
>>> >>
>>> >> I tried creating the index in different disks but still i see the
>>> issue
>>> >> :-(
>>> >>
>>> >> I tried to index documents in other disks also and got the same
>>> >> exception.
>>> >>
>>> >> I also tried
>>> >> $ java org.apache.lucene.index.CheckIndex
>>> >> /SYS$SYSDEVICE/RISHI/melon_1600/
>>> >> -segment _61
>>> >> NOTE: testing will be more thorough if you run java with
>>> >> '-ea:org.apache.lucene...', so assertions are enabled
>>> >> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
>>> >> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene
>>> >> 2.4]
>>> >>
>>> >> Checking only these segments: _61:
>>> >> No problems were detected with this index.
>>> >>
>>> >> Regards,
>>> >> Rishi
>>> >>
>>> >>
>>> >> Shai Erera wrote:
>>> >>>
>>> >>> I noticed the exception is "Caused by: java.io.FileNotFoundException:
>>> >>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
>>> >>>
>>> >>> I searched for i/o error (errno:5) and found some information which
>>> >>> associates it w/ a more native IO problem, like corrupt file due to
>>> >>> system
>>> >>> crash etc.
>>> >>>
>>> >>> Did you experience that? Can you try to create the index somewhere
>>> else,
>>> >>> or
>>> >>> on another drive?
>>> >>>
>>> >>> Shai
>>> >>>
>>> >>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
>>> >>> <[EMAIL PROTECTED]>wrote:
>>> >>>
>>> >>>>
>>> >>>> It is a local file system.
>>> >>>>
>>> >>>> We are using lucene 2.4 and java 1.5
>>> >>>>
>>> >>>> Regards,
>>> >>>> Rishi
>>> >>>>
>>> >>>>
>>> >>>> Shai Erera wrote:
>>> >>>> >
>>> >>>> > Is that a local file system, or a network share?
>>> >>>> >
>>> >>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
>>> >>>> > <[EMAIL PROTECTED]>wrote:
>>> >>>> >
>>> >>>> >>
>>> >>>> >> >>Is there any chance that two writers are open on this
>>> directory?
>>> >>>> >> No, thats not true.
>>> >>>> >>
>>> >>>> >> >>something external to Lucene is removing files from the
>>> directory.
>>> >>>> >> No this also has rare chances as I am the owner of these files
>>> and
>>> >>>> other
>>> >>>> >> then me no one can delete the, :-)
>>> >>>> >>

This is a big project having lots of source files.  The excerpt of the main
indexer program  is:
FileIndexer indexer = new FileIndexer(props);

Analyzer defAnalyzer = new StandardAnalyzer();
FileIndexer.analyzer = defAnalyzer;
IndexWriter writer = new IndexWriter(dir,FileIndexer.analyzer, true,
IndexWriter.MaxFieldLength.LIMITED);
writer.setMaxBufferedDocs(5);
//writer.setMaxMergeDocs(50);
writer.setMergeFactor(5);

long start = new Date().getTime();
try
{
for (int i=0; i<disks.size();i++)
{
index(writer, new File(disks.get(i)),filelist, FileIndexer.analyzer);

}
writer.setInfoStream(System.out);
writer.optimize();
}
catch(Exception e)
{
e.printStackTrace();
}
finally
{

writer.close();
}
Michael McCandless-2 wrote:
>
> Can you post the full sources for the test that hits the exception on
> your system?
>
> Mike
>
> On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<[EMAIL PROTECTED]>
> wrote:
>>
>> It is reproducible in my system. If the number of files are small I am
>> not
>> able to see the crash.
>>
>>>>can you turn on infoStream on IndexWriter and post the output?
>> I am not clear on this. If you are talking of
>> writer.setInfoStream(System.out);
>> I already have posted the output while raising the issue.
>>
>>
>>>>If CheckIndex succeeds, then I'd compare the output w/ and w/o
CheckIndex.
>> I am not clear on this too. As I know CheckIndex program has been run
>> after
>> I get the error and it shows no issues.
>>
>> Regards,
>> Rishi
>>
>>
>>
>> Shai Erera wrote:
>>>
>>> If it's easily reproducible, can you please provide a test that
>>> reproduces
>>> it, even if on just your FS? And also, can you turn on infoStream on
>>> IndexWriter and post the output?
>>>
>>> If CheckIndex succeeds, then I'd compare the output w/ and w/o
>>> CheckIndex.
>>>
>>> Shai
>>>
>>> On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
>>> <[EMAIL PROTECTED]>wrote:
>>>
>>>>
>>>> I am running this on OpenVMS V8.2-1 on IA64. For a small number of
>>>> files
>>>> this
>>>> works all fine.
>>>>
>>>> I checked the resources part and i have enough disk and ram available.
>>>>
>>>> Regards,
>>>> Rishi
>>>>
>>>>
>>>> Michael McCandless-2 wrote:
>>>> >
>>>> > It's very odd that CheckIndex has no trouble opening the segment's
>>>> > files, yet when you run optimize the OS reports a "file not found"
>>>> > exception (errno 5).  Something odd is happening at the OS/filesystem
>>>> > level.
>>>> >
>>>> > What OS are you running on?
>>>> >
>>>> > Can you boil this down to a smallish standalone test that shows the
>>>> > problem?
>>>> >
>>>> > Mike
>>>> >
>>>> > On Thu, Aug 13, 2009 at 8:21 AM,
>>>> rishisinghal<[EMAIL PROTECTED]>
>>>> > wrote:
>>>> >>
>>>> >> I tried creating the index in different disks but still i see the
>>>> issue
>>>> >> :-(
>>>> >>
>>>> >> I tried to index documents in other disks also and got the same
>>>> >> exception.
>>>> >>
>>>> >> I also tried
>>>> >> $ java org.apache.lucene.index.CheckIndex
>>>> >> /SYS$SYSDEVICE/RISHI/melon_1600/
>>>> >> -segment _61
>>>> >> NOTE: testing will be more thorough if you run java with
>>>> >> '-ea:org.apache.lucene...', so assertions are enabled
>>>> >> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
>>>> >> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX
>>>> [Lucene
>>>> >> 2.4]
>>>> >>
>>>> >> Checking only these segments: _61:
>>>> >> No problems were detected with this index.
>>>> >>
>>>> >> Regards,
>>>> >> Rishi
>>>> >>
>>>> >>
>>>> >> Shai Erera wrote:
>>>> >>>
>>>> >>> I noticed the exception is "Caused by:
>>>> java.io.FileNotFoundException:
>>>> >>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
>>>> >>>
>>>> >>> I searched for i/o error (errno:5) and found some information which
>>>> >>> associates it w/ a more native IO problem, like corrupt file due to
>>>> >>> system
>>>> >>> crash etc.
>>>> >>>
>>>> >>> Did you experience that? Can you try to create the index somewhere

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24970750.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

But, if you can break out just the Lucene indexing part into a
standalone test, that shows the exception, that can help us isolate
it.

At this point it seems likely the issue is something OS/filesystem
specific, because at one time (during optimize), we see a "file not
found" exception, yet at another time (running CheckIndex) the same
file is opened just fine.

Mike

On Fri, Aug 14, 2009 at 8:13 AM, rishisinghal<[EMAIL PROTECTED]> wrote:
>
> This is a big project having lots of source files.  The excerpt of the main
> indexer program  is:
> FileIndexer indexer = new FileIndexer(props);
>
>                Analyzer defAnalyzer = new StandardAnalyzer();
>                FileIndexer.analyzer = defAnalyzer;
>                IndexWriter writer = new IndexWriter(dir,FileIndexer.analyzer, true,
> IndexWriter.MaxFieldLength.LIMITED);
>                writer.setMaxBufferedDocs(5);
>                //writer.setMaxMergeDocs(50);
>                writer.setMergeFactor(5);
>
>                long start = new Date().getTime();
>                try
>                {
>                for (int i=0; i<disks.size();i++)
>                {
>                index(writer, new File(disks.get(i)),filelist, FileIndexer.analyzer);
>
>                }
>                writer.setInfoStream(System.out);
>                writer.optimize();
>                }
>                catch(Exception e)
>                {
>                                e.printStackTrace();
>                }
>                finally
>                {
>
>                        writer.close();
>                }
>
>
>
>
>
>
> Michael McCandless-2 wrote:
>>
>> Can you post the full sources for the test that hits the exception on
>> your system?
>>
>> Mike
>>
>> On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<[EMAIL PROTECTED]>
>> wrote:
>>>
>>> It is reproducible in my system. If the number of files are small I am
>>> not
>>> able to see the crash.
>>>
>>>>>can you turn on infoStream on IndexWriter and post the output?
>>> I am not clear on this. If you are talking of
>>> writer.setInfoStream(System.out);
>>> I already have posted the output while raising the issue.
>>>
>>>
>>>>>If CheckIndex succeeds, then I'd compare the output w/ and w/o
> CheckIndex.
>>> I am not clear on this too. As I know CheckIndex program has been run
>>> after
>>> I get the error and it shows no issues.
>>>
>>> Regards,
>>> Rishi
>>>
>>>
>>>
>>> Shai Erera wrote:
>>>>
>>>> If it's easily reproducible, can you please provide a test that
>>>> reproduces
>>>> it, even if on just your FS? And also, can you turn on infoStream on
>>>> IndexWriter and post the output?
>>>>
>>>> If CheckIndex succeeds, then I'd compare the output w/ and w/o
>>>> CheckIndex.
>>>>
>>>> Shai
>>>>
>>>> On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
>>>> <[EMAIL PROTECTED]>wrote:
>>>>
>>>>>
>>>>> I am running this on OpenVMS V8.2-1 on IA64. For a small number of
>>>>> files
>>>>> this
>>>>> works all fine.
>>>>>
>>>>> I checked the resources part and i have enough disk and ram available.
>>>>>
>>>>> Regards,
>>>>> Rishi
>>>>>
>>>>>
>>>>> Michael McCandless-2 wrote:
>>>>> >
>>>>> > It's very odd that CheckIndex has no trouble opening the segment's
>>>>> > files, yet when you run optimize the OS reports a "file not found"
>>>>> > exception (errno 5).  Something odd is happening at the OS/filesystem
>>>>> > level.
>>>>> >
>>>>> > What OS are you running on?
>>>>> >
>>>>> > Can you boil this down to a smallish standalone test that shows the
>>>>> > problem?
>>>>> >
>>>>> > Mike
>>>>> >
>>>>> > On Thu, Aug 13, 2009 at 8:21 AM,
>>>>> rishisinghal<[EMAIL PROTECTED]>
>>>>> > wrote:
>>>>> >>
>>>>> >> I tried creating the index in different disks but still i see the
>>>>> issue
>>>>> >> :-(
>>>>> >>
>>>>> >> I tried to index documents in other disks also and got the same
>>>>> >> exception.
>>>>> >>
>>>>> >> I also tried
>>>>> >> $ java org.apache.lucene.index.CheckIndex
>>>>> >> /SYS$SYSDEVICE/RISHI/melon_1600/

Okay let me try if i can create a standalone test which reproduces this issue
and get back.

Regards,
Rishi
Michael McCandless-2 wrote:
>
> But, if you can break out just the Lucene indexing part into a
> standalone test, that shows the exception, that can help us isolate
> it.
>
> At this point it seems likely the issue is something OS/filesystem
> specific, because at one time (during optimize), we see a "file not
> found" exception, yet at another time (running CheckIndex) the same
> file is opened just fine.
>
> Mike
>
> On Fri, Aug 14, 2009 at 8:13 AM, rishisinghal<[EMAIL PROTECTED]>
> wrote:
>>
>> This is a big project having lots of source files.  The excerpt of the
>> main
>> indexer program  is:
>> FileIndexer indexer = new FileIndexer(props);
>>
>>                Analyzer defAnalyzer = new StandardAnalyzer();
>>                FileIndexer.analyzer = defAnalyzer;
>>                IndexWriter writer = new
>> IndexWriter(dir,FileIndexer.analyzer, true,
>> IndexWriter.MaxFieldLength.LIMITED);
>>                writer.setMaxBufferedDocs(5);
>>                //writer.setMaxMergeDocs(50);
>>                writer.setMergeFactor(5);
>>
>>                long start = new Date().getTime();
>>                try
>>                {
>>                for (int i=0; i<disks.size();i++)
>>                {
>>                index(writer, new File(disks.get(i)),filelist,
>> FileIndexer.analyzer);
>>
>>                }
>>                writer.setInfoStream(System.out);
>>                writer.optimize();
>>                }
>>                catch(Exception e)
>>                {
>>                                e.printStackTrace();
>>                }
>>                finally
>>                {
>>
>>                        writer.close();
>>                }
>>
>>
>>
>>
>>
>>
>> Michael McCandless-2 wrote:
>>>
>>> Can you post the full sources for the test that hits the exception on
>>> your system?
>>>
>>> Mike
>>>
>>> On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<[EMAIL PROTECTED]>
>>> wrote:
>>>>
>>>> It is reproducible in my system. If the number of files are small I am
>>>> not
>>>> able to see the crash.
>>>>
>>>>>>can you turn on infoStream on IndexWriter and post the output?
>>>> I am not clear on this. If you are talking of
>>>> writer.setInfoStream(System.out);
>>>> I already have posted the output while raising the issue.
>>>>
>>>>
>>>>>>If CheckIndex succeeds, then I'd compare the output w/ and w/o
>> CheckIndex.
>>>> I am not clear on this too. As I know CheckIndex program has been run
>>>> after
>>>> I get the error and it shows no issues.
>>>>
>>>> Regards,
>>>> Rishi
>>>>
>>>>
>>>>
>>>> Shai Erera wrote:
>>>>>
>>>>> If it's easily reproducible, can you please provide a test that
>>>>> reproduces
>>>>> it, even if on just your FS? And also, can you turn on infoStream on
>>>>> IndexWriter and post the output?
>>>>>
>>>>> If CheckIndex succeeds, then I'd compare the output w/ and w/o
>>>>> CheckIndex.
>>>>>
>>>>> Shai
>>>>>
>>>>> On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
>>>>> <[EMAIL PROTECTED]>wrote:
>>>>>
>>>>>>
>>>>>> I am running this on OpenVMS V8.2-1 on IA64. For a small number of
>>>>>> files
>>>>>> this
>>>>>> works all fine.
>>>>>>
>>>>>> I checked the resources part and i have enough disk and ram
>>>>>> available.
>>>>>>
>>>>>> Regards,
>>>>>> Rishi
>>>>>>
>>>>>>
>>>>>> Michael McCandless-2 wrote:
>>>>>> >
>>>>>> > It's very odd that CheckIndex has no trouble opening the segment's
>>>>>> > files, yet when you run optimize the OS reports a "file not found"
>>>>>> > exception (errno 5).  Something odd is happening at the
>>>>>> OS/filesystem
>>>>>> > level.
>>>>>> >
>>>>>> > What OS are you running on?
>>>>>> >
>>>>>> > Can you boil this down to a smallish standalone test that shows the
>>>>>> > problem?
>>>>>> >
>>>>>> > Mike
>>>>>> >
>>>>>> > On Thu, Aug 13, 2009 at 8:21 AM,
>>>>>> rishisinghal<[EMAIL PROTECTED]>
>>>>>> > wrote:
>>>>>> >>
>>>>>> >> I tried creating the index in different disks but still i see the

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24971223.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.

hmm i tried to create a small reproducer but was not successful to reproduce
this. For less number of files my application also works fine. I tried with
many different individual directories and it runs fine. Only when the
directory structure is huge i get the io error.

So i have no idea what is causing this.

Regards,
Rishi
rishisinghal wrote:
>
> Okay let me try if i can create a standalone test which reproduces this
> issue and get back.
>
> Regards,
> Rishi
>
>
> Michael McCandless-2 wrote:
>>
>> But, if you can break out just the Lucene indexing part into a
>> standalone test, that shows the exception, that can help us isolate
>> it.
>>
>> At this point it seems likely the issue is something OS/filesystem
>> specific, because at one time (during optimize), we see a "file not
>> found" exception, yet at another time (running CheckIndex) the same
>> file is opened just fine.
>>
>> Mike
>>
>> On Fri, Aug 14, 2009 at 8:13 AM, rishisinghal<[EMAIL PROTECTED]>
>> wrote:
>>>
>>> This is a big project having lots of source files.  The excerpt of the
>>> main
>>> indexer program  is:
>>> FileIndexer indexer = new FileIndexer(props);
>>>
>>>                Analyzer defAnalyzer = new StandardAnalyzer();
>>>                FileIndexer.analyzer = defAnalyzer;
>>>                IndexWriter writer = new
>>> IndexWriter(dir,FileIndexer.analyzer, true,
>>> IndexWriter.MaxFieldLength.LIMITED);
>>>                writer.setMaxBufferedDocs(5);
>>>                //writer.setMaxMergeDocs(50);
>>>                writer.setMergeFactor(5);
>>>
>>>                long start = new Date().getTime();
>>>                try
>>>                {
>>>                for (int i=0; i<disks.size();i++)
>>>                {
>>>                index(writer, new File(disks.get(i)),filelist,
>>> FileIndexer.analyzer);
>>>
>>>                }
>>>                writer.setInfoStream(System.out);
>>>                writer.optimize();
>>>                }
>>>                catch(Exception e)
>>>                {
>>>                                e.printStackTrace();
>>>                }
>>>                finally
>>>                {
>>>
>>>                        writer.close();
>>>                }
>>>
>>>
>>>
>>>
>>>
>>>
>>> Michael McCandless-2 wrote:
>>>>
>>>> Can you post the full sources for the test that hits the exception on
>>>> your system?
>>>>
>>>> Mike
>>>>
>>>> On Fri, Aug 14, 2009 at 7:36 AM, rishisinghal<[EMAIL PROTECTED]>
>>>> wrote:
>>>>>
>>>>> It is reproducible in my system. If the number of files are small I am
>>>>> not
>>>>> able to see the crash.
>>>>>
>>>>>>>can you turn on infoStream on IndexWriter and post the output?
>>>>> I am not clear on this. If you are talking of
>>>>> writer.setInfoStream(System.out);
>>>>> I already have posted the output while raising the issue.
>>>>>
>>>>>
>>>>>>>If CheckIndex succeeds, then I'd compare the output w/ and w/o
>>> CheckIndex.
>>>>> I am not clear on this too. As I know CheckIndex program has been run
>>>>> after
>>>>> I get the error and it shows no issues.
>>>>>
>>>>> Regards,
>>>>> Rishi
>>>>>
>>>>>
>>>>>
>>>>> Shai Erera wrote:
>>>>>>
>>>>>> If it's easily reproducible, can you please provide a test that
>>>>>> reproduces
>>>>>> it, even if on just your FS? And also, can you turn on infoStream on
>>>>>> IndexWriter and post the output?
>>>>>>
>>>>>> If CheckIndex succeeds, then I'd compare the output w/ and w/o
>>>>>> CheckIndex.
>>>>>>
>>>>>> Shai
>>>>>>
>>>>>> On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal
>>>>>> <[EMAIL PROTECTED]>wrote:
>>>>>>
>>>>>>>
>>>>>>> I am running this on OpenVMS V8.2-1 on IA64. For a small number of
>>>>>>> files
>>>>>>> this
>>>>>>> works all fine.
>>>>>>>
>>>>>>> I checked the resources part and i have enough disk and ram
>>>>>>> available.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Rishi
>>>>>>>
>>>>>>>
>>>>>>> Michael McCandless-2 wrote:
>>>>>>> >
>>>>>>> > It's very odd that CheckIndex has no trouble opening the segment's

View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24996067.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.