Hi,

A friend from Hadoop told me someone in the list has code for spliting
the Lucene index, can anyone point me to the right place?

Thanks.

---------------------------------------------------------------------

howard chen wrote:
> Hi,
>
> A friend from Hadoop told me someone in the list has code for spliting
> the Lucene index, can anyone point me to the right place?

You probably refer to the emails we exchanged with Dennis Kubes - in
that case there was no index splitting involved, rather the body of
documents to be indexed was split into parts and then indexed separately
to form many smaller indexes.

True index splitter doesn't exist (yet), but it shouldn't be too
difficult to implement, just tedious - around 3 days of work ... Some
people on this list also contemplated a semi-splitter (also non-existent
yet), which splits the index only on segment boundaries - this should be
much easier to implement, as it's just a question of copying selected
segments into new places and re-creating "segments" files - although
this method is much less flexible than a true splitter.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

---------------------------------------------------------------------

On 12/8/06, Andrzej Bialecki <[EMAIL PROTECTED]> wrote:
> howard chen wrote:
> > Hi,
> >
> > A friend from Hadoop told me someone in the list has code for spliting
> > the Lucene index, can anyone point me to the right place?
>
> You probably refer to the emails we exchanged with Dennis Kubes - in
> that case there was no index splitting involved, rather the body of
> documents to be indexed was split into parts and then indexed separately
> to form many smaller indexes.
>

yes, i want this one, any more information?

thanks.

---------------------------------------------------------------------

howard chen wrote:
> On 12/8/06, Andrzej Bialecki <[EMAIL PROTECTED]> wrote:
>> howard chen wrote:
>> > Hi,
>> >
>> > A friend from Hadoop told me someone in the list has code for spliting
>> > the Lucene index, can anyone point me to the right place?
>>
>> You probably refer to the emails we exchanged with Dennis Kubes - in
>> that case there was no index splitting involved, rather the body of
>> documents to be indexed was split into parts and then indexed separately
>> to form many smaller indexes.
>>
>
> yes, i want this one, any more information?

Please check the recent archives of hadoop-dev.

--
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

---------------------------------------------------------------------

howard chen wrote:
> A friend from Hadoop told me someone in the list has code for spliting
> the Lucene index, can anyone point me to the right place?

I think they were referring to:

http://www.nabble.com/design-change-suggestion-t2769173.html

This is not a fully-general index splitter, but still may be useful.

Doug

---------------------------------------------------------------------

On 12/9/06, Doug Cutting <[EMAIL PROTECTED]> wrote:
> howard chen wrote:
> > A friend from Hadoop told me someone in the list has code for spliting
> > the Lucene index, can anyone point me to the right place?
>
> I think they were referring to:
>
> http://www.nabble.com/design-change-suggestion-t2769173.html
>
> This is not a fully-general index splitter, but still may be useful.
>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>

Hi Doug!

Can you suggest if using Hadoop + Lucene, how to make a simple
distributed indexing & searching program, i.e. what are the mapping /
reducing processes involved in both indexing abd searching?

Thanks.

---------------------------------------------------------------------

howard chen wrote:
> Can you suggest if using Hadoop + Lucene, how to make a simple
> distributed indexing & searching program, i.e. what are the mapping /
> reducing processes involved in both indexing abd searching?

There is not yet a universal, best practice for this.

Nutch provides an example of how to use Lucene for distributed indexing.
  Nutch's current distributed search implementation builds on Hadoop's
RPC mechanism, but is not based on Hadoop's MapReduce.

http://lucene.apache.org/nutch/apidocs/org/apache/nutch/searcher/DistributedSearch.html

There has been some discussion of MapReduce-based distributed search on
the Nutch lists, e.g.:

http://mail-archives.apache.org/mod_mbox/lucene-nutch-user/200604.mbox/%[EMAIL PROTECTED]%3E

I think Andrzej Bialecki has explored this approach some.

Another approach is to build a non-MapReduce-based system specifically
for supporting distributed search and indexing.  I started a discussion
about this a few months ago and hope to start work on this project
before long.

http://www.nabble.com/-PROPOSAL--index-server-project-tf2469695.html

Doug

---------------------------------------------------------------------