Hi all,

Pretty new to nutch.  Trying to create a setup where nutch repeatedly
crawls a selected set of webpages, to feed the content into a pipeline for
text analysis etc. (e.g. Nutch, Tika, GATE, ...)

We are unclear about what setup/version/approach to use for this.   To be
honest, the plethora of snippets of (outdated?) docs don't help in getting
a clear view on things.

The major hurdle seems to be the flexible access to the crawled content.
 Both from a search (mentions of certain words) as from a systematic (e.g.
database queries to process pages in batch) point of view.
Next to solr queries, the only way seems dumping the segments with the
SegmentReader, and processing those.
But access to the segments seems cumbersome and not very flexible to
integrate into a larger setup.  And slow.

I was happy to see the GORA access to e.g. MySQL in Nutch 2.0, but now that
seems to all have been side-tracked.  I got crawled pages in MySQL in 15
minutes, which is great !  I don't see what the alternative for a setup
like that is in Nutch 1.4 ?

Alternatives to write to MySQL from Nutch 1.4 seem less straightforward as
mentioned (extending nutch where the NutchPage gets written to SOLR and
diverting to MySQL .. ?  There must be a better way.)

Could somebody with some experience in these kinds of setups advise in what
direction we should consider going ?

I would like a flexible setup, where nutch can run continuously, being fed
with new seed URLs through time, and flexible and efficient access to the
crawled results to integrate this in a larger setup.

thanks !

pvremort

Behemoth [1] eats Nutch 1.x segments and can push them a.o. to GATE. Nutch
comes with its own Tika parser.

[1]: https://github.com/jnioche/behemoth

cheers

On Friday 09 March 2012 16:19:03 Piet van Remortel wrote:
> Hi all,
>
> Pretty new to nutch.  Trying to create a setup where nutch repeatedly
> crawls a selected set of webpages, to feed the content into a pipeline for
> text analysis etc. (e.g. Nutch, Tika, GATE, ...)
>
> We are unclear about what setup/version/approach to use for this.   To be
> honest, the plethora of snippets of (outdated?) docs don't help in getting
> a clear view on things.
>
> The major hurdle seems to be the flexible access to the crawled content.
>  Both from a search (mentions of certain words) as from a systematic (e.g.
> database queries to process pages in batch) point of view.
> Next to solr queries, the only way seems dumping the segments with the
> SegmentReader, and processing those.
> But access to the segments seems cumbersome and not very flexible to
> integrate into a larger setup.  And slow.
>
> I was happy to see the GORA access to e.g. MySQL in Nutch 2.0, but now that
> seems to all have been side-tracked.  I got crawled pages in MySQL in 15
> minutes, which is great !  I don't see what the alternative for a setup
> like that is in Nutch 1.4 ?
>
> Alternatives to write to MySQL from Nutch 1.4 seem less straightforward as
> mentioned (extending nutch where the NutchPage gets written to SOLR and
> diverting to MySQL .. ?  There must be a better way.)
>
> Could somebody with some experience in these kinds of setups advise in what
> direction we should consider going ?
>
> I would like a flexible setup, where nutch can run continuously, being fed
> with new seed URLs through time, and flexible and efficient access to the
> crawled results to integrate this in a larger setup.
>
> thanks !
>
> pvremort

--
Markus Jelsma - CTO - Openindex

Dear Piet,

First, you're absolutely right about the state of the documentation. We have to deal with this in the near future.
Now, although nutchgora currently is developed on a branch, it is actually still alive and kicking. More, a first Nutch 2.0 release, based on the nutchgora branch, is in the pipeline.
We are using the nutchgora branch with HBase on a multi-node Hadoop cluster. Not experimental, but for a real world customer.

If you had good experiences with nutch+gora+mysql in the past: just checkout the nutchgora head or wait until the release.

Mathijs

 

On Mar 9, 2012, at 16:19 , Piet van Remortel wrote:

> Hi all,
>
> Pretty new to nutch.  Trying to create a setup where nutch repeatedly
> crawls a selected set of webpages, to feed the content into a pipeline for
> text analysis etc. (e.g. Nutch, Tika, GATE, ...)
>
> We are unclear about what setup/version/approach to use for this.   To be
> honest, the plethora of snippets of (outdated?) docs don't help in getting
> a clear view on things.
>
> The major hurdle seems to be the flexible access to the crawled content.
> Both from a search (mentions of certain words) as from a systematic (e.g.
> database queries to process pages in batch) point of view.
> Next to solr queries, the only way seems dumping the segments with the
> SegmentReader, and processing those.
> But access to the segments seems cumbersome and not very flexible to
> integrate into a larger setup.  And slow.
>
> I was happy to see the GORA access to e.g. MySQL in Nutch 2.0, but now that
> seems to all have been side-tracked.  I got crawled pages in MySQL in 15
> minutes, which is great !  I don't see what the alternative for a setup
> like that is in Nutch 1.4 ?
>
> Alternatives to write to MySQL from Nutch 1.4 seem less straightforward as
> mentioned (extending nutch where the NutchPage gets written to SOLR and
> diverting to MySQL .. ?  There must be a better way.)
>
> Could somebody with some experience in these kinds of setups advise in what
> direction we should consider going ?
>
> I would like a flexible setup, where nutch can run continuously, being fed
> with new seed URLs through time, and flexible and efficient access to the
> crawled results to integrate this in a larger setup.
>
> thanks !
>
> pvremort