Hi all,

I need to write a plugin to process pages's content and decide whether to
accept the URL or not. I followed this tutorial
http://sujitpal.blogspot.com/2009/07/nutch-getting-my-feet-wet.html where
the author implements IndexingFilter, but I wanted to filter URLs in the
fetch time.
Which extention point (or points) should I implement to achieve that?
Thank you in advance, I have no experience with Nutch :(

Daniel Rodrigues

--
View this message in context: http://lucene.472066.n3.nabble.com/Choosing-the-correct-extension-point-tp3911074p3911074.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi Daniel,

Wouldn't regex-urlfilter work for this?

-sujit

On Apr 14, 2012, at 4:14 PM, kwimera wrote:

> Hi all,
>
> I need to write a plugin to process pages's content and decide whether to
> accept the URL or not. I followed this tutorial
> http://sujitpal.blogspot.com/2009/07/nutch-getting-my-feet-wet.html where
> the author implements IndexingFilter, but I wanted to filter URLs in the
> fetch time.
> Which extention point (or points) should I implement to achieve that?
> Thank you in advance, I have no experience with Nutch :(
>
> Daniel Rodrigues
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Choosing-the-correct-extension-point-tp3911074p3911074.html
> Sent from the Nutch - User mailing list archive at Nabble.com.

It works to filter URLs with regular expressions, but I need to process the
pages content. It's likely I need to use a parser and extend some fetch
filter but I'm not sure, I'm still a little confusing to know where to start
:S
Thanks for the reply

--
View this message in context: http://lucene.472066.n3.nabble.com/Choosing-the-correct-extension-point-tp3911074p3911103.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Sorry, didn't realize that, should have read your original post more closely. There is a patch here which addresses a scenario like yours, may help...

https://issues.apache.org/jira/browse/NUTCH-828

On Apr 14, 2012, at 4:31 PM, kwimera wrote:

> It works to filter URLs with regular expressions, but I need to process the
> pages content. It's likely I need to use a parser and extend some fetch
> filter but I'm not sure, I'm still a little confusing to know where to start
> :S
> Thanks for the reply
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Choosing-the-correct-extension-point-tp3911074p3911103.html
> Sent from the Nutch - User mailing list archive at Nabble.com.

Thanks Sujit Pal, that's a good starting point :)

--
View this message in context: http://lucene.472066.n3.nabble.com/Choosing-the-correct-extension-point-tp3911074p3911152.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi,

Probably I am just looking at the wrong place in the documentation but is
there once place where i can get a list of all the available filters. I
understand there is http://wiki.apache.org/nutch/AboutPlugins but then we
do not have something like a Fetch Filter there

Regards | Vikas
On Sun, Apr 15, 2012 at 5:39 AM, kmrz <[EMAIL PROTECTED]> wrote:

> Thanks Sujit Pal, that's a good starting point :)
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Choosing-the-correct-extension-point-tp3911074p3911152.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

Hi Vikas,
I've just updated the page you cited to cover all currently supported
extension points.

On Mon, Apr 16, 2012 at 12:49 PM, Vikas Hazrati <[EMAIL PROTECTED]> wrote:

> I understand there is http://wiki.apache.org/nutch/AboutPlugins but then
> we
> do not have something like a Fetch Filter there
>

Correct. I guess you would need to hack the current Fetcher implementation
for custom fetching implementation. Most common work requires custom
parsing and indexing capability. However I think there might be some
tickets open on the Jira regarding custom fetching.