Is it possible to crawl yahoo answer? Below is an example page of yahoo answer page. It has fields of entries index;_ylt=AtKz1xss1AS6RGeAQTFz1kyf5HNG;_ylv=3?qid=20110715030336AAzXnNs

http://answers.yahoo.com/question/index;_ylt=AtKz1xss1AS6RGeAQTFz1kyf5HNG;_ylv=3?qid=20110715030336AAzXnNs

Is it possible for nutch to recognize these fields and crawl it?

Don't think that should be a problem. Though I still feel you would have to
try to actually know, because am not sure if it is going to crawl to an
encrypted url (Experts please help here)

Just make sure the following line is coomented out in crawl-urlfilter.txt:

# skip URLs containing certain characters as probable queries, etc.
#-[?*!@=]

And add the following line:

+^http://answers.yahoo.com/([a-zA-Z0-9-_/]*)

Hopefully it should work. Good luck.

--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-crawl-yahoo-answer-tp3171559p3171764.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi Tamanjit,

Thank you for your help. I tried your suggestion, but it crawl every normal url except url of this type

answers.yahoo.com/question/index;_ylt=AtKz1xss1AS6RGeAQTFz1kyf5HNG;_ylv=3?qid=20110715030336AAzXnNs

I also try this suggestion by

lucene.472066.n3.nabble.com/How-to-make-nutch-crawl-within-a-sub-category-of-an-URL-td619381.html

Use http://answers.yahoo.com/dir/index;_ylt=AqH5s00Y0dXDEjwmdUrxNabpy6IX;_ylv=3?link=list&sid=396545660  as the url to crawl.

Add this in the  crawl-urlfilter.txt

  +^http://answers.yahoo.com/dir/index;_ylt=AqH5s00Y0dXDEjwmdUrxNabpy6IX;_ylv=3?link=list&sid=396545660
   +^http://answers.yahoo.com/question
   -.

but it couldn't crawl anything

Does this mean that nutch can only crawl normal hyperlink?
________________________________
From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Friday, 15 July 2011 8:04 PM
Subject: Re: Is it possible to crawl yahoo answer?

Don't think that should be a problem. Though I still feel you would have to
try to actually know, because am not sure if it is going to crawl to an
encrypted url (Experts please help here)

Just make sure the following line is coomented out in crawl-urlfilter.txt:

# skip URLs containing certain characters as probable queries, etc.
#-[?*!@=]

And add the following line:

+^http://answers.yahoo.com/([a-zA-Z0-9-_/]*)

Hopefully it should work. Good luck.

--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-crawl-yahoo-answer-tp3171559p3171764.html
Sent from the Nutch - User mailing list archive at Nabble.com.

I am not sure if i was able to convey what i meant. But I guess it was a bit
confusing now that I re-read my previous comment.

You are supposed to un-comment the line

-[?*!@=]

This will help nutch crawl through urls with special characters.

Please revert back if you have done this.

--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-crawl-yahoo-answer-tp3171559p3178185.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Hi,
I wonder if you would able to crawl the Yahoo Answer. I am trying to do it
now, I would be appreciate it if you  could give me some advise before hand.

Thanks for your help,

Best,

--
View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-crawl-yahoo-answer-tp3171559p3411176.html
Sent from the Nutch - User mailing list archive at Nabble.com.