I have been working on an off for about a year now on developing a prototype
for Enterprise Search using Nutch and Solr.  I have also incorporated a
plugin using the hive-mrc google code for automatic tagging based on a
custom taxonomy that my customer uses.  I have been slowly migrating up the
chain of machines available and I have been given one machine for my
"prototype" that is fairly powerful.  

Some problems still remain that I beleive can be fixed and others make me
question my decision to use Nutch.  

One problem has to do with the fact that I am doing vertical searching.  The
side effect of this is that the crawl process is SO slow.  It took about 48
hours to crawl about 350,000 urls all from the same website. I am am
crawling a shared file system and I am sure that constitutes vertical
crawling.  The other web crawling I am doing also only comes from a handful
of urls.  Maybe nutch is not the solution to use based on this?

The other problem is the fact that I would like to use the
AdaptiveFetchSchedule and the developers I work with refuse to use caching
and Last Modified time for our PHP pages.  This should be a nightmare :-(  

I love the solr aspect of our prototype.  It is very fast and reliable and I
have not had lots of issues.

In the real world, how to production environments use Nutch?  Do they have a
separate custom script that runs each of the crawl commands separately?  Do
they run this script once a day?  What about vertical crawling, are there
any special setting that could help Nutch run faster?
--
View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2923289.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Nutch was never meant for vertical or enterprise search. Solr, is a
great engine but obviously you need to get to the documents first. In
order for me to state any further opinion I should ask the following:

1) What kind of documents/repositories are you trying to provide search for?
2) Are security and user access/permissions important for you?
3) What is the typical size of the document universe you which your
software to handle (in number of documents + avg size and/or total
GB)?

-- J

On Tue, May 10, 2011 at 7:37 AM, webdev1977 <[EMAIL PROTECTED]> wrote:
> I have been working on an off for about a year now on developing a prototype
> for Enterprise Search using Nutch and Solr.  I have also incorporated a
> plugin using the hive-mrc google code for automatic tagging based on a
> custom taxonomy that my customer uses.  I have been slowly migrating up the
> chain of machines available and I have been given one machine for my
> "prototype" that is fairly powerful.
>
> Some problems still remain that I beleive can be fixed and others make me
> question my decision to use Nutch.
>
> One problem has to do with the fact that I am doing vertical searching.  The
> side effect of this is that the crawl process is SO slow.  It took about 48
> hours to crawl about 350,000 urls all from the same website. I am am
> crawling a shared file system and I am sure that constitutes vertical
> crawling.  The other web crawling I am doing also only comes from a handful
> of urls.  Maybe nutch is not the solution to use based on this?
>
> The other problem is the fact that I would like to use the
> AdaptiveFetchSchedule and the developers I work with refuse to use caching
> and Last Modified time for our PHP pages.  This should be a nightmare :-(
>
> I love the solr aspect of our prototype.  It is very fast and reliable and I
> have not had lots of issues.
>
> In the real world, how to production environments use Nutch?  Do they have a
> separate custom script that runs each of the crawl commands separately?  Do
> they run this script once a day?  What about vertical crawling, are there
> any special setting that could help Nutch run faster?
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2923289.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

Thanks for your reply.. I was kind of afraid someone was going to say that
:-( I have invested so much time into developing plugins for Nutch that I am
deathly afraid of moving on to something else.

To answer your questions:
1) What kind of documents/repositories are you trying to provide search for?

I have several internal websites I am crawling (most of which are web front
for database info), I am also crawling a local shared file system.  The
document types run the gamut.. html, pdf, word, excel, powerpoint, txt,
images, etc, etc.  (and any other crap the users throw on the file system)
  
2) Are security and user access/permissions important for you?

Somewhat.. but not as much as you would think.  I actually have/had more
problems with accessing sites that required SSL certificates.  But, I fixed
that by modifying the protocol-httpclient to use a java keystore and a
client cert to pass while fetching the page.

3) What is the typical size of the document universe you which your
software to handle (in number of documents + avg size and/or total
GB)?

The documents are all under 200mb or so.  Most of them are html or pdf files
that are of a normal size.  The total size of the documents to be crawled is
fairly large about 500gb.  The other stuff, maybe about 100gb total.  

--
View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2923807.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Dare I ask... What sorts of crawlers are meant for use with vertical search
systems?

--
View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2932884.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Why do you think Nutch is not suited for vertical search? I am in the
process of building just that, and am planning to use a Hadoop cluster
(most likely on AWS) for crawling.

On Tue, May 10, 2011 at 12:05 PM, J. Delgado <[EMAIL PROTECTED]> wrote:
> Nutch was never meant for vertical or enterprise search. Solr, is a
> great engine but obviously you need to get to the documents first. In
> order for me to state any further opinion I should ask the following:
>
> 1) What kind of documents/repositories are you trying to provide search for?
> 2) Are security and user access/permissions important for you?
> 3) What is the typical size of the document universe you which your
> software to handle (in number of documents + avg size and/or total
> GB)?
>
> -- J
>
> On Tue, May 10, 2011 at 7:37 AM, webdev1977 <[EMAIL PROTECTED]> wrote:
>> I have been working on an off for about a year now on developing a prototype
>> for Enterprise Search using Nutch and Solr.  I have also incorporated a
>> plugin using the hive-mrc google code for automatic tagging based on a
>> custom taxonomy that my customer uses.  I have been slowly migrating up the
>> chain of machines available and I have been given one machine for my
>> "prototype" that is fairly powerful.
>>
>> Some problems still remain that I beleive can be fixed and others make me
>> question my decision to use Nutch.
>>
>> One problem has to do with the fact that I am doing vertical searching.  The
>> side effect of this is that the crawl process is SO slow.  It took about 48
>> hours to crawl about 350,000 urls all from the same website. I am am
>> crawling a shared file system and I am sure that constitutes vertical
>> crawling.  The other web crawling I am doing also only comes from a handful
>> of urls.  Maybe nutch is not the solution to use based on this?
>>
>> The other problem is the fact that I would like to use the
>> AdaptiveFetchSchedule and the developers I work with refuse to use caching
>> and Last Modified time for our PHP pages.  This should be a nightmare :-(
>>
>> I love the solr aspect of our prototype.  It is very fast and reliable and I
>> have not had lots of issues.
>>
>> In the real world, how to production environments use Nutch?  Do they have a
>> separate custom script that runs each of the crawl commands separately?  Do
>> they run this script once a day?  What about vertical crawling, are there
>> any special setting that could help Nutch run faster?
>>
>>
>>
>>
>> --
>> View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2923289.html
>> Sent from the Nutch - User mailing list archive at Nabble.com.
>>
>

I was saying that based on what the previous poster stated.  Also the fact
that I have read through quite a bit of posts stating that the problem with
crawling in a vertical environment has to do with the way fetcher2 was
built.  The fetches are grouped by domain name and if you have a lot of urls
from the same domain then you are not able to do quick mapreduce jobs.  

I hope this is wrong though ;-)

--
View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2932969.html
Sent from the Nutch - User mailing list archive at Nabble.com.

On Thu, May 12, 2011 at 2:24 PM, webdev1977 <[EMAIL PROTECTED]> wrote:
> I was saying that based on what the previous poster stated.  Also the fact
> that I have read through quite a bit of posts stating that the problem with
> crawling in a vertical environment has to do with the way fetcher2 was
> built.  The fetches are grouped by domain name and if you have a lot of urls
> from the same domain then you are not able to do quick mapreduce jobs.
>
It is true that the more domains you have, the faster you will be able
to crawl. But I don't agree that this has anything to do with the way
the fetching is done. It's true for any fetcher:
If you need to crawl ~1000 domains or so (fairly typical for a
vertical search), you should be able to run quite a few threads
partitioned across all of them without having to allow more than five
concurrent threads per host.

> I was saying that based on what the previous poster stated.  Also the fact
> that I have read through quite a bit of posts stating that the problem with
> crawling in a vertical environment has to do with the way fetcher2 was
> built.  The fetches are grouped by domain name and if you have a lot of
> urls
> from the same domain then you are not able to do quick mapreduce jobs.
>

Nutch's default behaviour is to be polite to the hosts it visits. If you own
the hosts (or have an agreement with the owner) you can of course hit them
as hard as you want and set a higher number of threads per host or time
between hits. If you don't own the hosts then you simply should not do that
and use the defaults used in Nutch as a matter of courtesy. (moreover if you
are too aggressive in your choice of parameters then you'll probably be
blacklisted by the target servers and won't be allowed to fetch any content)

Let's be completely clear once and for all : there is no particular issue
with using Nutch for vertical crawls - loads of people have done and still
do that.

Julien

--
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com

Julien Nioche-4 wrote:
>
>> I was saying that based on what the previous poster stated.  Also the
>> fact
>> that I have read through quite a bit of posts stating that the problem
>> with
>> crawling in a vertical environment has to do with the way fetcher2 was
>> built.  The fetches are grouped by domain name and if you have a lot of
>> urls
>> from the same domain then you are not able to do quick mapreduce jobs.
>>
>
> Nutch's default behaviour is to be polite to the hosts it visits. If you
> own
> the hosts (or have an agreement with the owner) you can of course hit them
> as hard as you want and set a higher number of threads per host or time
> between hits. If you don't own the hosts then you simply should not do
> that
> and use the defaults used in Nutch as a matter of courtesy. (moreover if
> you
> are too aggressive in your choice of parameters then you'll probably be
> blacklisted by the target servers and won't be allowed to fetch any
> content)
>
> Let's be completely clear once and for all : there is no particular issue
> with using Nutch for vertical crawls - loads of people have done and still
> do that.
>
> Julien
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
>

Julien,

We indeed own the hosts, and I have been experimenting with the number of
threads I am able to use without crashing our web server/ database.  This
has led me to the refactoring of some of our code to improve connection
pooling and resource allocation.  

What I don't know how to speed up is the mapreduce jobs..  It takes
approximately 12 hours JUST do to the fetching of 250,000 or so urls.  The
map reduce part takes about 36 hours.

Is this normal?  Is there anyway to speed this up?  

I have seen talk of setting generate.max.per.host setting, but I don't want
to limit the number of urls I fetch.  And to me, this is what this setting
would accomplish.
--
View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2947297.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Any ideas on how (even if it requires code changes) to speed up the mapreduce
portion for a vertical crawl with a very (three right now) small number of
sites?

--
View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2984011.html
Sent from the Nutch - User mailing list archive at Nabble.com.

I assume you've already read
http://wiki.apache.org/nutch/OptimizingCrawls<%20http://wiki.apache.org/nutch/OptimizingCrawls>
and tried different values for :

<property>
  <name>fetcher.server.delay</name>
  <value>5.0</value>
  <description>The number of seconds the fetcher will delay between
   successive requests to the same server.</description>
</property>

<property>
  <name>fetcher.server.min.delay</name>
  <value>0.0</value>
  <description>The minimum number of seconds the fetcher will delay between
  successive requests to the same server. This value is applicable ONLY
  if fetcher.threads.per.host is greater than 1 (i.e. the host blocking
  is turned off).</description>
</property>
On 25 May 2011 12:52, webdev1977 <[EMAIL PROTECTED]> wrote:

> Any ideas on how (even if it requires code changes) to speed up the
> mapreduce
> portion for a vertical crawl with a very (three right now) small number of
> sites?
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2984011.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

--
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com

I sure have! And I cranked it up at one point sooo much that I crashed the
TNS listener on our DB!!

The fetching part didn't take the longest, it was the mapreduce that took
forever..

--
View this message in context: http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2984815.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Pretty much everything in Nutch is 'a mapreduce' so it is not clear what you
meant. Which step took most of the time, the udpate? BTW do you parse at the
same time as you fetch?

On 25 May 2011 16:55, webdev1977 <[EMAIL PROTECTED]> wrote:

> I sure have! And I cranked it up at one point sooo much that I crashed the
> TNS listener on our DB!!
>
> The fetching part didn't take the longest, it was the mapreduce that took
> forever..
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Going-Beyond-the-Prototype-tp2923289p2984815.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>

--
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com

On Tue, May 10, 2011 at 10:37 AM, webdev1977 <[EMAIL PROTECTED]> wrote:
> One problem has to do with the fact that I am doing vertical searching.  The
> side effect of this is that the crawl process is SO slow.  It took about 48
> hours to crawl about 350,000 urls all from the same website. I am am
> crawling a shared file system and I am sure that constitutes vertical
> crawling.  The other web crawling I am doing also only comes from a handful
> of urls.  Maybe nutch is not the solution to use based on this?
>
There are two options, number one is mandatory:
- don't crawl one web site at a time. Crawl all of your sites at once
so the threads can be partitioned across all sites. You will be able
to crawl much faster without overloading any of the sites. This one is
a must
- use a Hadoop cluster to crawl, not a single machine

Dietrich