Hi,

Somebody (Paul?) mentioned using Droids for doing a 50M page crawl.  Anyone else
using Droids for crawls of that size?

I'm asking because I have a need to do a "semi-vertical" crawl on up to 10K
domains and I'm considering Droids vs. Nutch.  This may translate to several
times that many different servers - say 100K.  And that may translate to a few
100M web pages.  Too big for Droids without having a persistent link queue,
right?

Thanks,
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/

On Fri, 25 Mar 2011 13:08:40 -0700 (PDT), Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> Somebody (Paul?) mentioned using Droids for doing a 50M page crawl.
> Anyone else
> using Droids for crawls of that size?

Yes, but A "little" bit more :) I had a seed of 60M hosts, crawled about 2
billion pages on a 16 node cluster. The crawl took about 3 weeks with an
average bandwidth of 35mbit per node.
 
> I'm asking because I have a need to do a "semi-vertical" crawl on up to
> 10K
> domains and I'm considering Droids vs. Nutch.  This may translate to
> several
> times that many different servers - say 100K.  And that may translate to
a
> few
> 100M web pages.  Too big for Droids without having a persistent link
> queue,
> right?
I had no more than 128 Droid-Threads running per machine with a total
in-memory-queue limit (for all threads per node) of ~ 500.000 pages. Had a
couple of tweaks here and there plus an efficient tree structure for
storing visited urls. With 10gb you could easily go up to 2 million entries
when running only one droid thread per VM. Perhaps even more. Here is
something pretty to look at: http://twitpic.com/4d87i7 ;) My crawler design
was rather simple: ((one host - one droid, stay on the host) * 128 threads
) * 16 nodes and one master node for the global seed queue. If your crawl
is to behave more organic and may be executed in a less controlled
environment: take a look at 80Legs.  

If you choose to go with droids you will have to spend some time on the
http client settings and implement some workarounds in droids / http client
to prevent stale/stuck sockets, handle chunked stream aborts correctly
(crawler traps serving you an endless stream of links for instance), take
care of rather critical robots issue. The neko html parser is also full of
surprises, be careful with that. At least when you are lazy like me and use
the DOM model instead of SAX to preparse some meta-tags and do content
processing.

I don't see how my spare time would allow to clean up my battlefield and
release all of it back to the community any time soon, but I am not sitting
on that code either. Talk to me if you want to port aspects of that back to
droids. I would be more than happy to rip out chunks of that code and pass
it along for proper integration into the main branch.

Regards,
Paul.

On Fri, 2011-03-25 at 22:09 +0100, paul.vc wrote:
...
> I don't see how my spare time would allow to clean up my battlefield and
> release all of it back to the community any time soon, but I am not sitting
> on that code either. Talk to me if you want to port aspects of that back to
> droids. I would be more than happy to rip out chunks of that code and pass
> it along for proper integration into the main branch.
>
> Regards,
> Paul.

Paul I always read your emails with a lot of interest. It seems you
brought Droids to a pretty industrial use case. What a pity that we can
not merge the whole back.

I wonder if we could commit your branch as is into our rep and then
merge the best approaches and patches back into the trunk. Would that be
possible?

salu2
--
Thorsten Scherler <thorsten.at.apache.org>
codeBusters S.L. - web based systems
<consulting, training and solutions>
http://www.codebusters.es/

I like Thorsten's idea.  That way Droids developers and contributors could take
piece by piece from that branch and get it in Droids core/trunk over time.  Is
that doable, Paul?

Thanks,
Otis
----- Original Message ----
> From: Thorsten Scherler <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Sent: Mon, March 28, 2011 5:06:23 AM
> Subject: Re: Droids suitability for a 100M+ page crawl
>
> On Fri, 2011-03-25 at 22:09 +0100, paul.vc wrote:
> ...
> > I don't see how  my spare time would allow to clean up my battlefield and
> > release all of  it back to the community any time soon, but I am not sitting
> > on that  code either. Talk to me if you want to port aspects of that back to
> >  droids. I would be more than happy to rip out chunks of that code and  pass
> > it along for proper integration into the main branch.
> >
> > Regards,
> > Paul.
>
> Paul I always read your emails with a lot  of interest. It seems you
> brought Droids to a pretty industrial use case.  What a pity that we can
> not merge the whole back.
>
> I wonder if we  could commit your branch as is into our rep and then
> merge the best  approaches and patches back into the trunk. Would that  be
> possible?
>
> salu2
> --
> Thorsten Scherler  <thorsten.at.apache.org>
> codeBusters S.L. - web based  systems
> <consulting, training and solutions>
> http://www.codebusters.es/
>
>

Hi Otis,

if you have to handle a very large set of tasks you may be interested
in having a look at the hazelcast distribuded queue. You should be
able to use it with the current version of the trunk since droids-56
has been applied. In such cluster no task will be lost in case of
failure.

Best regards,

Bertil

Sent from my iPad

On Mar 25, 2011, at 9:09 PM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:

> Hi,
>
> Somebody (Paul?) mentioned using Droids for doing a 50M page crawl.  Anyone else
> using Droids for crawls of that size?
>
> I'm asking because I have a need to do a "semi-vertical" crawl on up to 10K
> domains and I'm considering Droids vs. Nutch.  This may translate to several
> times that many different servers - say 100K.  And that may translate to a few
> 100M web pages.  Too big for Droids without having a persistent link queue,
> right?
>
> Thanks,
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>