Hi Guys,

The NutchHadoopTutorial [0] on  our wiki has finally been updated after
quite some time. It's a rather long beast, but covers (hopefully)
everything you require to get cracking with leveraging the lastest versions
Nutch and Hadoop on a distributed platform and making best use of the great
technologies.

We would really appreciate feedback as there will undoubtedly be some
errors or data missing.

Thanks

Lewis

[0] http://wiki.apache.org/nutch/NutchHadoopTutorial

--
*Lewis*

Thanks for the heads up, dude...you rock, per usual!

Cheers,
Chris

On Mar 19, 2012, at 4:19 PM, Lewis John Mcgibbney wrote:

> Hi Guys,
>
> The NutchHadoopTutorial [0] on  our wiki has finally been updated after
> quite some time. It's a rather long beast, but covers (hopefully)
> everything you require to get cracking with leveraging the lastest versions
> Nutch and Hadoop on a distributed platform and making best use of the great
> technologies.
>
> We would really appreciate feedback as there will undoubtedly be some
> errors or data missing.
>
> Thanks
>
> Lewis
>
> [0] http://wiki.apache.org/nutch/NutchHadoopTutorial
>
> --
> *Lewis*
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

This is great work!! Thanks Lewis!

I must say that when I read the tutorial it stroke me how much of the effort goes into getting Hadoop up and running.

It would be great if we could start with:
"First, make sure you have a healthy Hadoop cluster running, see here for the Hadoop tutorial" ;-)

About the section "Deploy Nutch to Multiple Machines": this is not necessary right? The job jar should be self containing and ship with all the configuration files necessary.
Nutch should be able to run on any vanilla Hadoop cluster.

Anyway, looking at the questions that arrive at nutch-user, this is really really helpful.

Cheers,
Mathijs
On Mar 19, 2012, at 16:19 , Lewis John Mcgibbney wrote:

> Hi Guys,
>
> The NutchHadoopTutorial [0] on  our wiki has finally been updated after
> quite some time. It's a rather long beast, but covers (hopefully)
> everything you require to get cracking with leveraging the lastest versions
> Nutch and Hadoop on a distributed platform and making best use of the great
> technologies.
>
> We would really appreciate feedback as there will undoubtedly be some
> errors or data missing.
>
> Thanks
>
> Lewis
>
> [0] http://wiki.apache.org/nutch/NutchHadoopTutorial
>
> --
> *Lewis*

On Tuesday 20 March 2012 14:23:47 Mathijs Homminga wrote:
> This is great work!! Thanks Lewis!
>
> I must say that when I read the tutorial it stroke me how much of the
> effort goes into getting Hadoop up and running.
>
> It would be great if we could start with:
> "First, make sure you have a healthy Hadoop cluster running, see here for
> the Hadoop tutorial" ;-)
>
> About the section "Deploy Nutch to Multiple Machines": this is not
> necessary right? The job jar should be self containing and ship with all
> the configuration files necessary. Nutch should be able to run on any
> vanilla Hadoop cluster.

It does. All you need is a healthy cluster and a Hadoop environment (cluster
or local) that points to the jobtracker.

>
> Anyway, looking at the questions that arrive at nutch-user, this is really
> really helpful.
>
> Cheers,
> Mathijs
>
> On Mar 19, 2012, at 16:19 , Lewis John Mcgibbney wrote:
> > Hi Guys,
> >
> > The NutchHadoopTutorial [0] on  our wiki has finally been updated after
> > quite some time. It's a rather long beast, but covers (hopefully)
> > everything you require to get cracking with leveraging the lastest
> > versions Nutch and Hadoop on a distributed platform and making best use
> > of the great technologies.
> >
> > We would really appreciate feedback as there will undoubtedly be some
> > errors or data missing.
> >
> > Thanks
> >
> > Lewis
> >
> > [0] http://wiki.apache.org/nutch/NutchHadoopTutorial

--
Markus Jelsma - CTO - Openindex

>> About the section "Deploy Nutch to Multiple Machines": this is not
>> necessary right? The job jar should be self containing and ship with all
>> the configuration files necessary. Nutch should be able to run on any
>> vanilla Hadoop cluster.
>
> It does. All you need is a healthy cluster and a Hadoop environment (cluster
> or local) that points to the jobtracker.

Exactly ;)
Lewis, any reason to keep this section in there?

Mathijs

No I'm taking it out right now. Thanks troops. :)

On Tue, Mar 20, 2012 at 1:38 PM, Mathijs Homminga <
[EMAIL PROTECTED]> wrote:

>
> >> About the section "Deploy Nutch to Multiple Machines": this is not
> >> necessary right? The job jar should be self containing and ship with all
> >> the configuration files necessary. Nutch should be able to run on any
> >> vanilla Hadoop cluster.
> >
> > It does. All you need is a healthy cluster and a Hadoop environment
> (cluster
> > or local) that points to the jobtracker.
>
> Exactly ;)
> Lewis, any reason to keep this section in there?
>
> Mathijs
--
*Lewis*

The section Deploy Nutch to Single Machine is probably based on an old
version of Nutch and quite misleading. Wether you are in fully or pseudo
distributed mode all you need to do is build the job file from the Nutch
root, go to runtime deploy and use the Nutch command from the bin
directory. There aren't any conf files or hadoop executable anymore. If you
need to change something in the conf e.g. url filter files, you need to
rebuild a new job file.

This is definitely a good effort but IMHO most of it is about Hadoop
configuration which is very well explained on the Hadoop pages
http://hadoop.apache.org/common/docs/stable/single_node_setup.html. I think
we should refer to them systematically and focus on the Nutch specific
parts instead.

Julien
On 20 March 2012 13:45, Lewis John Mcgibbney <[EMAIL PROTECTED]>wrote:

> No I'm taking it out right now. Thanks troops. :)
>
> On Tue, Mar 20, 2012 at 1:38 PM, Mathijs Homminga <
> [EMAIL PROTECTED]> wrote:
>
> >
> > >> About the section "Deploy Nutch to Multiple Machines": this is not
> > >> necessary right? The job jar should be self containing and ship with
> all
> > >> the configuration files necessary. Nutch should be able to run on any
> > >> vanilla Hadoop cluster.
> > >
> > > It does. All you need is a healthy cluster and a Hadoop environment
> > (cluster
> > > or local) that points to the jobtracker.
> >
> > Exactly ;)
> > Lewis, any reason to keep this section in there?
> >
> > Mathijs
>
>
>
>
> --
> *Lewis*
>

--
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

Note : the case variation in my previous email is purely accidental. I did
not intend to shout or make the first part more important than the second
:-)

On 20 March 2012 14:19, Julien Nioche <[EMAIL PROTECTED]> wrote:

> The section Deploy Nutch to Single Machine is probably based on an old
> version of Nutch and quite misleading. Wether you are in fully or pseudo
> distributed mode all you need to do is build the job file from the Nutch
> root, go to runtime deploy and use the Nutch command from the bin
> directory. There aren't any conf files or hadoop executable anymore. If you
> need to change something in the conf e.g. url filter files, you need to
> rebuild a new job file.
>
> This is definitely a good effort but IMHO most of it is about Hadoop
> configuration which is very well explained on the Hadoop pages
> http://hadoop.apache.org/common/docs/stable/single_node_setup.html. I
> think we should refer to them systematically and focus on the Nutch
> specific parts instead.
>
> Julien
> On 20 March 2012 13:45, Lewis John Mcgibbney <[EMAIL PROTECTED]>wrote:
>
>> No I'm taking it out right now. Thanks troops. :)
>>
>> On Tue, Mar 20, 2012 at 1:38 PM, Mathijs Homminga <
>> [EMAIL PROTECTED]> wrote:
>>
>> >
>> > >> About the section "Deploy Nutch to Multiple Machines": this is not
>> > >> necessary right? The job jar should be self containing and ship with
>> all
>> > >> the configuration files necessary. Nutch should be able to run on any
>> > >> vanilla Hadoop cluster.
>> > >
>> > > It does. All you need is a healthy cluster and a Hadoop environment
>> > (cluster
>> > > or local) that points to the jobtracker.
>> >
>> > Exactly ;)
>> > Lewis, any reason to keep this section in there?
>> >
>> > Mathijs
>>
>>
>>
>>
>> --
>> *Lewis*
>>
>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
> http://twitter.com/digitalpebble
>
>
--
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

Done... thanks

On Tue, Mar 20, 2012 at 1:45 PM, Lewis John Mcgibbney <
[EMAIL PROTECTED]> wrote:

> No I'm taking it out right now. Thanks troops. :)
>
>
> On Tue, Mar 20, 2012 at 1:38 PM, Mathijs Homminga <
> [EMAIL PROTECTED]> wrote:
>
>>
>> >> About the section "Deploy Nutch to Multiple Machines": this is not
>> >> necessary right? The job jar should be self containing and ship with
>> all
>> >> the configuration files necessary. Nutch should be able to run on any
>> >> vanilla Hadoop cluster.
>> >
>> > It does. All you need is a healthy cluster and a Hadoop environment
>> (cluster
>> > or local) that points to the jobtracker.
>>
>> Exactly ;)
>> Lewis, any reason to keep this section in there?
>>
>> Mathijs
>
>
>
>
> --
> *Lewis*
>
>
--
*Lewis*