Hi all,

I am trying to run a RecommenderJob from a Java program. I have added the
files users.txt and input.txt to a Hadoop VM and use the run-method of
RecommenderJob to start the calculation. But the the following error
message occurs while running the MapReducer:

Error: java.lang.ClassNotFoundException: com.google.common.primitives.Longs
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
 at
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
at
org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)

I have added the required guava-r09.jar explicitly to my jar which also
lays on the hadoop cluster.
This may be a stupid question, but does anyone know where this error comes
from? This would help me a lot.

Thanks and greetings,
Janina

You shouldn't have to add anything to your jar, if you use the
supplied 'job' file which contains all transitive dependencies.
If you do add your own jars, I think you need to unpack and repack
them, not put them into the overall jar as a jar file, even with a
MANIFEST.MF entry. I am not sure that works on Hadoop.

On Thu, Mar 15, 2012 at 9:42 AM, Janina <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I am trying to run a RecommenderJob from a Java program. I have added the
> files users.txt and input.txt to a Hadoop VM and use the run-method of
> RecommenderJob to start the calculation. But the the following error
> message occurs while running the MapReducer:
>
> Error: java.lang.ClassNotFoundException: com.google.common.primitives.Longs
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>  at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> at
> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
>  at
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
> at
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> I have added the required guava-r09.jar explicitly to my jar which also
> lays on the hadoop cluster.
> This may be a stupid question, but does anyone know where this error comes
> from? This would help me a lot.
>
> Thanks and greetings,
> Janina

Thanks for your fast answer.

I haven't added the jar manually, but by adding the dependency to the
pom.xml. I tried it with and without the dependendy and with different
versions of the dependency, but it remained the same error message.

But the RecommenderJob is meant to be to run a pseudo distributed
recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
have another possibility to run recommendations on a Hadoop Cluster? I have
read that only the clustering and classification parts of mahout are really
able to be distributed on a hadoop cluster.

2012/3/15 Sean Owen <[EMAIL PROTECTED]>

> You shouldn't have to add anything to your jar, if you use the
> supplied 'job' file which contains all transitive dependencies.
> If you do add your own jars, I think you need to unpack and repack
> them, not put them into the overall jar as a jar file, even with a
> MANIFEST.MF entry. I am not sure that works on Hadoop.
>
> On Thu, Mar 15, 2012 at 9:42 AM, Janina <[EMAIL PROTECTED]>
> wrote:
> > Hi all,
> >
> > I am trying to run a RecommenderJob from a Java program. I have added the
> > files users.txt and input.txt to a Hadoop VM and use the run-method of
> > RecommenderJob to start the calculation. But the the following error
> > message occurs while running the MapReducer:
> >
> > Error: java.lang.ClassNotFoundException:
> com.google.common.primitives.Longs
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >  at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> > at
> >
> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
> >  at
> >
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
> > at
> >
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
> >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
> > I have added the required guava-r09.jar explicitly to my jar which also
> > lays on the hadoop cluster.
> > This may be a stupid question, but does anyone know where this error
> comes
> > from? This would help me a lot.
> >
> > Thanks and greetings,
> > Janina
>

You would still need to use the 'job' file generated by the build to
get an artifact with all the dependencies.
You don't need to add Guava as a dependency; it already is one. It's
the job file that you're missing.

There are two RecommenderJobs. One is what I call pseudo-distributed,
yes. The other is a fully distributed item-based recommender.

Sean

On Thu, Mar 15, 2012 at 10:01 AM, Janina <[EMAIL PROTECTED]> wrote:
> Thanks for your fast answer.
>
> I haven't added the jar manually, but by adding the dependency to the
> pom.xml. I tried it with and without the dependendy and with different
> versions of the dependency, but it remained the same error message.
>
> But the RecommenderJob is meant to be to run a pseudo distributed
> recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
> have another possibility to run recommendations on a Hadoop Cluster? I have
> read that only the clustering and classification parts of mahout are really
> able to be distributed on a hadoop cluster.
>
> 2012/3/15 Sean Owen <[EMAIL PROTECTED]>
>
>> You shouldn't have to add anything to your jar, if you use the
>> supplied 'job' file which contains all transitive dependencies.
>> If you do add your own jars, I think you need to unpack and repack
>> them, not put them into the overall jar as a jar file, even with a
>> MANIFEST.MF entry. I am not sure that works on Hadoop.
>>
>> On Thu, Mar 15, 2012 at 9:42 AM, Janina <[EMAIL PROTECTED]>
>> wrote:
>> > Hi all,
>> >
>> > I am trying to run a RecommenderJob from a Java program. I have added the
>> > files users.txt and input.txt to a Hadoop VM and use the run-method of
>> > RecommenderJob to start the calculation. But the the following error
>> > message occurs while running the MapReducer:
>> >
>> > Error: java.lang.ClassNotFoundException:
>> com.google.common.primitives.Longs
>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >  at java.security.AccessController.doPrivileged(Native Method)
>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> > at
>> >
>> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
>> >  at
>> >
>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
>> > at
>> >
>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
>> >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >
>> > I have added the required guava-r09.jar explicitly to my jar which also
>> > lays on the hadoop cluster.
>> > This may be a stupid question, but does anyone know where this error
>> comes
>> > from? This would help me a lot.
>> >
>> > Thanks and greetings,
>> > Janina
>>

These are great news! I was not quite sure if the item based recommender is
fully distributes, but this helps! Thanks!

I have removed the dependency now. But although it may again be a stupid
question: what do you exactly mean with "I am missing the job file"? Sorry,
but I am completely new to Mahout and Hadoop and I am very thankful for
your help :)

2012/3/15 Sean Owen <[EMAIL PROTECTED]>

> You would still need to use the 'job' file generated by the build to
> get an artifact with all the dependencies.
> You don't need to add Guava as a dependency; it already is one. It's
> the job file that you're missing.
>
> There are two RecommenderJobs. One is what I call pseudo-distributed,
> yes. The other is a fully distributed item-based recommender.
>
> Sean
>
> On Thu, Mar 15, 2012 at 10:01 AM, Janina <[EMAIL PROTECTED]>
> wrote:
> > Thanks for your fast answer.
> >
> > I haven't added the jar manually, but by adding the dependency to the
> > pom.xml. I tried it with and without the dependendy and with different
> > versions of the dependency, but it remained the same error message.
> >
> > But the RecommenderJob is meant to be to run a pseudo distributed
> > recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
> > have another possibility to run recommendations on a Hadoop Cluster? I
> have
> > read that only the clustering and classification parts of mahout are
> really
> > able to be distributed on a hadoop cluster.
> >
> > 2012/3/15 Sean Owen <[EMAIL PROTECTED]>
> >
> >> You shouldn't have to add anything to your jar, if you use the
> >> supplied 'job' file which contains all transitive dependencies.
> >> If you do add your own jars, I think you need to unpack and repack
> >> them, not put them into the overall jar as a jar file, even with a
> >> MANIFEST.MF entry. I am not sure that works on Hadoop.
> >>
> >> On Thu, Mar 15, 2012 at 9:42 AM, Janina <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi all,
> >> >
> >> > I am trying to run a RecommenderJob from a Java program. I have added
> the
> >> > files users.txt and input.txt to a Hadoop VM and use the run-method of
> >> > RecommenderJob to start the calculation. But the the following error
> >> > message occurs while running the MapReducer:
> >> >
> >> > Error: java.lang.ClassNotFoundException:
> >> com.google.common.primitives.Longs
> >> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >> >  at java.security.AccessController.doPrivileged(Native Method)
> >> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> > at
> >> >
> >>
> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
> >> >  at
> >> >
> >>
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
> >> > at
> >> >
> >>
> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
> >> >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
> >> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >> >
> >> > I have added the required guava-r09.jar explicitly to my jar which
> also
> >> > lays on the hadoop cluster.
> >> > This may be a stupid question, but does anyone know where this error
> >> comes
> >> > from? This would help me a lot.
> >> >
> >> > Thanks and greetings,
> >> > Janina
> >>
>

After 'mvn package' you should see a file ending in 'job.jar' under
target/ This is the jar file to use with Hadoop.

On Thu, Mar 15, 2012 at 10:56 AM, Janina <[EMAIL PROTECTED]> wrote:
> These are great news! I was not quite sure if the item based recommender is
> fully distributes, but this helps! Thanks!
>
> I have removed the dependency now. But although it may again be a stupid
> question: what do you exactly mean with "I am missing the job file"? Sorry,
> but I am completely new to Mahout and Hadoop and I am very thankful for
> your help :)

Ahhh, ok...yes, I am using this generated jar file with Hadoop, but this
does not change the error message about com.google.common.primitives.Longs
.....

2012/3/15 Sean Owen <[EMAIL PROTECTED]>

> After 'mvn package' you should see a file ending in 'job.jar' under
> target/ This is the jar file to use with Hadoop.
>
> On Thu, Mar 15, 2012 at 10:56 AM, Janina <[EMAIL PROTECTED]>
> wrote:
> > These are great news! I was not quite sure if the item based recommender
> is
> > fully distributes, but this helps! Thanks!
> >
> > I have removed the dependency now. But although it may again be a stupid
> > question: what do you exactly mean with "I am missing the job file"?
> Sorry,
> > but I am completely new to Mahout and Hadoop and I am very thankful for
> > your help :)
>

Pe 15.03.2012 13:14, Janina a scris:
> Ahhh, ok...yes, I am using this generated jar file with Hadoop, but this
> does not change the error message about com.google.common.primitives.Longs
> .....
Hello Janina,

Could you please enable bash debugging [1] and show us the command line
that submits your job?

Also, if you submit a job there are two class paths that you must be
aware of:
- the one that starts the job submitting app
- the one that will be available to the job on each node
Another thing, I see you are using both org.apache.hadoop.mapreduce and
org.apache.hadoop.mapred packages there. From what I know, mapred is
deprecated in favor of the new mapreduce so try to use the new api in
your code.
[1] http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html

Cheers,

--
Ioan Eugen Stan
http://ieugen.blogspot.com

Can you elaborate on how to set these classpaths?

The job takes a classpath param, (from mahout  --help)

      -libjars <paths>               comma separated jar files to include in
                                                 the classpath.

is the needed code bundled with the job or does the path need to be
available to hadoop/mahout on all nodes? Also is this the only thing
that affects the classpath to any code outside the jar file?

On 3/15/12 5:33 AM, Ioan Eugen Stan wrote:
> Also, if you submit a job there are two class paths that you must be
> aware of:
> - the one that starts the job submitting app
> - the one that will be available to the job on each node

Wait! I thought there was only one RecommenderJob?

On Thu, Mar 15, 2012 at 3:44 AM, Sean Owen <[EMAIL PROTECTED]> wrote:
> You would still need to use the 'job' file generated by the build to
> get an artifact with all the dependencies.
> You don't need to add Guava as a dependency; it already is one. It's
> the job file that you're missing.
>
> There are two RecommenderJobs. One is what I call pseudo-distributed,
> yes. The other is a fully distributed item-based recommender.
>
> Sean
>
> On Thu, Mar 15, 2012 at 10:01 AM, Janina <[EMAIL PROTECTED]> wrote:
>> Thanks for your fast answer.
>>
>> I haven't added the jar manually, but by adding the dependency to the
>> pom.xml. I tried it with and without the dependendy and with different
>> versions of the dependency, but it remained the same error message.
>>
>> But the RecommenderJob is meant to be to run a pseudo distributed
>> recommender on a Hadoop cluster? Am I guessing something wrong? Or do I
>> have another possibility to run recommendations on a Hadoop Cluster? I have
>> read that only the clustering and classification parts of mahout are really
>> able to be distributed on a hadoop cluster.
>>
>> 2012/3/15 Sean Owen <[EMAIL PROTECTED]>
>>
>>> You shouldn't have to add anything to your jar, if you use the
>>> supplied 'job' file which contains all transitive dependencies.
>>> If you do add your own jars, I think you need to unpack and repack
>>> them, not put them into the overall jar as a jar file, even with a
>>> MANIFEST.MF entry. I am not sure that works on Hadoop.
>>>
>>> On Thu, Mar 15, 2012 at 9:42 AM, Janina <[EMAIL PROTECTED]>
>>> wrote:
>>> > Hi all,
>>> >
>>> > I am trying to run a RecommenderJob from a Java program. I have added the
>>> > files users.txt and input.txt to a Hadoop VM and use the run-method of
>>> > RecommenderJob to start the calculation. But the the following error
>>> > message occurs while running the MapReducer:
>>> >
>>> > Error: java.lang.ClassNotFoundException:
>>> com.google.common.primitives.Longs
>>> > at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>> >  at java.security.AccessController.doPrivileged(Native Method)
>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>> > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>> >  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>>> > at
>>> >
>>> org.apache.mahout.cf.taste.hadoop.TasteHadoopUtils.idToIndex(TasteHadoopUtils.java:61)
>>> >  at
>>> >
>>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:48)
>>> > at
>>> >
>>> org.apache.mahout.cf.taste.hadoop.item.ItemIDIndexMapper.map(ItemIDIndexMapper.java:31)
>>> >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>>> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>> >
>>> > I have added the required guava-r09.jar explicitly to my jar which also
>>> > lays on the hadoop cluster.
>>> > This may be a stupid question, but does anyone know where this error
>>> comes
>>> > from? This would help me a lot.
>>> >
>>> > Thanks and greetings,
>>> > Janina
>>>

--
Lance Norskog
[EMAIL PROTECTED]

Pe 15.03.2012 15:51, Pat Ferrel a scris:
> Can you elaborate on how to set these classpaths?
>
> The job takes a classpath param, (from mahout --help)
>
> -libjars <paths> comma separated jar files to include in
> the classpath.
>
> is the needed code bundled with the job or does the path need to be
> available to hadoop/mahout on all nodes? Also is this the only thing
> that affects the classpath to any code outside the jar file?
>
> On 3/15/12 5:33 AM, Ioan Eugen Stan wrote:
>> Also, if you submit a job there are two class paths that you must be
>> aware of:
>> - the one that starts the job submitting app
>> - the one that will be available to the job on each node
>

Hi Pat,

To set up your main app class-path use HADOOP_CLASS_PATH env var. For
the Job class-path it's best explained here [1], although, from my
experience -libjars did not work as advertised. It's hadoop version
dependant probably but I may be wrong. Test and see for yourself.

Also see my post on common-user about this issue [2]. Search for
ClassNotFoundException: libjars .
[1]
http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
[2] http://hadoop.markmail.org/thread/st3cy7alcodlztng

Hope this helps,

--
Ioan Eugen Stan
http://ieugen.blogspot.com

Thanks, [1] is a very good reference for ClassNotFoundException.

As you say in [2] the mahout script does not accept hadoop job
parameters in all cases although I hope it does in the future,
especially where a parameter to the job is a classname (seq2sparse for
instance).

What I had to do was copy my custom analyzer jar into $HADOOP_HOME/lib
on the master node. Evidently a symlink does not work, it appears you
have to copy each jar you want to the directory.

Then don't forget to stop and start hadoop because as [1] says it
packages the libs at startup.

On 3/16/12 7:12 AM, Ioan Eugen Stan wrote:
> Pe 15.03.2012 15:51, Pat Ferrel a scris:
>> Can you elaborate on how to set these classpaths?
>>
>> The job takes a classpath param, (from mahout --help)
>>
>> -libjars <paths> comma separated jar files to include in
>> the classpath.
>>
>> is the needed code bundled with the job or does the path need to be
>> available to hadoop/mahout on all nodes? Also is this the only thing
>> that affects the classpath to any code outside the jar file?
>>
>> On 3/15/12 5:33 AM, Ioan Eugen Stan wrote:
>>> Also, if you submit a job there are two class paths that you must be
>>> aware of:
>>> - the one that starts the job submitting app
>>> - the one that will be available to the job on each node
>>
>
> Hi Pat,
>
> To set up your main app class-path use HADOOP_CLASS_PATH env var. For
> the Job class-path it's best explained here [1], although, from my
> experience -libjars did not work as advertised. It's hadoop version
> dependant probably but I may be wrong. Test and see for yourself.
>
> Also see my post on common-user about this issue [2]. Search for
> ClassNotFoundException: libjars .
>
>
> [1]
> http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
> [2] http://hadoop.markmail.org/thread/st3cy7alcodlztng
>
> Hope this helps,
>