Hello all,

I have started to play around with Mahout today.  During installation,
the unit test testStartParallelFPGrowth is failing. (I am not sure whether
to send this post
here or to the developer list).

Adding the -e option to maven, we get the culprit

Failed tests:
  testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0, E]=1}>

Unfortunately, this is an algorithmic area in which I have no background (I
work in optimization/linear algebra) so I don't feel comfortable digging
into this myself.

I downloaded the Mahout src, version .5 this afternoon.  The only small
hitch in my setup
is that I am running this under the IBM Java JDK:
$java -version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pxa6460sr7-20091215_02(SR7))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64
jvmxa6460sr7-20091214_49398 (JIT enabled, AOT enabled)
J9VM - 20091214_049398
JIT  - r9_20091123_13891
GC   - 20091111_AA)
JCL  - 20091202_01

With Maven version 3.03

$mvn -version
mvn -version
Apache Maven 3.0.3 (r1075438; 2011-02-28 12:31:09-0500)
Maven home: /home/mszetom/apache-maven-3.0.3
Java version: 1.6.0, vendor: IBM Corporation
Java home: /opt/ibm/ibm-java-x86_64-60/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.31-22-server", arch: "amd64", family: "unix"

I should also add that I am not currently using Hadoop beneath Mahout yet, I
have not had
the time to get it running on our cluster.
I do not plan on using this functionality; however, I am more than happy to
help drill into this to
find the issue.
Thanks in advance,
marc

Did you check out the Mahout trunk? That test might be broken now; it
should have worked in the 0.5 source release (checkpoint, really).

On Fri, Jul 8, 2011 at 5:41 PM, Marc Millstone <[EMAIL PROTECTED]> wrote:
> Hello all,
>
> I have started to play around with Mahout today.  During installation,
> the unit test testStartParallelFPGrowth is failing. (I am not sure whether
> to send this post
> here or to the developer list).
>
> Adding the -e option to maven, we get the culprit
>
> Failed tests:
>  testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0, E]=1}>
>
> Unfortunately, this is an algorithmic area in which I have no background (I
> work in optimization/linear algebra) so I don't feel comfortable digging
> into this myself.
>
> I downloaded the Mahout src, version .5 this afternoon.  The only small
> hitch in my setup
> is that I am running this under the IBM Java JDK:
>
>
> $java -version
> java version "1.6.0"
> Java(TM) SE Runtime Environment (build pxa6460sr7-20091215_02(SR7))
> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64
> jvmxa6460sr7-20091214_49398 (JIT enabled, AOT enabled)
> J9VM - 20091214_049398
> JIT  - r9_20091123_13891
> GC   - 20091111_AA)
> JCL  - 20091202_01
>
> With Maven version 3.03
>
> $mvn -version
> mvn -version
> Apache Maven 3.0.3 (r1075438; 2011-02-28 12:31:09-0500)
> Maven home: /home/mszetom/apache-maven-3.0.3
> Java version: 1.6.0, vendor: IBM Corporation
> Java home: /opt/ibm/ibm-java-x86_64-60/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "2.6.31-22-server", arch: "amd64", family: "unix"
>
> I should also add that I am not currently using Hadoop beneath Mahout yet, I
> have not had
> the time to get it running on our cluster.
>
>
> I do not plan on using this functionality; however, I am more than happy to
> help drill into this to
> find the issue.
>
>
> Thanks in advance,
> marc
>

--
Lance Norskog
[EMAIL PROTECTED]

No, I downloaded the src from the Mahout website with the .5 checkpoint.
(file mahout-distribution-0.5-src.tar.gz)
I will try to throw a Sun JVM into my home directory on the cluster  and try
to isolate the failed unit test from the JVM choice.
As you can see from the failed test, the answer is "almost" right.
On Fri, Jul 8, 2011 at 7:42 PM, Lance Norskog <[EMAIL PROTECTED]> wrote:

> Did you check out the Mahout trunk? That test might be broken now; it
> should have worked in the 0.5 source release (checkpoint, really).
>
> On Fri, Jul 8, 2011 at 5:41 PM, Marc Millstone <[EMAIL PROTECTED]>
> wrote:
> > Hello all,
> >
> > I have started to play around with Mahout today.  During installation,
> > the unit test testStartParallelFPGrowth is failing. (I am not sure
> whether
> > to send this post
> > here or to the developer list).
> >
> > Adding the -e option to maven, we get the culprit
> >
> > Failed tests:
> >
>  testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> > expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
> E]=1}>
> >
> > Unfortunately, this is an algorithmic area in which I have no background
> (I
> > work in optimization/linear algebra) so I don't feel comfortable digging
> > into this myself.
> >
> > I downloaded the Mahout src, version .5 this afternoon.  The only small
> > hitch in my setup
> > is that I am running this under the IBM Java JDK:
> >
> >
> > $java -version
> > java version "1.6.0"
> > Java(TM) SE Runtime Environment (build pxa6460sr7-20091215_02(SR7))
> > IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64
> > jvmxa6460sr7-20091214_49398 (JIT enabled, AOT enabled)
> > J9VM - 20091214_049398
> > JIT  - r9_20091123_13891
> > GC   - 20091111_AA)
> > JCL  - 20091202_01
> >
> > With Maven version 3.03
> >
> > $mvn -version
> > mvn -version
> > Apache Maven 3.0.3 (r1075438; 2011-02-28 12:31:09-0500)
> > Maven home: /home/mszetom/apache-maven-3.0.3
> > Java version: 1.6.0, vendor: IBM Corporation
> > Java home: /opt/ibm/ibm-java-x86_64-60/jre
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "linux", version: "2.6.31-22-server", arch: "amd64", family:
> "unix"
> >
> > I should also add that I am not currently using Hadoop beneath Mahout
> yet, I
> > have not had
> > the time to get it running on our cluster.
> >
> >
> > I do not plan on using this functionality; however, I am more than happy
> to
> > help drill into this to
> > find the issue.
> >
> >
> > Thanks in advance,
> > marc
> >
>
>
>
> --
> Lance Norskog
> [EMAIL PROTECTED]
>

Mahout releases are not product releases (at this point), but more
milestones denoting accomplishment. If there is a problem with the IBM
JVM and the Mahout trunk, that is important. I'm sure that whatever
the problem, it is the same in 0.5 and the trunk, but please test with
the trunk.

On Fri, Jul 8, 2011 at 7:48 PM, Marc Millstone <[EMAIL PROTECTED]> wrote:
> No, I downloaded the src from the Mahout website with the .5 checkpoint.
> (file mahout-distribution-0.5-src.tar.gz)
>
>
> I will try to throw a Sun JVM into my home directory on the cluster  and try
> to isolate the failed unit test from the JVM choice.
> As you can see from the failed test, the answer is "almost" right.
>
>
> On Fri, Jul 8, 2011 at 7:42 PM, Lance Norskog <[EMAIL PROTECTED]> wrote:
>
>> Did you check out the Mahout trunk? That test might be broken now; it
>> should have worked in the 0.5 source release (checkpoint, really).
>>
>> On Fri, Jul 8, 2011 at 5:41 PM, Marc Millstone <[EMAIL PROTECTED]>
>> wrote:
>> > Hello all,
>> >
>> > I have started to play around with Mahout today.  During installation,
>> > the unit test testStartParallelFPGrowth is failing. (I am not sure
>> whether
>> > to send this post
>> > here or to the developer list).
>> >
>> > Adding the -e option to maven, we get the culprit
>> >
>> > Failed tests:
>> >
>>  testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
>> > expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
>> E]=1}>
>> >
>> > Unfortunately, this is an algorithmic area in which I have no background
>> (I
>> > work in optimization/linear algebra) so I don't feel comfortable digging
>> > into this myself.
>> >
>> > I downloaded the Mahout src, version .5 this afternoon.  The only small
>> > hitch in my setup
>> > is that I am running this under the IBM Java JDK:
>> >
>> >
>> > $java -version
>> > java version "1.6.0"
>> > Java(TM) SE Runtime Environment (build pxa6460sr7-20091215_02(SR7))
>> > IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64
>> > jvmxa6460sr7-20091214_49398 (JIT enabled, AOT enabled)
>> > J9VM - 20091214_049398
>> > JIT  - r9_20091123_13891
>> > GC   - 20091111_AA)
>> > JCL  - 20091202_01
>> >
>> > With Maven version 3.03
>> >
>> > $mvn -version
>> > mvn -version
>> > Apache Maven 3.0.3 (r1075438; 2011-02-28 12:31:09-0500)
>> > Maven home: /home/mszetom/apache-maven-3.0.3
>> > Java version: 1.6.0, vendor: IBM Corporation
>> > Java home: /opt/ibm/ibm-java-x86_64-60/jre
>> > Default locale: en_US, platform encoding: UTF-8
>> > OS name: "linux", version: "2.6.31-22-server", arch: "amd64", family:
>> "unix"
>> >
>> > I should also add that I am not currently using Hadoop beneath Mahout
>> yet, I
>> > have not had
>> > the time to get it running on our cluster.
>> >
>> >
>> > I do not plan on using this functionality; however, I am more than happy
>> to
>> > help drill into this to
>> > find the issue.
>> >
>> >
>> > Thanks in advance,
>> > marc
>> >
>>
>>
>>
>> --
>> Lance Norskog
>> [EMAIL PROTECTED]
>>
>

--
Lance Norskog
[EMAIL PROTECTED]

The tests passed at 0.5, and now at trunk. I have tested on Apple JVMs and I
assume Jenkins uses a Sun JVM.
I doubt it is a JVM issue, though possible. But I don't see what it has to
do with being a product?

On Sat, Jul 9, 2011 at 6:11 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:

> Mahout releases are not product releases (at this point), but more
> milestones denoting accomplishment. If there is a problem with the IBM
> JVM and the Mahout trunk, that is important. I'm sure that whatever
> the problem, it is the same in 0.5 and the trunk, but please test with
> the trunk.
>
>

Thanks all.  I too noticed that everything worked with the Apple JVM on my
mac.   I will
checkout trunk, run tests and get back to you all.    What is confusing is
that
the test "almost" passes.  The numbers are right, just assigned to the wrong
variables.

In general, is this type of a post better to be placed on user or dev?

On Sat, Jul 9, 2011 at 12:50 AM, Sean Owen <[EMAIL PROTECTED]> wrote:

> The tests passed at 0.5, and now at trunk. I have tested on Apple JVMs and
> I
> assume Jenkins uses a Sun JVM.
> I doubt it is a JVM issue, though possible. But I don't see what it has to
> do with being a product?
>
> On Sat, Jul 9, 2011 at 6:11 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
>
> > Mahout releases are not product releases (at this point), but more
> > milestones denoting accomplishment. If there is a problem with the IBM
> > JVM and the Mahout trunk, that is important. I'm sure that whatever
> > the problem, it is the same in 0.5 and the trunk, but please test with
> > the trunk.
> >
> >
>

Hello all,

Well I have checked out trunk (
Last Changed Author: ssc
Last Changed Rev: 1145213
Last Changed Date: 2011-07-11 11:48:16 -0400 (Mon, 11 Jul 2011)
)
and am getting many many failing tests due to a

org.apache.hadoop.security.UserGroupInformation (initialization failure)

This is a different failure than with the .5 src.

I appreciate your help in advance.  I am not certain as to how to best go
about fixing this (but again, I am more than happy to try
anything/read anything, etc to help).

Thanks!
marc

On Mon, Jul 11, 2011 at 8:51 AM, Marc Millstone <[EMAIL PROTECTED]> wrote:

> Thanks all.  I too noticed that everything worked with the Apple JVM on my
> mac.   I will
> checkout trunk, run tests and get back to you all.    What is confusing is
> that
> the test "almost" passes.  The numbers are right, just assigned to the
> wrong variables.
>
> In general, is this type of a post better to be placed on user or dev?
>
>
> On Sat, Jul 9, 2011 at 12:50 AM, Sean Owen <[EMAIL PROTECTED]> wrote:
>
>> The tests passed at 0.5, and now at trunk. I have tested on Apple JVMs and
>> I
>> assume Jenkins uses a Sun JVM.
>> I doubt it is a JVM issue, though possible. But I don't see what it has to
>> do with being a product?
>>
>> On Sat, Jul 9, 2011 at 6:11 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
>>
>> > Mahout releases are not product releases (at this point), but more
>> > milestones denoting accomplishment. If there is a problem with the IBM
>> > JVM and the Mahout trunk, that is important. I'm sure that whatever
>> > the problem, it is the same in 0.5 and the trunk, but please test with
>> > the trunk.
>> >
>> >
>>
>
>

I'm not seeing this error -- can you please post more about the stack trace?
Coming from Hadoop, it seems more like an environment problem.

On Mon, Jul 11, 2011 at 5:46 PM, Marc Millstone <[EMAIL PROTECTED]> wrote:

> Hello all,
>
> Well I have checked out trunk (
> Last Changed Author: ssc
> Last Changed Rev: 1145213
> Last Changed Date: 2011-07-11 11:48:16 -0400 (Mon, 11 Jul 2011)
> )
> and am getting many many failing tests due to a
>
> org.apache.hadoop.security.UserGroupInformation (initialization failure)
>
> This is a different failure than with the .5 src.
>
> I appreciate your help in advance.  I am not certain as to how to best go
> about fixing this (but again, I am more than happy to try
> anything/read anything, etc to help).
>
> Thanks!
> marc
>
>
>
> On Mon, Jul 11, 2011 at 8:51 AM, Marc Millstone <[EMAIL PROTECTED]>
> wrote:
>
> > Thanks all.  I too noticed that everything worked with the Apple JVM on
> my
> > mac.   I will
> > checkout trunk, run tests and get back to you all.    What is confusing
> is
> > that
> > the test "almost" passes.  The numbers are right, just assigned to the
> > wrong variables.
> >
> > In general, is this type of a post better to be placed on user or dev?
> >
> >
> > On Sat, Jul 9, 2011 at 12:50 AM, Sean Owen <[EMAIL PROTECTED]> wrote:
> >
> >> The tests passed at 0.5, and now at trunk. I have tested on Apple JVMs
> and
> >> I
> >> assume Jenkins uses a Sun JVM.
> >> I doubt it is a JVM issue, though possible. But I don't see what it has
> to
> >> do with being a product?
> >>
> >> On Sat, Jul 9, 2011 at 6:11 AM, Lance Norskog <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > Mahout releases are not product releases (at this point), but more
> >> > milestones denoting accomplishment. If there is a problem with the IBM
> >> > JVM and the Mahout trunk, that is important. I'm sure that whatever
> >> > the problem, it is the same in 0.5 and the trunk, but please test with
> >> > the trunk.
> >> >
> >> >
> >>
> >
> >
>

Sean,

I am seeing stack-traces such as

    at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
    at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:600)
    at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
    at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
    at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test
failures.
and
    at
org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:74)
    at
org.apache.maven.plugin.surefire.SurefirePlugin.writeSummary(SurefirePlugin.java:644)
    at
org.apache.maven.plugin.surefire.SurefirePlugin.executeAfterPreconditionsChecked(SurefirePlugin.java:640)
    at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:102)
    at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
These traces seem to be simple test failure traces I think.

How can I better get information to help debug this?  I am running mvn -e -X
install in the core directory.
Everything is compiling perfectly, I am just seeing these unit test failures
(now with the reason
given above, with an initialization failure).   In an earlier post, I list
all
of my version information.  With version .5 of Mahout, the unit test failure
is actually something
where the output does not match expected output.  With trunk, it is due to
these user issues.

(Sorry I can't do more on my own.  I am not a java hacker/mvn guru, as I
spend most of my time in Scala).

Thanks in advance!
marc
On Mon, Jul 11, 2011 at 11:21 AM, Sean Owen <[EMAIL PROTECTED]> wrote:

> I'm not seeing this error -- can you please post more about the stack
> trace?
> Coming from Hadoop, it seems more like an environment problem.
>
> On Mon, Jul 11, 2011 at 5:46 PM, Marc Millstone <[EMAIL PROTECTED]>
> wrote:
>
> > Hello all,
> >
> > Well I have checked out trunk (
> > Last Changed Author: ssc
> > Last Changed Rev: 1145213
> > Last Changed Date: 2011-07-11 11:48:16 -0400 (Mon, 11 Jul 2011)
> > )
> > and am getting many many failing tests due to a
> >
> > org.apache.hadoop.security.UserGroupInformation (initialization failure)
> >
> > This is a different failure than with the .5 src.
> >
> > I appreciate your help in advance.  I am not certain as to how to best go
> > about fixing this (but again, I am more than happy to try
> > anything/read anything, etc to help).
> >
> > Thanks!
> > marc
> >
> >
> >
> > On Mon, Jul 11, 2011 at 8:51 AM, Marc Millstone <[EMAIL PROTECTED]>

Yeah that's not it. It's just internal exceptions because a test failed.

What were you referring to by "org.apache.hadoop.security.UserGroupInformation
(initialization failure)" ? That sounds closer to a problem.

In the output of the failed unit test results, the reason for failure is
given as
testLoad(org.apache.mahout.df.mapreduce.partial.InterResultsTest):
com.sun.security.auth.UnixPrincipal
  testStore(org.apache.mahout.df.mapreduce.partial.InterResultsTest):
org.apache.hadoop.security.UserGroupInformation (initialization failure)
  testProcessOutput(org.apache.mahout.df.mapreduce.partial.PartialBuilderTest):
org.apache.hadoop.security.UserGroupInformation (initialization failure)
  testStep0Mapper(org.apache.mahout.df.mapreduce.partial.Step0JobTest):
org.apache.hadoop.security.UserGroupInformation (initialization failure)

.....

In .5 the unit test failure was

  testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0, E]=1}>

On Mon, Jul 11, 2011 at 12:05 PM, Sean Owen <[EMAIL PROTECTED]> wrote:

> Yeah that's not it. It's just internal exceptions because a test failed.
>
> What were you referring to by
> "org.apache.hadoop.security.UserGroupInformation
> (initialization failure)" ? That sounds closer to a problem.
>

That's no good. Sounds like some local file system permissions issue? Are
there any restrictions on /tmp? maybe blow away anything that looks Hadoop-
or Mahout-related there.

You can dig into core/target/surefire-reports and find a file with all
output for every test. It will surely have more detail on what happened.

(Or you can perhaps run this test alone in your IDE; that's what I do.)

On Mon, Jul 11, 2011 at 8:07 PM, Marc Millstone <[EMAIL PROTECTED]> wrote:

> In the output of the failed unit test results, the reason for failure is
> given as
>
>
> testLoad(org.apache.mahout.df.mapreduce.partial.InterResultsTest):
> com.sun.security.auth.UnixPrincipal
>  testStore(org.apache.mahout.df.mapreduce.partial.InterResultsTest):
> org.apache.hadoop.security.UserGroupInformation (initialization failure)
>  testProcessOutput(org.apache.mahout.df.mapreduce.partial.PartialBuilderTest):
> org.apache.hadoop.security.UserGroupInformation (initialization failure)
>   testStep0Mapper(org.apache.mahout.df.mapreduce.partial.Step0JobTest):
> org.apache.hadoop.security.UserGroupInformation (initialization failure)
>
> .....
>
> In .5 the unit test failure was
>
>  testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
> E]=1}>
>
> On Mon, Jul 11, 2011 at 12:05 PM, Sean Owen <[EMAIL PROTECTED]> wrote:
>
> > Yeah that's not it. It's just internal exceptions because a test failed.
> >
> > What were you referring to by
> > "org.apache.hadoop.security.UserGroupInformation
> > (initialization failure)" ? That sounds closer to a problem.
> >
>

Ok.  I am starting to think this could be a IBM JVM thing
after finding this JIRA: https://issues.apache.org/jira/browse/HADOOP-7211
Could it be that something changed between trunk and .5 that caused this
issue to manifest itself in different ways?
In .5, we just got the wrong answer and in trunk, we are getting this
permission issue?

marc

On Mon, Jul 11, 2011 at 12:09 PM, Sean Owen <[EMAIL PROTECTED]> wrote:

> That's no good. Sounds like some local file system permissions issue? Are
> there any restrictions on /tmp? maybe blow away anything that looks Hadoop-
> or Mahout-related there.
>
> You can dig into core/target/surefire-reports and find a file with all
> output for every test. It will surely have more detail on what happened.
>
> (Or you can perhaps run this test alone in your IDE; that's what I do.)
>
> On Mon, Jul 11, 2011 at 8:07 PM, Marc Millstone <[EMAIL PROTECTED]>
> wrote:
>
> > In the output of the failed unit test results, the reason for failure is
> > given as
> >
> >
> > testLoad(org.apache.mahout.df.mapreduce.partial.InterResultsTest):
> > com.sun.security.auth.UnixPrincipal
> >  testStore(org.apache.mahout.df.mapreduce.partial.InterResultsTest):
> > org.apache.hadoop.security.UserGroupInformation (initialization failure)
> >
>  testProcessOutput(org.apache.mahout.df.mapreduce.partial.PartialBuilderTest):
> > org.apache.hadoop.security.UserGroupInformation (initialization failure)
> >   testStep0Mapper(org.apache.mahout.df.mapreduce.partial.Step0JobTest):
> > org.apache.hadoop.security.UserGroupInformation (initialization failure)
> >
> > .....
> >
> > In .5 the unit test failure was
> >
> >
>  testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> > expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
> > E]=1}>
> >
> > On Mon, Jul 11, 2011 at 12:05 PM, Sean Owen <[EMAIL PROTECTED]> wrote:
> >
> > > Yeah that's not it. It's just internal exceptions because a test
> failed.
> > >
> > > What were you referring to by
> > > "org.apache.hadoop.security.UserGroupInformation
> > > (initialization failure)" ? That sounds closer to a problem.
> > >
> >
>

I strongly suspect it is because we updated to Hadoop 0.20.203.0. 0.5
depended on 0.20.2. You could locally go back to depending on 0.20.2; it
ought to still work fine with the previous version.

On Mon, Jul 11, 2011 at 8:14 PM, Marc Millstone <[EMAIL PROTECTED]> wrote:

> Ok.  I am starting to think this could be a IBM JVM thing
> after finding this JIRA: https://issues.apache.org/jira/browse/HADOOP-7211
>
>
> Could it be that something changed between trunk and .5 that caused this
> issue to manifest itself in different ways?
> In .5, we just got the wrong answer and in trunk, we are getting this
> permission issue?
>
> marc
>
>
>

Ok.  Let me play with this some more.  I appreciate your help.

In .5, under the IBM JVM we still get the Unit test failure in PFPGrowth
though.  It seems
to be a true failure too, not an initialization issue.

I am now going to install the Sun JVM locally on my cluster and link against
that to see if things pass for Mahout .5.

Feel free to contact me off list if this is getting to be too detail
oriented, etc for this list.

On Mon, Jul 11, 2011 at 12:18 PM, Sean Owen <[EMAIL PROTECTED]> wrote:

> I strongly suspect it is because we updated to Hadoop 0.20.203.0. 0.5
> depended on 0.20.2. You could locally go back to depending on 0.20.2; it
> ought to still work fine with the previous version.
>
> On Mon, Jul 11, 2011 at 8:14 PM, Marc Millstone <[EMAIL PROTECTED]>
> wrote:
>
> > Ok.  I am starting to think this could be a IBM JVM thing
> > after finding this JIRA:
> https://issues.apache.org/jira/browse/HADOOP-7211
> >
> >
> > Could it be that something changed between trunk and .5 that caused this
> > issue to manifest itself in different ways?
> > In .5, we just got the wrong answer and in trunk, we are getting this
> > permission issue?
> >
> > marc
> >
> >
> >
>

Good News:  The offending unit tests passes with the combination of

IBM JVM + Mahout Trunk + Hadoop 0.20.2
To recap:

IBM JVM + Mahout .5 + Hadoop 0.20.2 fails with the error

testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0, E]=1}>

IBM JVM + Mahout trunk + Hadoop  0.20.203.0  fails with user initialization
problems due to IBM JVM and the sun com.sun.security.auth.UnixPrincipal class.
 Is this dependency going to be updated in the future?  It does seem to
limit the use of non-Sun JVM's.

Are there any tickets that should be filed?

On Mon, Jul 11, 2011 at 12:22 PM, Marc Millstone <[EMAIL PROTECTED]>wrote:

> Ok.  Let me play with this some more.  I appreciate your help.
>
> In .5, under the IBM JVM we still get the Unit test failure in PFPGrowth
> though.  It seems
> to be a true failure too, not an initialization issue.
>
> I am now going to install the Sun JVM locally on my cluster and link
> against
> that to see if things pass for Mahout .5.
>
> Feel free to contact me off list if this is getting to be too detail
> oriented, etc for this list.
>
> On Mon, Jul 11, 2011 at 12:18 PM, Sean Owen <[EMAIL PROTECTED]> wrote:
>
>> I strongly suspect it is because we updated to Hadoop 0.20.203.0. 0.5
>> depended on 0.20.2. You could locally go back to depending on 0.20.2; it
>> ought to still work fine with the previous version.
>>
>> On Mon, Jul 11, 2011 at 8:14 PM, Marc Millstone <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Ok.  I am starting to think this could be a IBM JVM thing
>> > after finding this JIRA:
>> https://issues.apache.org/jira/browse/HADOOP-7211
>> >
>> >
>> > Could it be that something changed between trunk and .5 that caused this
>> > issue to manifest itself in different ways?
>> > In .5, we just got the wrong answer and in trunk, we are getting this
>> > permission issue?
>> >
>> > marc
>> >
>> >
>> >
>>
>
>

It sounds like the actual test failure does not occur in HEAD, so sounds
like whatever it is is already fixed or rendered moot.

The issue about Hadoop 0.20.203.0 is a Hadoop issue at best, and looks like
they're already tracking it. I think it should be fixed of course, but it's
not a Mahout thing.

I am not seeing a Mahout issue here then.

On Mon, Jul 11, 2011 at 8:43 PM, Marc Millstone <[EMAIL PROTECTED]> wrote:

> Good News:  The offending unit tests passes with the combination of
>
> IBM JVM + Mahout Trunk + Hadoop 0.20.2
>
>
> To recap:
>
> IBM JVM + Mahout .5 + Hadoop 0.20.2 fails with the error
>
> testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
> E]=1}>
>
> IBM JVM + Mahout trunk + Hadoop  0.20.203.0  fails with user initialization
> problems due to IBM JVM and the sun com.sun.security.auth.UnixPrincipal
> class.
>  Is this dependency going to be updated in the future?  It does seem to
> limit the use of non-Sun JVM's.
>
> Are there any tickets that should be filed?
>
>
>
> On Mon, Jul 11, 2011 at 12:22 PM, Marc Millstone <[EMAIL PROTECTED]
> >wrote:
>
> > Ok.  Let me play with this some more.  I appreciate your help.
> >
> > In .5, under the IBM JVM we still get the Unit test failure in PFPGrowth
> > though.  It seems
> > to be a true failure too, not an initialization issue.
> >
> > I am now going to install the Sun JVM locally on my cluster and link
> > against
> > that to see if things pass for Mahout .5.
> >
> > Feel free to contact me off list if this is getting to be too detail
> > oriented, etc for this list.
> >
> > On Mon, Jul 11, 2011 at 12:18 PM, Sean Owen <[EMAIL PROTECTED]> wrote:
> >
> >> I strongly suspect it is because we updated to Hadoop 0.20.203.0. 0.5
> >> depended on 0.20.2. You could locally go back to depending on 0.20.2; it
> >> ought to still work fine with the previous version.
> >>
> >> On Mon, Jul 11, 2011 at 8:14 PM, Marc Millstone <[EMAIL PROTECTED]>
> >> wrote:
> >>
> >> > Ok.  I am starting to think this could be a IBM JVM thing
> >> > after finding this JIRA:
> >> https://issues.apache.org/jira/browse/HADOOP-7211
> >> >
> >> >
> >> > Could it be that something changed between trunk and .5 that caused
> this
> >> > issue to manifest itself in different ways?
> >> > In .5, we just got the wrong answer and in trunk, we are getting this
> >> > permission issue?
> >> >
> >> > marc
> >> >
> >> >
> >> >
> >>
> >
> >
>

Awesome.  I agree entirely with your assessment.

Thank you for your help, I do appreciate the time it takes to help
strangers over the internet.  And it is what makes this community
great.

Is there a place on the wiki for little facts like this?  I am more
than happy to add something to help others that may encounter
this issue.  (I can't be the only person using the IBM JVM with
Mahout).
On Mon, Jul 11, 2011 at 12:47 PM, Sean Owen <[EMAIL PROTECTED]> wrote:

> It sounds like the actual test failure does not occur in HEAD, so sounds
> like whatever it is is already fixed or rendered moot.
>
> The issue about Hadoop 0.20.203.0 is a Hadoop issue at best, and looks like
> they're already tracking it. I think it should be fixed of course, but it's
> not a Mahout thing.
>
> I am not seeing a Mahout issue here then.
>
> On Mon, Jul 11, 2011 at 8:43 PM, Marc Millstone <[EMAIL PROTECTED]>
> wrote:
>
> > Good News:  The offending unit tests passes with the combination of
> >
> > IBM JVM + Mahout Trunk + Hadoop 0.20.2
> >
> >
> > To recap:
> >
> > IBM JVM + Mahout .5 + Hadoop 0.20.2 fails with the error
> >
> > testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> > expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
> > E]=1}>
> >
> > IBM JVM + Mahout trunk + Hadoop  0.20.203.0  fails with user
> initialization
> > problems due to IBM JVM and the sun com.sun.security.auth.UnixPrincipal
> > class.
> >  Is this dependency going to be updated in the future?  It does seem to
> > limit the use of non-Sun JVM's.
> >
> > Are there any tickets that should be filed?
> >
> >
> >
> > On Mon, Jul 11, 2011 at 12:22 PM, Marc Millstone <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Ok.  Let me play with this some more.  I appreciate your help.
> > >
> > > In .5, under the IBM JVM we still get the Unit test failure in
> PFPGrowth
> > > though.  It seems
> > > to be a true failure too, not an initialization issue.
> > >
> > > I am now going to install the Sun JVM locally on my cluster and link
> > > against
> > > that to see if things pass for Mahout .5.
> > >
> > > Feel free to contact me off list if this is getting to be too detail
> > > oriented, etc for this list.
> > >
> > > On Mon, Jul 11, 2011 at 12:18 PM, Sean Owen <[EMAIL PROTECTED]> wrote:
> > >
> > >> I strongly suspect it is because we updated to Hadoop 0.20.203.0. 0.5
> > >> depended on 0.20.2. You could locally go back to depending on 0.20.2;
> it
> > >> ought to still work fine with the previous version.
> > >>
> > >> On Mon, Jul 11, 2011 at 8:14 PM, Marc Millstone <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > Ok.  I am starting to think this could be a IBM JVM thing
> > >> > after finding this JIRA:
> > >> https://issues.apache.org/jira/browse/HADOOP-7211
> > >> >
> > >> >
> > >> > Could it be that something changed between trunk and .5 that caused
> > this
> > >> > issue to manifest itself in different ways?
> > >> > In .5, we just got the wrong answer and in trunk, we are getting
> this
> > >> > permission issue?
> > >> >
> > >> > marc
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

You're welcome, and welcome to add to the wiki as you see fit:
https://cwiki.apache.org/MAHOUT/mahout-wiki.html
Maybe one of the Installation/Setup pages is a good home for a note?

On Mon, Jul 11, 2011 at 8:53 PM, Marc Millstone <[EMAIL PROTECTED]> wrote:

> Awesome.  I agree entirely with your assessment.
>
> Thank you for your help, I do appreciate the time it takes to help
> strangers over the internet.  And it is what makes this community
> great.
>
> Is there a place on the wiki for little facts like this?  I am more
> than happy to add something to help others that may encounter
> this issue.  (I can't be the only person using the IBM JVM with
> Mahout).
>
>
>
>
> On Mon, Jul 11, 2011 at 12:47 PM, Sean Owen <[EMAIL PROTECTED]> wrote:
>
> > It sounds like the actual test failure does not occur in HEAD, so sounds
> > like whatever it is is already fixed or rendered moot.
> >
> > The issue about Hadoop 0.20.203.0 is a Hadoop issue at best, and looks
> like
> > they're already tracking it. I think it should be fixed of course, but
> it's
> > not a Mahout thing.
> >
> > I am not seeing a Mahout issue here then.
> >
> > On Mon, Jul 11, 2011 at 8:43 PM, Marc Millstone <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Good News:  The offending unit tests passes with the combination of
> > >
> > > IBM JVM + Mahout Trunk + Hadoop 0.20.2
> > >
> > >
> > > To recap:
> > >
> > > IBM JVM + Mahout .5 + Hadoop 0.20.2 fails with the error
> > >
> > >
> testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> > > expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
> > > E]=1}>
> > >
> > > IBM JVM + Mahout trunk + Hadoop  0.20.203.0  fails with user
> > initialization
> > > problems due to IBM JVM and the sun com.sun.security.auth.UnixPrincipal
> > > class.
> > >  Is this dependency going to be updated in the future?  It does seem to
> > > limit the use of non-Sun JVM's.
> > >
> > > Are there any tickets that should be filed?
> > >
> > >
> > >
> > > On Mon, Jul 11, 2011 at 12:22 PM, Marc Millstone <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Ok.  Let me play with this some more.  I appreciate your help.
> > > >
> > > > In .5, under the IBM JVM we still get the Unit test failure in
> > PFPGrowth
> > > > though.  It seems
> > > > to be a true failure too, not an initialization issue.
> > > >
> > > > I am now going to install the Sun JVM locally on my cluster and link
> > > > against
> > > > that to see if things pass for Mahout .5.
> > > >
> > > > Feel free to contact me off list if this is getting to be too detail
> > > > oriented, etc for this list.
> > > >
> > > > On Mon, Jul 11, 2011 at 12:18 PM, Sean Owen <[EMAIL PROTECTED]>
> wrote:
> > > >
> > > >> I strongly suspect it is because we updated to Hadoop 0.20.203.0.
> 0.5
> > > >> depended on 0.20.2. You could locally go back to depending on
> 0.20.2;
> > it
> > > >> ought to still work fine with the previous version.
> > > >>
> > > >> On Mon, Jul 11, 2011 at 8:14 PM, Marc Millstone <
> [EMAIL PROTECTED]>
> > > >> wrote:
> > > >>
> > > >> > Ok.  I am starting to think this could be a IBM JVM thing
> > > >> > after finding this JIRA:
> > > >> https://issues.apache.org/jira/browse/HADOOP-7211
> > > >> >
> > > >> >
> > > >> > Could it be that something changed between trunk and .5 that
> caused
> > > this
> > > >> > issue to manifest itself in different ways?
> > > >> > In .5, we just got the wrong answer and in trunk, we are getting
> > this
> > > >> > permission issue?
> > > >> >
> > > >> > marc
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Great.  I will do this later this afternoon.
On Mon, Jul 11, 2011 at 12:55 PM, Sean Owen <[EMAIL PROTECTED]> wrote:

> You're welcome, and welcome to add to the wiki as you see fit:
> https://cwiki.apache.org/MAHOUT/mahout-wiki.html
> Maybe one of the Installation/Setup pages is a good home for a note?
>
> On Mon, Jul 11, 2011 at 8:53 PM, Marc Millstone <[EMAIL PROTECTED]>
> wrote:
>
> > Awesome.  I agree entirely with your assessment.
> >
> > Thank you for your help, I do appreciate the time it takes to help
> > strangers over the internet.  And it is what makes this community
> > great.
> >
> > Is there a place on the wiki for little facts like this?  I am more
> > than happy to add something to help others that may encounter
> > this issue.  (I can't be the only person using the IBM JVM with
> > Mahout).
> >
> >
> >
> >
> > On Mon, Jul 11, 2011 at 12:47 PM, Sean Owen <[EMAIL PROTECTED]> wrote:
> >
> > > It sounds like the actual test failure does not occur in HEAD, so
> sounds
> > > like whatever it is is already fixed or rendered moot.
> > >
> > > The issue about Hadoop 0.20.203.0 is a Hadoop issue at best, and looks
> > like
> > > they're already tracking it. I think it should be fixed of course, but
> > it's
> > > not a Mahout thing.
> > >
> > > I am not seeing a Mahout issue here then.
> > >
> > > On Mon, Jul 11, 2011 at 8:43 PM, Marc Millstone <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Good News:  The offending unit tests passes with the combination of
> > > >
> > > > IBM JVM + Mahout Trunk + Hadoop 0.20.2
> > > >
> > > >
> > > > To recap:
> > > >
> > > > IBM JVM + Mahout .5 + Hadoop 0.20.2 fails with the error
> > > >
> > > >
> > testStartParallelFPGrowth(org.apache.mahout.fpm.pfpgrowth.PFPGrowthTest):
> > > > expected:<{[D=0, E=1, A=0, B=0, C]=1}> but was:<{[A=0, B=0, C=1, D=0,
> > > > E]=1}>
> > > >
> > > > IBM JVM + Mahout trunk + Hadoop  0.20.203.0  fails with user
> > > initialization
> > > > problems due to IBM JVM and the sun
> com.sun.security.auth.UnixPrincipal
> > > > class.
> > > >  Is this dependency going to be updated in the future?  It does seem
> to
> > > > limit the use of non-Sun JVM's.
> > > >
> > > > Are there any tickets that should be filed?
> > > >
> > > >
> > > >
> > > > On Mon, Jul 11, 2011 at 12:22 PM, Marc Millstone <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Ok.  Let me play with this some more.  I appreciate your help.
> > > > >
> > > > > In .5, under the IBM JVM we still get the Unit test failure in
> > > PFPGrowth
> > > > > though.  It seems
> > > > > to be a true failure too, not an initialization issue.
> > > > >
> > > > > I am now going to install the Sun JVM locally on my cluster and
> link
> > > > > against
> > > > > that to see if things pass for Mahout .5.
> > > > >
> > > > > Feel free to contact me off list if this is getting to be too
> detail
> > > > > oriented, etc for this list.
> > > > >
> > > > > On Mon, Jul 11, 2011 at 12:18 PM, Sean Owen <[EMAIL PROTECTED]>
> > wrote:
> > > > >
> > > > >> I strongly suspect it is because we updated to Hadoop 0.20.203.0.
> > 0.5
> > > > >> depended on 0.20.2. You could locally go back to depending on
> > 0.20.2;
> > > it
> > > > >> ought to still work fine with the previous version.
> > > > >>
> > > > >> On Mon, Jul 11, 2011 at 8:14 PM, Marc Millstone <
> > [EMAIL PROTECTED]>
> > > > >> wrote:
> > > > >>
> > > > >> > Ok.  I am starting to think this could be a IBM JVM thing
> > > > >> > after finding this JIRA:
> > > > >> https://issues.apache.org/jira/browse/HADOOP-7211
> > > > >> >
> > > > >> >
> > > > >> > Could it be that something changed between trunk and .5 that
> > caused
> > > > this
> > > > >> > issue to manifest itself in different ways?
> > > > >> > In .5, we just got the wrong answer and in trunk, we are getting
> > > this
> > > > >> > permission issue?
> > > > >> >
> > > > >> > marc
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>