I just realize that there is mismatch between the trainclassifier command
versus TrainClassifier.java program.

Here is why:

1. I run trainclassifier program with source:hdfs. So, I provide input and
output locations on my hdfs. Using training files, the program generates the
model and puts it on my hdfs, as expected. When I provide input and output
locations on my local fs while the source is set to hdfs, it complains and
gives me "Input path does not exist" error, which is understandable.

2. However, when I run TrainClassifier.java program with source set to hdfs
and input and output set to locations on my local fs, it accepts the
arguments with no complaints and generates the model on my local fs (instead
of hdfs).

In addition, the models generated by these two programs are slightly
different as far as the log files go.

Is this a known case, or am I missing something?

--
View this message in context: http://lucene.472066.n3.nabble.com/trainclassifier-as-a-command-vs-TrainClassifier-java-tp3508652p3508652.html
Sent from the Mahout User List mailing list archive at Nabble.com.

I just realize that there is mismatch between the trainclassifier command versus
TrainClassifier.java program.

Here is why:

1. I run trainclassifier program with source:hdfs. So, I provide input and
output locations on my hdfs. Using training files, the program generates the
model and puts it on my hdfs, as expected. When I provide input and output
locations on my local fs while the source is set to hdfs, it complains and gives
me "Input path does not exist" error, which is understandable.

2. However, when I run TrainClassifier.java program with source set to hdfs and
input and output set to locations on my local fs, it accepts the arguments with
no complaints and generates the model on my local fs (instead of hdfs).

In addition, the models generated by these two programs are slightly different
as far as the log files go.

Is this a known case, or am I missing something?

Well, the worst thing is that when I run the TrainClassifier.java, it doesn't
distribute the job to datanodes even though the source is set to hdfs.
Below, I am providing outputs for running trainclassifier command and
TrainClassifier.java.

Here is the output of running trainclassifier command:

sayhan@A4915037:~$ $MAHOUT_HOME/bin/mahout trainclassifier -i articles-train
-o articles-model -type cbayes -ng 1 -source hdfs
Running on hadoop, using HADOOP_HOME=/usr/local/hadoop-0.20.2
HADOOP_CONF_DIR=/usr/local/hadoop-0.20.2/conf
11/11/14 22:14:25 WARN driver.MahoutDriver: No trainclassifier.props found
on classpath, will use command-line arguments only
11/11/14 22:14:25 INFO bayes.TrainClassifier: Training Complementary Bayes
Classifier
11/11/14 22:14:26 INFO common.HadoopUtil: Deleting articles-model
11/11/14 22:14:26 INFO cbayes.CBayesDriver: Reading features...
11/11/14 22:14:26 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/11/14 22:14:26 INFO mapred.FileInputFormat: Total input paths to process
: 7
11/11/14 22:14:27 INFO mapred.JobClient: Running job: job_201111142210_0001
11/11/14 22:14:28 INFO mapred.JobClient:  map 0% reduce 0%
11/11/14 22:14:39 INFO mapred.JobClient:  map 28% reduce 0%
11/11/14 22:14:45 INFO mapred.JobClient:  map 57% reduce 0%
11/11/14 22:14:48 INFO mapred.JobClient:  map 71% reduce 9%
11/11/14 22:14:51 INFO mapred.JobClient:  map 100% reduce 9%
11/11/14 22:14:57 INFO mapred.JobClient:  map 100% reduce 23%
11/11/14 22:15:03 INFO mapred.JobClient:  map 100% reduce 100%
11/11/14 22:15:05 INFO mapred.JobClient: Job complete: job_201111142210_0001
11/11/14 22:15:05 INFO mapred.JobClient: Counters: 18
11/11/14 22:15:05 INFO mapred.JobClient:   Job Counters
11/11/14 22:15:05 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/14 22:15:05 INFO mapred.JobClient:     Launched map tasks=7
11/11/14 22:15:05 INFO mapred.JobClient:     Data-local map tasks=7
11/11/14 22:15:05 INFO mapred.JobClient:   FileSystemCounters
11/11/14 22:15:05 INFO mapred.JobClient:     FILE_BYTES_READ=2473171
11/11/14 22:15:05 INFO mapred.JobClient:     HDFS_BYTES_READ=404467
11/11/14 22:15:05 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=4946602
11/11/14 22:15:05 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=2078615
11/11/14 22:15:05 INFO mapred.JobClient:   Map-Reduce Framework
11/11/14 22:15:05 INFO mapred.JobClient:     Reduce input groups=64653
11/11/14 22:15:05 INFO mapred.JobClient:     Combine output records=79675
11/11/14 22:15:05 INFO mapred.JobClient:     Map input records=97
11/11/14 22:15:05 INFO mapred.JobClient:     Reduce shuffle bytes=2288236
11/11/14 22:15:05 INFO mapred.JobClient:     Reduce output records=52247
11/11/14 22:15:05 INFO mapred.JobClient:     Spilled Records=159350
11/11/14 22:15:05 INFO mapred.JobClient:     Map output bytes=4039212
11/11/14 22:15:05 INFO mapred.JobClient:     Map input bytes=404467
11/11/14 22:15:05 INFO mapred.JobClient:     Combine input records=142673
11/11/14 22:15:05 INFO mapred.JobClient:     Map output records=142673
11/11/14 22:15:05 INFO mapred.JobClient:     Reduce input records=79675
11/11/14 22:15:05 INFO cbayes.CBayesDriver: Calculating Tf-Idf...
11/11/14 22:15:05 INFO common.BayesTfIdfDriver: Counts of documents in Each
Label
11/11/14 22:15:05 INFO common.BayesTfIdfDriver: {General=29.0,
Business=21.0, Politics=10.0, SciTech=9.0, Entertainment=8.0, Health=9.0,
Sports=11.0}
11/11/14 22:15:05 INFO common.BayesTfIdfDriver: {dataSource=hdfs,
alpha_i=1.0, minDf=1, gramSize=1}
11/11/14 22:15:05 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/11/14 22:15:05 INFO mapred.FileInputFormat: Total input paths to process
: 3
11/11/14 22:15:06 INFO mapred.JobClient: Running job: job_201111142210_0002
11/11/14 22:15:07 INFO mapred.JobClient:  map 0% reduce 0%
11/11/14 22:15:18 INFO mapred.JobClient:  map 66% reduce 0%
11/11/14 22:15:21 INFO mapred.JobClient:  map 100% reduce 0%
11/11/14 22:15:30 INFO mapred.JobClient:  map 100% reduce 100%
11/11/14 22:15:32 INFO mapred.JobClient: Job complete: job_201111142210_0002
11/11/14 22:15:32 INFO mapred.JobClient: Counters: 18
11/11/14 22:15:32 INFO mapred.JobClient:   Job Counters
11/11/14 22:15:32 INFO mapred.JobClient:     Launched reduce tasks=1
11/11/14 22:15:32 INFO mapred.JobClient:     Launched map tasks=3
11/11/14 22:15:32 INFO mapred.JobClient:     Data-local map tasks=3
11/11/14 22:15:32 INFO mapred.JobClient:   FileSystemCounters
11/11/14 22:15:32 INFO mapred.JobClient:     FILE_BYTES_READ=1405021
11/11/14 22:15:32 INFO mapred.JobClient:     HDFS_BYTES_READ=2078279
11/11/14 22:15:32 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=2810150
11/11/14 22:15:32 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=830343
11/11/14 22:15:32 INFO mapred.JobClient:   Map-Reduce Framework
11/11/14 22:15:32 INFO mapred.JobClient:     Reduce input groups=19918
11/11/14 22:15:32 INFO mapred.JobClient:     Combine output records=39835
11/11/14 22:15:32 INFO mapred.JobClient:     Map input records=52240
11/11/14 22:15:32 INFO mapred.JobClient:     Reduce shuffle bytes=1405033
11/11/14 22:15:32 INFO mapred.JobClient:     Reduce output records=19918
11/11/14 22:15:32 INFO mapred.JobClient:     Spilled Records=79670
11/11/14 22:15:32 INFO mapred.JobClient:     Map output bytes=1536230
11/11/14 22:15:32 INFO mapred.JobClient:     Map input bytes=2077982
11/11/14 22:15:32 INFO mapred.JobClient:     Combine input records=52240
11/11/14 22:15:32 INFO mapred.JobClient:     Map output records=52240
11/11/14 22:15:32 INFO mapred.JobClient:     Reduce input records=39835
11/11/14 22:15:32 INFO cbayes.CBayesDriver: Calculating weight sums for
labels and features...
11/11/14 22:15:32 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
11/11/14 22:15:32 INFO mapred.FileInputFormat: Total input paths to

Well, the worst thing is that when I run the TrainClassifier.java, it doesn't
distribute the job to datanodes even though the source is set to hdfs.

On 15.11.2011 Sam Cunningham wrote:
> 2. However, when I run TrainClassifier.java program with source set to hdfs
> and input and output set to locations on my local fs, it accepts the
> arguments with no complaints and generates the model on my local fs
> (instead of hdfs).
>
> In addition, the models generated by these two programs are slightly
> different as far as the log files go.
>
> Is this a known case, or am I missing something?

The inconsitency may have been caused by the fact that the mahout script uses
"MahoutDriver.java" to start individual jobs and handle common arguments.

Anyone else have input on this one?

Isabel