Sure Paritosh, go ahead and take a crack at it. I am moving from CO to
PA for the next few weeks and won't be able to do much coding during
that period. I suspect you will also need to modify Canopy to emit
ClusterWritable and also the RandomSeedGenerator.

Smooth sailing,
Jeff

On 3/14/12 8:28 PM, Paritosh Ranjan (Commented) (JIRA) wrote:
>      [ https://issues.apache.org/jira/browse/MAHOUT-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229840#comment-13229840 ]
>
> Paritosh Ranjan commented on MAHOUT-988:
> ----------------------------------------
>
> Jeff, I would like to work on this issue (or MAHOUT-989, or MAHOUT-990). Can I? I might also need some help ( at least the first patch review ).
>
>
>> Convert K-means buildClusters to use new ClusterIterator
>> --------------------------------------------------------
>>
>>                  Key: MAHOUT-988
>>                  URL: https://issues.apache.org/jira/browse/MAHOUT-988
>>              Project: Mahout
>>           Issue Type: Sub-task
>>           Components: Clustering
>>     Affects Versions: 0.6
>>             Reporter: Jeff Eastman
>>             Assignee: Jeff Eastman
>>              Fix For: 0.7
>>
>>
>> Refactor the current K-means implementation to use the ClusterIterator/Classifier implementation. This will replace the mapper, combiner, reducer, clusterer and many unit tests but will not modify the other driver APIs, thus retaining compatibility with existing CLI.
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
>

Thanks Jeff. One question, are "Use ClusterIterator" tasks dependent on
"Modify Canopy etc to use ClusterWritable" task ?
I am assuming that all subtasks in MAHOUT-933
<https://issues.apache.org/jira/browse/MAHOUT-933> are independent of
each other and the order to pick them does not matter. Am I correct?

On 15-03-2012 09:23, Jeff Eastman wrote:
> Sure Paritosh, go ahead and take a crack at it. I am moving from CO to
> PA for the next few weeks and won't be able to do much coding during
> that period. I suspect you will also need to modify Canopy to emit
> ClusterWritable and also the RandomSeedGenerator.
>
> Smooth sailing,
> Jeff
>
> On 3/14/12 8:28 PM, Paritosh Ranjan (Commented) (JIRA) wrote:
>>      [
>> https://issues.apache.org/jira/browse/MAHOUT-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229840#comment-13229840
>> ]
>>
>> Paritosh Ranjan commented on MAHOUT-988:
>> ----------------------------------------
>>
>> Jeff, I would like to work on this issue (or MAHOUT-989, or
>> MAHOUT-990). Can I? I might also need some help ( at least the first
>> patch review ).
>>
>>
>>> Convert K-means buildClusters to use new ClusterIterator
>>> --------------------------------------------------------
>>>
>>>                  Key: MAHOUT-988
>>>                  URL: https://issues.apache.org/jira/browse/MAHOUT-988
>>>              Project: Mahout
>>>           Issue Type: Sub-task
>>>           Components: Clustering
>>>     Affects Versions: 0.6
>>>             Reporter: Jeff Eastman
>>>             Assignee: Jeff Eastman
>>>              Fix For: 0.7
>>>
>>>
>>> Refactor the current K-means implementation to use the
>>> ClusterIterator/Classifier implementation. This will replace the
>>> mapper, combiner, reducer, clusterer and many unit tests but will
>>> not modify the other driver APIs, thus retaining compatibility with
>>> existing CLI.
>> --
>> This message is automatically generated by JIRA.
>> If you think it was sent incorrectly, please contact your JIRA
>> administrators:
>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>>
>>
>>
>>
>

Yes, that was my point. below It may, in fact, be impossible to
implement and commit them independently since so much of Mahout
clustering depends upon the Cluster sequenceFile. You may be able to get
part way by moving the Canopy mods into the kmeans issue, but then the
cluster dumper and evaluator will not work with kmeans.

Ideas?

On 3/14/12 10:15 PM, Paritosh Ranjan wrote:
> Thanks Jeff. One question, are "Use ClusterIterator" tasks dependent
> on "Modify Canopy etc to use ClusterWritable" task ?
> I am assuming that all subtasks in MAHOUT-933
> <https://issues.apache.org/jira/browse/MAHOUT-933> are independent of
> each other and the order to pick them does not matter. Am I correct?
>
> On 15-03-2012 09:23, Jeff Eastman wrote:
>> Sure Paritosh, go ahead and take a crack at it. I am moving from CO
>> to PA for the next few weeks and won't be able to do much coding
>> during that period. I suspect you will also need to modify Canopy to
>> emit ClusterWritable and also the RandomSeedGenerator.
>>
>> Smooth sailing,
>> Jeff
>>
>> On 3/14/12 8:28 PM, Paritosh Ranjan (Commented) (JIRA) wrote:
>>>      [
>>> https://issues.apache.org/jira/browse/MAHOUT-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229840#comment-13229840
>>> ]
>>>
>>> Paritosh Ranjan commented on MAHOUT-988:
>>> ----------------------------------------
>>>
>>> Jeff, I would like to work on this issue (or MAHOUT-989, or
>>> MAHOUT-990). Can I? I might also need some help ( at least the first
>>> patch review ).
>>>
>>>
>>>> Convert K-means buildClusters to use new ClusterIterator
>>>> --------------------------------------------------------
>>>>
>>>>                  Key: MAHOUT-988
>>>>                  URL: https://issues.apache.org/jira/browse/MAHOUT-988
>>>>              Project: Mahout
>>>>           Issue Type: Sub-task
>>>>           Components: Clustering
>>>>     Affects Versions: 0.6
>>>>             Reporter: Jeff Eastman
>>>>             Assignee: Jeff Eastman
>>>>              Fix For: 0.7
>>>>
>>>>
>>>> Refactor the current K-means implementation to use the
>>>> ClusterIterator/Classifier implementation. This will replace the
>>>> mapper, combiner, reducer, clusterer and many unit tests but will
>>>> not modify the other driver APIs, thus retaining compatibility with
>>>> existing CLI.
>>> --
>>> This message is automatically generated by JIRA.
>>> If you think it was sent incorrectly, please contact your JIRA
>>> administrators:
>>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>>>
>>> For more information on JIRA, see:
>>> http://www.atlassian.com/software/jira
>>>
>>>
>>>
>>>
>>
>
>

I saw the code and my understanding of the new implementation is:
a) K-Means, Fuzzy K-Means and Dirichlet will ClusterIterator and write
IntWritable, ClusterWritbale in buildClusters phase ( Instead of
Kluster, SoftCluster and DirichletCluster )
b) Canopy and MeanShift will NOT use ClusterIterator but will emit
IntWritable, ClusterWritable ( Instead of Canopy and MeanShiftCanopy )

There are tools ( ClusterDumper and ClusterEvaluator ) which expect
<Cluster> when they read from the output file after clustering ( ~
buildCluster phase ).

KMeans is expecting Canopy and KCluster, but will get ClusterWritable.

So, everything needs to be in sync ( i.e. ClusterWritable )

I propose to wrap everything in ClusterWritable first, as everything is
a Cluster ( eg. DirichletCluster, SoftCluster, Kluster, Canopy and
MeanShiftCanopy ). This will remove the inconsistency without much
chaos. Once ClusterWritable is uniformly used, then refactor all algorithms.

I am also not against making ClusterDumper unavailable for a week or so
since we have ClusterOutputPostProcessor now.

Is my understanding correct? If not, please help me understand it.
If yes, which way do you propose to refactor?

On 15-03-2012 19:24, Jeff Eastman wrote:
> Yes, that was my point. below It may, in fact, be impossible to
> implement and commit them independently since so much of Mahout
> clustering depends upon the Cluster sequenceFile. You may be able to
> get part way by moving the Canopy mods into the kmeans issue, but then
> the cluster dumper and evaluator will not work with kmeans.
>
> Ideas?
>
> On 3/14/12 10:15 PM, Paritosh Ranjan wrote:
>> Thanks Jeff. One question, are "Use ClusterIterator" tasks dependent
>> on "Modify Canopy etc to use ClusterWritable" task ?
>> I am assuming that all subtasks in MAHOUT-933
>> <https://issues.apache.org/jira/browse/MAHOUT-933> are independent of
>> each other and the order to pick them does not matter. Am I correct?
>>
>> On 15-03-2012 09:23, Jeff Eastman wrote:
>>> Sure Paritosh, go ahead and take a crack at it. I am moving from CO
>>> to PA for the next few weeks and won't be able to do much coding
>>> during that period. I suspect you will also need to modify Canopy to
>>> emit ClusterWritable and also the RandomSeedGenerator.
>>>
>>> Smooth sailing,
>>> Jeff
>>>
>>> On 3/14/12 8:28 PM, Paritosh Ranjan (Commented) (JIRA) wrote:
>>>>      [
>>>> https://issues.apache.org/jira/browse/MAHOUT-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229840#comment-13229840
>>>> ]
>>>>
>>>> Paritosh Ranjan commented on MAHOUT-988:
>>>> ----------------------------------------
>>>>
>>>> Jeff, I would like to work on this issue (or MAHOUT-989, or
>>>> MAHOUT-990). Can I? I might also need some help ( at least the
>>>> first patch review ).
>>>>
>>>>
>>>>> Convert K-means buildClusters to use new ClusterIterator
>>>>> --------------------------------------------------------
>>>>>
>>>>>                  Key: MAHOUT-988
>>>>>                  URL:
>>>>> https://issues.apache.org/jira/browse/MAHOUT-988
>>>>>              Project: Mahout
>>>>>           Issue Type: Sub-task
>>>>>           Components: Clustering
>>>>>     Affects Versions: 0.6
>>>>>             Reporter: Jeff Eastman
>>>>>             Assignee: Jeff Eastman
>>>>>              Fix For: 0.7
>>>>>
>>>>>
>>>>> Refactor the current K-means implementation to use the
>>>>> ClusterIterator/Classifier implementation. This will replace the
>>>>> mapper, combiner, reducer, clusterer and many unit tests but will
>>>>> not modify the other driver APIs, thus retaining compatibility
>>>>> with existing CLI.
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> If you think it was sent incorrectly, please contact your JIRA
>>>> administrators:
>>>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
>>>>
>>>> For more information on JIRA, see:
>>>> http://www.atlassian.com/software/jira

+1 Paritosh, this is exactly what I envisioned. And I also like your
idea of first converting them all to use ClusterWritable. Go for it!

On 3/15/12 10:42 AM, Paritosh Ranjan wrote:
> I saw the code and my understanding of the new implementation is:
> a) K-Means, Fuzzy K-Means and Dirichlet will ClusterIterator and write
> IntWritable, ClusterWritbale in buildClusters phase ( Instead of
> Kluster, SoftCluster and DirichletCluster )
> b) Canopy and MeanShift will NOT use ClusterIterator but will emit
> IntWritable, ClusterWritable ( Instead of Canopy and MeanShiftCanopy )
>
> There are tools ( ClusterDumper and ClusterEvaluator ) which expect
> <Cluster> when they read from the output file after clustering ( ~
> buildCluster phase ).
>
> KMeans is expecting Canopy and KCluster, but will get ClusterWritable.
>
> So, everything needs to be in sync ( i.e. ClusterWritable )
>
> I propose to wrap everything in ClusterWritable first, as everything
> is a Cluster ( eg. DirichletCluster, SoftCluster, Kluster, Canopy and
> MeanShiftCanopy ). This will remove the inconsistency without much
> chaos. Once ClusterWritable is uniformly used, then refactor all
> algorithms.
>
> I am also not against making ClusterDumper unavailable for a week or
> so since we have ClusterOutputPostProcessor now.
>
> Is my understanding correct? If not, please help me understand it.
> If yes, which way do you propose to refactor?
>
> On 15-03-2012 19:24, Jeff Eastman wrote:
>> Yes, that was my point. below It may, in fact, be impossible to
>> implement and commit them independently since so much of Mahout
>> clustering depends upon the Cluster sequenceFile. You may be able to
>> get part way by moving the Canopy mods into the kmeans issue, but
>> then the cluster dumper and evaluator will not work with kmeans.
>>
>> Ideas?
>>
>> On 3/14/12 10:15 PM, Paritosh Ranjan wrote:
>>> Thanks Jeff. One question, are "Use ClusterIterator" tasks dependent
>>> on "Modify Canopy etc to use ClusterWritable" task ?
>>> I am assuming that all subtasks in MAHOUT-933
>>> <https://issues.apache.org/jira/browse/MAHOUT-933> are independent
>>> of each other and the order to pick them does not matter. Am I correct?
>>>
>>> On 15-03-2012 09:23, Jeff Eastman wrote:
>>>> Sure Paritosh, go ahead and take a crack at it. I am moving from CO
>>>> to PA for the next few weeks and won't be able to do much coding
>>>> during that period. I suspect you will also need to modify Canopy
>>>> to emit ClusterWritable and also the RandomSeedGenerator.
>>>>
>>>> Smooth sailing,
>>>> Jeff
>>>>
>>>> On 3/14/12 8:28 PM, Paritosh Ranjan (Commented) (JIRA) wrote:
>>>>>      [
>>>>> https://issues.apache.org/jira/browse/MAHOUT-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229840#comment-13229840
>>>>> ]
>>>>>
>>>>> Paritosh Ranjan commented on MAHOUT-988:
>>>>> ----------------------------------------
>>>>>
>>>>> Jeff, I would like to work on this issue (or MAHOUT-989, or
>>>>> MAHOUT-990). Can I? I might also need some help ( at least the
>>>>> first patch review ).
>>>>>
>>>>>
>>>>>> Convert K-means buildClusters to use new ClusterIterator
>>>>>> --------------------------------------------------------
>>>>>>
>>>>>>                  Key: MAHOUT-988
>>>>>>                  URL:
>>>>>> https://issues.apache.org/jira/browse/MAHOUT-988
>>>>>>              Project: Mahout
>>>>>>           Issue Type: Sub-task
>>>>>>           Components: Clustering
>>>>>>     Affects Versions: 0.6
>>>>>>             Reporter: Jeff Eastman
>>>>>>             Assignee: Jeff Eastman
>>>>>>              Fix For: 0.7
>>>>>>
>>>>>>
>>>>>> Refactor the current K-means implementation to use the
>>>>>> ClusterIterator/Classifier implementation. This will replace the
>>>>>> mapper, combiner, reducer, clusterer and many unit tests but will
>>>>>> not modify the other driver APIs, thus retaining compatibility
>>>>>> with existing CLI.
>>>>> --
>>>>> This message is automatically generated by JIRA.