All,

I have begun work on an integration of Apache Solr and Mahout,
http://github.com/algoriffic/lsa4solr   which is related to #MAHOUT-343
(https://issues.apache.org/jira/browse/MAHOUT-343 ).  The
implementation is in Clojure and interfaces with both the
DistributedLanczosSolver and the distributed k-means clustering
algorithm from Mahout.  I am about to begin implementing a
hierarchical clustering algorithm so that the number of clusters does
not need to be specified in advance.  Has anyone done anything like
this in Mahout yet?  Also, I'd be happy to contribute the code to
Mahout if anyone is interested.

Thanks,
Anthony

On Fri, Apr 16, 2010 at 9:50 AM, Jake Mannix <[EMAIL PROTECTED]> wrote:
> Hey Anthony,
>   We would love to have hierarchical clustering in Mahout, in Clojure or
> pure java.  Come on over to the mahout-dev mailing list, and/or file
> a JIRA ticket, and join the fun, we'd love to work with you (and over
> on mahout-dev, you'll get even more positive feedback).
>   If you'd rather, and aren't as familiar with the whole Apache process,
> I can file a JIRA ticket for you, and you can just comment there and
> start the conversation that way.
>   Do you subscribe to the [EMAIL PROTECTED] / mahout-user@
> mailing lists?  Their not too high traffic.
>   -jake

Great to see you on here, Anthony!

Hey the-rest-of-you!  You should definitely check out his
lsa4solr code, it's mostly very nice thin Clojure wrappers
around the DistributedRowMatrix and related code, which
allows for using Clojure's REPL to play interactively with
the matrix (which in itself is awesome, beyond integration
with Lucene or Solr).

We should make a new maven submodule for Clojure
contributions, and can make some of Anthony's code the
base of it.

Are any of us Clojure users, who could help Anthony get
his git repo (it's already Apache Licensed) integrated into
our build framework and make a patch so we can start
working with it?

  -jake

(sorry for giving you the wrong list address at first!)
On Fri, Apr 16, 2010 at 7:22 AM, Anthony <[EMAIL PROTECTED]> wrote:

> All,
>
> I have begun work on an integration of Apache Solr and Mahout,
> http://github.com/algoriffic/lsa4solr   which is related to #MAHOUT-343
> (https://issues.apache.org/jira/browse/MAHOUT-343 ).  The
> implementation is in Clojure and interfaces with both the
> DistributedLanczosSolver and the distributed k-means clustering
> algorithm from Mahout.  I am about to begin implementing a
> hierarchical clustering algorithm so that the number of clusters does
> not need to be specified in advance.  Has anyone done anything like
> this in Mahout yet?  Also, I'd be happy to contribute the code to
> Mahout if anyone is interested.
>
> Thanks,
> Anthony
>
> On Fri, Apr 16, 2010 at 9:50 AM, Jake Mannix <[EMAIL PROTECTED]> wrote:
> > Hey Anthony,
> >   We would love to have hierarchical clustering in Mahout, in Clojure or
> > pure java.  Come on over to the mahout-dev mailing list, and/or file
> > a JIRA ticket, and join the fun, we'd love to work with you (and over
> > on mahout-dev, you'll get even more positive feedback).
> >   If you'd rather, and aren't as familiar with the whole Apache process,
> > I can file a JIRA ticket for you, and you can just comment there and
> > start the conversation that way.
> >   Do you subscribe to the [EMAIL PROTECTED] / mahout-user@
> > mailing lists?  Their not too high traffic.
> >   -jake
>

Clojure isn't my cup of tea but that's not important.

It's an interesting question, how much belongs under the Mahout tent?
There's a tradeoff between excluding useful extensions to the project
on the one hand, and becoming a spare parts bin of code of varying
levels of maturity and support.

I'm inclined to see Mahout built, like any project, in layers. We've
done a good job of that, with collections supporting math, supporting
core, supporting examples/utils. core remains uneven, but, it's taking
shape and it's to be expected at this stage that the project has lots
of fragments that naturally come together or else are weeded out.

So I have some concern with getting even core into better shape before
considering broadening the tent to include other consumers. (We're not
talking about integrating Clojure code into core right? Yeah it ends
up as Java bytecode, but that's not quite the issue... it's not
understandable and usable to the devs that would be using Java-based
Mahout.)

And so I have moderate preference for not mixing in other languages
yet, just like with the talk of a C# port earlier today.

I'm sure it's good code and cool and useful, and deserves support and
collaboration and liaising; I'm only wondering about whether it ought
be a piece of Mahout - what is the benefit, against the subtle but
real drawbacks.

On 4/16/10 10:05 AM, Sean Owen wrote:
> Clojure isn't my cup of tea but that's not important.
>
> It's an interesting question, how much belongs under the Mahout tent?
> There's a tradeoff between excluding useful extensions to the project
> on the one hand, and becoming a spare parts bin of code of varying
> levels of maturity and support.
>
> I'm inclined to see Mahout built, like any project, in layers. We've
> done a good job of that, with collections supporting math, supporting
> core, supporting examples/utils. core remains uneven, but, it's taking
> shape and it's to be expected at this stage that the project has lots
> of fragments that naturally come together or else are weeded out.
>
> So I have some concern with getting even core into better shape before
> considering broadening the tent to include other consumers. (We're not
> talking about integrating Clojure code into core right? Yeah it ends
> up as Java bytecode, but that's not quite the issue... it's not
> understandable and usable to the devs that would be using Java-based
> Mahout.)
>
> And so I have moderate preference for not mixing in other languages
> yet, just like with the talk of a C# port earlier today.
>
> I'm sure it's good code and cool and useful, and deserves support and
> collaboration and liaising; I'm only wondering about whether it ought
> be a piece of Mahout - what is the benefit, against the subtle but
> real drawbacks.
>
>    
+1 I'd like to see us get to at least a 1.0 level of maturity before we
branch out into other languages. We still have a lot of refactoring and
reorganization to do and adding other languages would complicate matters
and probably end up being whip-sawed by the volatility in the Java
implementations.

So here's my take: once we're a TLP (next month sometime?), it is
a good time to start allowing subprojects or submodules which are
"scripting" layers on top of Mahout - whether they are PigLatin, or
Cascalog, JRuby, or Clojure.  If it's JVM-based, especially, having
code/scripts which are "drivers" and wrappers for what is currently
for the most part a library which has a shell script (Taste-web is
the exception) is a huge useful addition.

To compare, the guys doing Incanter ( http://data-sorcery.org/ )
are using Clojure as a wrapper around Parallel-Colt
( http://sites.google.com/site/piotrwendykier/software/parallelcolt ),
and does all of the heavy lifting in pure Java.  That's why I'm not
too worried about performance stuff (to respond to Robin's concern
down-thread while I'm writing this).

Personally, I'm not a Clojure guy, but Lisp has long been a mainstay
of the AI world, and if we're going to interface with academic ML and
AI more (which we should!!!), Clojure is going to be our best bet,
as it's just Lisp on the JVM, and while were I going to write a REPL
for us, I'd do it in JRuby (much like HBase uses JRuby's jirb for
their shell), and may still, but more easy interactive ways of
using Mahout, the better, I'd say.

Hmm... this was a bit scattered of a response, but I'm really loathe
to turn away a) nice hooks between Solr and Mahout, b) scripting-style
wrappers which could expand our community, and c) simply new
functionality.

I'm certainly game to help shepherd in any code we can use, although
I guess I'm fine waiting to help make a sub-project once we're a TLP
if that's the right way to go.

  -jake

On Fri, Apr 16, 2010 at 10:05 AM, Sean Owen <[EMAIL PROTECTED]> wrote:

> Clojure isn't my cup of tea but that's not important.
>
> It's an interesting question, how much belongs under the Mahout tent?
> There's a tradeoff between excluding useful extensions to the project
> on the one hand, and becoming a spare parts bin of code of varying
> levels of maturity and support.
> I'm inclined to see Mahout built, like any project, in layers. We've
> done a good job of that, with collections supporting math, supporting
> core, supporting examples/utils. core remains uneven, but, it's taking
> shape and it's to be expected at this stage that the project has lots
> of fragments that naturally come together or else are weeded out.
>
> So I have some concern with getting even core into better shape before
> considering broadening the tent to include other consumers. (We're not
> talking about integrating Clojure code into core right? Yeah it ends
> up as Java bytecode, but that's not quite the issue... it's not
> understandable and usable to the devs that would be using Java-based
> Mahout.)
>
> And so I have moderate preference for not mixing in other languages
> yet, just like with the talk of a C# port earlier today.
>
> I'm sure it's good code and cool and useful, and deserves support and
> collaboration and liaising; I'm only wondering about whether it ought
> be a piece of Mahout - what is the benefit, against the subtle but
> real drawbacks.
>

On Apr 16, 2010, at 2:21 PM, Jake Mannix wrote:

> So here's my take: once we're a TLP (next month sometime?), it is
> a good time to start allowing subprojects or submodules which are

Submodules, yes, subprojects, not so much, unless the committers are the same.  We can definitely release different artifacts, etc. but the Board has made it pretty clear that umbrella projects are not good.

> "scripting" layers on top of Mahout - whether they are PigLatin, or
> Cascalog, JRuby, or Clojure.  If it's JVM-based, especially, having
> code/scripts which are "drivers" and wrappers for what is currently
> for the most part a library which has a shell script (Taste-web is
> the exception) is a huge useful addition.
>
> To compare, the guys doing Incanter ( http://data-sorcery.org/ )
> are using Clojure as a wrapper around Parallel-Colt
> ( http://sites.google.com/site/piotrwendykier/software/parallelcolt ),
> and does all of the heavy lifting in pure Java.  That's why I'm not
> too worried about performance stuff (to respond to Robin's concern
> down-thread while I'm writing this).
>
> Personally, I'm not a Clojure guy, but Lisp has long been a mainstay
> of the AI world, and if we're going to interface with academic ML and
> AI more (which we should!!!), Clojure is going to be our best bet,
> as it's just Lisp on the JVM, and while were I going to write a REPL
> for us, I'd do it in JRuby (much like HBase uses JRuby's jirb for
> their shell), and may still, but more easy interactive ways of
> using Mahout, the better, I'd say.
>
> Hmm... this was a bit scattered of a response, but I'm really loathe
> to turn away a) nice hooks between Solr and Mahout, b) scripting-style
> wrappers which could expand our community, and c) simply new
> functionality.

+1.  I hope to add in some more Solr hooks in the near future, too, including an implementation of the ClusteringEngine and other things like DocumentProcessor chains, etc. for classifiers (help wanted!)
>
> I'm certainly game to help shepherd in any code we can use, although
> I guess I'm fine waiting to help make a sub-project once we're a TLP
> if that's the right way to go.

Let's see a patch first.  As with all of this stuff, if there are people who will work on it and maintain it and it's related to Mahout, then I think we should take it in.

-Grant

On Fri, Apr 16, 2010 at 11:26 AM, Grant Ingersoll <[EMAIL PROTECTED]>wrote:

>
> On Apr 16, 2010, at 2:21 PM, Jake Mannix wrote:
>
> > So here's my take: once we're a TLP (next month sometime?), it is
> > a good time to start allowing subprojects or submodules which are
>
> Submodules, yes, subprojects, not so much, unless the committers are the
> same.  We can definitely release different artifacts, etc. but the Board has
> made it pretty clear that umbrella projects are not good.
>

I wasn't suggesting different committers.  Is the Board against having
subprojects if the committer list is unified?
> +1.  I hope to add in some more Solr hooks in the near future, too,
> including an implementation of the ClusteringEngine and other things like
> DocumentProcessor chains, etc. for classifiers (help wanted!)
>

Definitely.  I'd like to write some simple stuff which can just live in
pure Lucene-land too, for people who have their own service on
top of it (so some simple Query and Filter classes, and maybe
some specialized IndexWriter which makes some synthetic
fields to index which have the LSA projection of other fields,
and also maybe tags the document with cluster labels, etc).
> > I'm certainly game to help shepherd in any code we can use, although
> > I guess I'm fine waiting to help make a sub-project once we're a TLP
> > if that's the right way to go.
>
> Let's see a patch first.  As with all of this stuff, if there are people
> who will work on it and maintain it and it's related to Mahout, then I think
> we should take it in.
>
>
I will start playing around with Anthony's github-based stuff, and
see where a patch can be made.  The question is where it would
go?  It's a fully functioning project already over on its own.

  -jake
> -Grant
>
>
>

On Fri, Apr 16, 2010 at 7:39 PM, Jake Mannix <[EMAIL PROTECTED]> wrote:
> I will start playing around with Anthony's github-based stuff, and
> see where a patch can be made.  The question is where it would
> go?  It's a fully functioning project already over on its own.
I suppose that's my question too -- what is being fixed by a move?

The point about integrating with the ML community by having a
'LISP-speaking' module, to be friendlier, is a good one. It does call
into question the Mahout identity -- is it for tinkering with in a lab
to explore new algorithms (for which Clojure/LISP makes sense)? or is
it for engineers and production systems at scale -- where Hadoop/Java
is the lingua franca? Yeah, this is not just another language, but for
a somewhat different audience.

Maybe "both" is nice. Before version 1.0 I think it can be harmful to
let the project remit range too broadly. We all know how open-source
goes. It's for-fun, spare-time. It's easy to start things and hard to
finish them. I'm just getting concerned we end up with 10
half-finished modules rather than 5 finished ones. I don't have reason
to believe this module would be orphaned; this is tilting at windmils.
It's just a general concern raised by early expansion.

After the foundation we have now is solid -- naturally, careful
expansion is a next step. Do I hear consensus to think about this
post-1.0, post TLP, post book? and continue working together to see
where the projects go? (There's some value to staying separate --
forces you to not integrate the code in cheap and tangled ways -- have
to proceed through public APIs.)

Or is there a significant synergy from tight integration, which
warrants combining projects right now?

I don't want to make too much hay over this one question as much as
bring up the larger issue. I wouldn't scream if Clojure landed in the
repo.

I have a friend right now who needs a jruby integration layer.  It is about
how code is glued together.  Java is lingua franca, but there are lots of
sites that are using Groovy (in the finance world especially), Ruby (in the
web 2.0 world) or cloJure (in the too-cool crowd).

So I definitely see a place for small compatibility layers to make Mahout
easily accessible from these languages.  Having a repl is a very common
pattern for tuning learning algorithms.  If that repl is the same as your
glue layer, so much the better.  Right now, our only repl option is shell
command lines for a few top level functions.  It would be sooo much better
if we had tighter integration in a real scripting language.

Aside from my support for nice scripting layers, I think it might be better
to take Anthony's clojure support, but have the LSA/Solr/Mahout integration
be a consumer of Mahout rather than a part of Mahout.  Obviously since he
would be an early "customer", it would behoove us to work well with him to
help him succeed, but I don't see a big advantage to integrating his main
code until HE starts seeing some adoption and we see some independent pull.
Mahout can't be such a big tent that we include every project or piece of
code that uses anything from Mahout.  At some point, it is more useful to
draw a line.

On Fri, Apr 16, 2010 at 11:56 AM, Sean Owen <[EMAIL PROTECTED]> wrote:

> On Fri, Apr 16, 2010 at 7:39 PM, Jake Mannix <[EMAIL PROTECTED]>
> wrote:
> > I will start playing around with Anthony's github-based stuff, and
> > see where a patch can be made.  The question is where it would
> > go?  It's a fully functioning project already over on its own.
>
>
> I suppose that's my question too -- what is being fixed by a move?
>
> The point about integrating with the ML community by having a
> 'LISP-speaking' module, to be friendlier, is a good one. It does call
> into question the Mahout identity -- is it for tinkering with in a lab
> to explore new algorithms (for which Clojure/LISP makes sense)? or is
> it for engineers and production systems at scale -- where Hadoop/Java
> is the lingua franca? Yeah, this is not just another language, but for
> a somewhat different audience.

On Fri, Apr 16, 2010 at 11:56 AM, Sean Owen <[EMAIL PROTECTED]> wrote:

> On Fri, Apr 16, 2010 at 7:39 PM, Jake Mannix <[EMAIL PROTECTED]>
> wrote:
> > I will start playing around with Anthony's github-based stuff, and
> > see where a patch can be made.  The question is where it would
> > go?  It's a fully functioning project already over on its own.
>
>
> I suppose that's my question too -- what is being fixed by a move?
>

What is being fixed is that we currently have an open ticket for providing
LSA hooks for Solr, and lsa4solr provides an end-to-end solution for that
particular task, along with a bunch of other nice things (clojure wrapping
and thus the REPL).  If we want to say "MAHOUT-343 is a Don't Fix",
and that's the consensus, then that's fine.  It could also be implemented
in pure java inside of mahout-core, but I don't see anyone stepping up to
the plate to write that, and here's someone who's done it in another
JVM language we could use.

> The point about integrating with the ML community by having a
> 'LISP-speaking' module, to be friendlier, is a good one. It does call
> into question the Mahout identity -- is it for tinkering with in a lab
> to explore new algorithms (for which Clojure/LISP makes sense)? or is
> it for engineers and production systems at scale -- where Hadoop/Java
> is the lingua franca? Yeah, this is not just another language, but for
> a somewhat different audience.
>
> Maybe "both" is nice.
I think "both" is *necessary*.  At least at all the smallish bay-area
startups doing some scalable production ML these days, there is not
much distinction between "researcher" and "production code-monkey" -
you prototype in whatever language you can, you try it out (maybe using
hadoop streaming) on bigger data sets, then you productionalize it (
usually the same engineers involved in all steps).

If Mahout could be helping along with that entire process, that would
be fantastic.  We'd have shell scripts and an actual REPL to tinker
with, and then when it came time to optimize performance, the same
exact libraries could be used and extended, no more "first we do
stuff with Matlab or R, then port all of the code over to java/c++ later".
  -jake

On Apr 16, 2010, at 3:11 PM, Jake Mannix wrote:

> On Fri, Apr 16, 2010 at 11:56 AM, Sean Owen <[EMAIL PROTECTED]> wrote:
>
>> On Fri, Apr 16, 2010 at 7:39 PM, Jake Mannix <[EMAIL PROTECTED]>
>> wrote:
>>> I will start playing around with Anthony's github-based stuff, and
>>> see where a patch can be made.  The question is where it would
>>> go?  It's a fully functioning project already over on its own.
>>
>>
>> I suppose that's my question too -- what is being fixed by a move?
>>
>
> What is being fixed is that we currently have an open ticket for providing
> LSA hooks for Solr, and lsa4solr provides an end-to-end solution for that
> particular task, along with a bunch of other nice things (clojure wrapping
> and thus the REPL).  If we want to say "MAHOUT-343 is a Don't Fix",

Well, M-343 isn't necessarily an LSA spot.  It was a placeholder for a GSOC student to come in and do some integration.  But, still I see this as a really good start.  For the record, I'd love to see a Taste-Lucene/Solr marriage too.  :-)

Lots of opportunities here!  Just need more time.

-Grant

At that point, they aren't really sub-projects, just multiple deliverable
binaries.  We have that already (collections, math, core) and the
unification of Lucene and Solr was based on that as well.

So it would be one project (mailing list, committer list) but multiple
modules.

On Fri, Apr 16, 2010 at 11:39 AM, Jake Mannix <[EMAIL PROTECTED]> wrote:

> > Submodules, yes, subprojects, not so much, unless the committers are the
> > same.  We can definitely release different artifacts, etc. but the Board
> has
> > made it pretty clear that umbrella projects are not good.
> >
>
> I wasn't suggesting different committers.  Is the Board against having
> subprojects if the committer list is unified?

>
>
>
> Hmm... this was a bit scattered of a response, but I'm really loathe
> to turn away a) nice hooks between Solr and Mahout, b) scripting-style
> wrappers which could expand our community, and c) simply new
> functionality.
>
> +1
I definitely don't want to turn it away, I want to ensure that duplication
of stuff don't happen and the heavy lifting is indeed done in Java and it
should be easy enough for anyone to consume in any existing app in any
language, Even something as crazy as COBOL apps calling Mahout classifiers
http://www.emunix.emich.edu/info/cobol/books/dpjafc.htm

LSA implementation should actually be in Mahout core. I would love it to be

Robin

On Fri, Apr 16, 2010 at 11:31 AM, Robin Anil <[EMAIL PROTECTED]> wrote:

> >
> >
> >
> > Hmm... this was a bit scattered of a response, but I'm really loathe
> > to turn away a) nice hooks between Solr and Mahout, b) scripting-style
> > wrappers which could expand our community, and c) simply new
> > functionality.
> >
> > +1
> I definitely don't want to turn it away, I want to ensure that duplication
> of stuff don't happen and the heavy lifting is indeed done in Java and it
> should be easy enough for anyone to consume in any existing app in any
> language, Even something as crazy as COBOL apps calling Mahout classifiers
> http://www.emunix.emich.edu/info/cobol/books/dpjafc.htm
>
> LSA implementation should actually be in Mahout core. I would love it to be
>

LSA in pure java is on my roadmap, it's not much work to do on top of what
we already have, actually.  The question is what you do with it - putting it
inside
of Lucene is one way, and making a simple standalone latent semantic index
is another, but that's definitely a core feature, regardless of what wrapper
we
have.

  -jake

On Fri, Apr 16, 2010 at 7:52 PM, Anthony <[EMAIL PROTECTED]> wrote:

> All,
>
> I have begun work on an integration of Apache Solr and Mahout,
> http://github.com/algoriffic/lsa4solr   which is related to #MAHOUT-343
> (https://issues.apache.org/jira/browse/MAHOUT-343 ).  The
> implementation is in Clojure and interfaces with both the
> DistributedLanczosSolver and the distributed k-means clustering
> algorithm from Mahout.

Java would have been nicer (Though I am saying that without knowing how well
Clojure binaries can talk with Java ones and vice versa)

>  I am about to begin implementing a
> hierarchical clustering algorithm so that the number of clusters does
> not need to be specified in advance.  Has anyone done anything like
> this in Mahout yet?  Also, I'd be happy to contribute the code to
> Mahout if anyone is interested.
>
Hierarchical clustering is missing from Mahout at the moment, It will be
great if you can help bring it to Mahout.

>
> Thanks,
> Anthony
>
> On Fri, Apr 16, 2010 at 9:50 AM, Jake Mannix <[EMAIL PROTECTED]> wrote:
> > Hey Anthony,
> >   We would love to have hierarchical clustering in Mahout, in Clojure or
> > pure java.  Come on over to the mahout-dev mailing list, and/or file
> > a JIRA ticket, and join the fun, we'd love to work with you (and over
> > on mahout-dev, you'll get even more positive feedback).
> >   If you'd rather, and aren't as familiar with the whole Apache process,
> > I can file a JIRA ticket for you, and you can just comment there and
> > start the conversation that way.
> >   Do you subscribe to the [EMAIL PROTECTED] / mahout-user@
> > mailing lists?  Their not too high traffic.
> >   -jake
>

On Fri, Apr 16, 2010 at 7:56 AM, Robin Anil <[EMAIL PROTECTED]> wrote:

> On Fri, Apr 16, 2010 at 7:52 PM, Anthony <[EMAIL PROTECTED]> wrote:
>
> > All,
> >
> > I have begun work on an integration of Apache Solr and Mahout,
> > http://github.com/algoriffic/lsa4solr   which is related to #MAHOUT-343
> > (https://issues.apache.org/jira/browse/MAHOUT-343 ).  The
> > implementation is in Clojure and interfaces with both the
> > DistributedLanczosSolver and the distributed k-means clustering
> > algorithm from Mahout.
>
> Java would have been nicer (Though I am saying that without knowing how
> well
> Clojure binaries can talk with Java ones and vice versa)
>

Clojure is on the JVM, Robin!  http://clojure.org/ - the "j" in the name
should
have given the hint! :)

  -jake

I>

> > Java would have been nicer (Though I am saying that without knowing how
> > well
> > Clojure binaries can talk with Java ones and vice versa)
> >
>
> Clojure is on the JVM, Robin!  http://clojure.org/ - the "j" in the name
> should
> have given the hint! :)
>
> :) Yeah I saw that, I had this really bad ordeal when an F# prototype code
was later re-coded using C# as it hit the performance wall. Just don't want
the same fate with Clojure, especially when its a library that deals with
algorithms that need high performance

Robin