Hi All,

It is fairly apparent to me that the Lucy project is not making any  
progress community-wise or code-wise.  Neither Marvin, Dave or Doug  
are active at all on it, and that accounts for all three committers.  
There has been very little mailing list traffic, and when there is, it  
seems to be of the nature of "is Lucy alive?"   Furthermore, I have my  
doubts about the development process being employed, which seems to be  
the notion that KinoSearch is going to be donated by Marvin at some  
point in the future [1], which would only work if it were to go  
through the Software Grant or Incubation process (which I would be  
happy to support.), or at least that is how I understand the process  
to be when code is developed outside of the ASF.  Even if KS were the  
plan, in looking at KS, it seems there is not much community activity  
there, either.

On the flip side, one might ask what's the harm in letting it stand as  
is?  Admittedly, not much, other than I think it confuses people b/c  
they think there is a C port of Lucene and then they go and find it is  
dead.

Therefore, it is with some hesitation that I suggest we mothball  
Lucy.  Mostly, I hesitate, because I hate to see any project be  
archived on the hope that someone will come in and pick it up.  
However, I just don't see that happening.  If Marvin wishes to  
resurrect it, he can donate KS (or whatever core part of it is Lucy)  
and go through incubation and prove there is a community and then we  
can turn it back on.

-Grant

[1] http://www.lucidimagination.com/search/document/152a1a9d00b7d08a/is_there_anybody_here

Grant,

I am currently employed by Eventful, Inc, in San Diego, CA.  They are paying
me to work full-time on KinoSearch and Lucy.

I went out of my way when we negotiated the terms of my employment to ensure
that there was no way my contract could hamper or compromise progress towards
Lucy.  The actual document is confidential of course, but I feel comfortable
saying that first, our lawyers hammered out the legal nuts and bolts to my
satisfaction, and second, Eventful is fully on board with regards to Lucy.  By
way of illustration, my boss regularly hassles me about publishing a Lucy C
API, even though since Eventful uses the Perl bindings the benefits would be
indirect.

In my opinion, it is not in the best interests of the Apache Lucene project to
make it more difficult for my employer and myself to contribute.

> It is fairly apparent to me that the Lucy project is not making any
> progress community-wise or code-wise.  Neither Marvin, Dave or Doug
> are active at all on it, and that accounts for all three committers.
> There has been very little mailing list traffic,

You may have noticed that up until about three weeks ago (when I dove back
into the code cave), I was quite active on [EMAIL PROTECTED] and in
the Lucene JIRA forums.  Significant design innovations were realized,
particularly in the area of real-time search.

In the past, many designs have been hashed out cooperatively on the KinoSearch
and Lucy mailing lists: the Schema class, revisions to QueryParser and the
boolean Query hierarchy, the implementation of human-readable index metadata,
C configuration probing, the OO model, index designs which exploit memory
mapping, and so on.

In this particular case, however, I was assigned the task of solving real-time
search, for which the Lucy and KinoSearch forums were not ideal.  There is a
very limited number of people who have both the familiarity with the
Lucene/Lucy segment-based inverted index model and the interest to discuss
real-time search at the level I desired, where concepts like "segment-centric
search" could be bandied about.  Basically, I needed Mike McCandless -- so I
went to where he could be found.

The conversations that we had in JIRA and on java-dev were beneficial to both
Lucene and Lucy; should I have posted to the Lucy dev list instead simply to
demonstrate activity, which would have been less useful to Mike, to me, to
Lucy, and to Lucene?  To my mind, the Lucene community is also part of the
Lucy community.  Mike's insights were welcome and useful, and it didn't seem
important to me which specific mailing list they wound up on -- they're all
under the domain lucene.apache.org, after all.  Weren't we all moving forward
together, and wouldn't that be apparent to members of the PMC such as
yourself?

Or is this a zero-sum game where design innovations which help Lucy don't
count as "progress" if they also help Lucene?

> Furthermore, I have my doubts about the development process being employed,
> which seems to be the notion that KinoSearch is going to be donated by
> Marvin at some point in the future [1], which would only work if it were to
> go through the Software Grant or Incubation process (which I would be happy
> to support.), or at least that is how I understand the process to be when
> code is developed outside of the ASF.  

I understand why you might have thought that, but that's not how things will
play out, and it's a misreading of the post that you cite.
(<http://www.lucidimagination.com/search/document/152a1a9d00b7d08a/is_there_anybody_here>)

As you note, simply importing KinoSearch wholesale into the Lucy repository
with cosmetic changes would violate the terms of the project.  But even if
that were possible, it would represent a *horrendous missed opportunity*.

A KinoSearch 1.0 release, with permanent API and file format backwards
compatibility guarantees -- i.e. "there will never be a KinoSearch 2.0" --
will be very beneficial for Lucy's development.  Imposing such discipline
allows library users to proceed with maximum confidence.  For instance, it
allows Peter Karman, who has long planned to build a KS backend for Swish, to
move forward without having to worry about the upstream library pulling the
rug out from underneath his users.

Going that route will maximize our ability to learn the limitations and
weaknesses of the design.  Using the knowledge we gain, we can then forge
ahead as we have in the past: chunk by chuck, class by class.  And even though
I am very pleased with how pluggable index components, C API user interface
improvements, "OS-as-JVM" file format changes, and so on are coming along, I
anticipate lots of healthy debate and major discrepancies between what ends up
in KS 1.0 and what ends up in Lucy.
This is largely due to the fact that it has been a long time since I released
any significant public updates.  I choose to release significant updates
infrequently because breaking backwards compatibility has severe consequences
for CPAN modules: as soon as the install completes, live apps start crashing.

Since there is no sane deprecation mechanism for dynamically loaded Perl
modules, minimizing backwards compatibility problems is a responsibility I
take seriously.
Indeed.  It's not like Lucy in its present form causes harm to the bottom line
of Lucid Imagination, Inc. ;)
Please give me two to three months to make the next dev release of KinoSearch.
FWIW, if I can't get a release out within that time frame, I'm going to have
to answer to Eventful. :)

This release will introduce real-time search, improved subclassing support, an
mmap-friendly index file format, and pluggable indexing components.  I suspect
aspects of it may be of interest to the Java Lucene dev community -- but if
that's the case, I won't hold it against you. ;)

Cheers,

Marvin Humphrey

Hi,

On Fri, Mar 6, 2009 at 10:58 PM, Marvin Humphrey <[EMAIL PROTECTED]> wrote:
> Please give me two to three months to make the next dev release of KinoSearch.

What will happen then? KinoSearch is not Lucy, so I'm not sure how
this is relevant. Are you then planning to start working on Lucy, or
to contribute some of the existing KinoSearch code into Lucy? Are
there other people who are planning to start working on Lucy?

Lucy has been around for almost three years already, but I don't see
any search code in the Lucy trunk. When and how is Lucy development
going to start?

You mention that in many cases other forums have been better for
discussing related design issues. What's the benefit of keeping the
Lucy project alive if there's next to no code or even discussion
there?

I'm sure that everyone here would love to see Lucy become more active.
How could we help make that happen?

As a wild idea: would there be interest in bringing the KinoSearch
codebase over to Apache through incubation? It sounds like the two
projects are closely related, so having them both under the same
organization might make many things easier.

BR,

Jukka Zitting

On Sat, Mar 07, 2009 at 03:02:22AM +0100, Jukka Zitting wrote:
> > Please give me two to three months to make the next dev release of KinoSearch.
>
> What will happen then?

The next dev release of KS will present real world implementations of many
designs that have been discussed in Lucene and Lucy forums over the last year.

Some might see that as "progress". ;)

> When and how is Lucy development going to start?

It *is* actively progressing.  It's just that neither you nor Grant are
willing to acknowledge that any of the design work I just did (in happy
collaboration with Java Lucene devs) applies to Lucy.

Please go read <https://issues.apache.org/jira/browse/LUCENE-1458> and see if
you can still assert after you read it that no work is being done on Lucy.  I
warn you, it is a long thread. :)

> You mention that in many cases other forums have been better for
> discussing related design issues. What's the benefit of keeping the
> Lucy project alive if there's next to no code or even discussion
> there?

The proposal remains sound, and there is a deep hunger out there for a solid C
IR library similar to Lucene.  The KS-then-Lucy progression is the fastest and
best way to get there.

Things would have gone more smoothly and quickly if Dave Balmain had been able
to contribute more, but even with that setback, we will still reach the
finish.

> I'm sure that everyone here would love to see Lucy become more active.
> How could we help make that happen?

Help Mike McCandless and Jason Rutherglen finish up their work on the designs
we've all been discussing.  This is a multi-way collaboration, and Lucy
benefits when I'm able to study alternatate implementations, just as Java
Lucene benefits from being able to see what other projects have done.

Cross-pollination has worked very well in the past.  The indexing speedups a
while back started with McCandless riffing on the KinoSearch merge model.  (He
followed that up with plenty of interesting innovating on his own.)

> As a wild idea: would there be interest in bringing the KinoSearch
> codebase over to Apache through incubation?

My main reservation is that I really want to see KS and Lucy play out
sequentially, because I want Lucy to benefit from having seen how the features
now in KS work in the real world.  There's no sane versioning under Perl/CPAN.
You can't move from Lucy version 1 to Lucy version 2 without screwing over
your users, and therefore I don't want to merge the two projects into one
namespace.  If we did that, the unified project has to stay as an "alpha" for
that much longer, and it never really gets the benefit of seeing how a
real-world release goes.

If, then, we're proceeding sequentially as I recommend, I don't see how
putting KS through incubation does anything but slow us down.  All we're doing
is adding extra hoops to jump through.  It might be politically expedient, but
the engineer in me rebels at the waste, as does the loyal employee.

>From my perspective, what we have is an optics problem.  I'm working full
time, and I've been plenty active in the Lucene forums, but you and Grant only
see a big fat zero.  :(

Marvin Humphrey

: It *is* actively progressing.  It's just that neither you nor Grant are
: willing to acknowledge that any of the design work I just did (in happy
: collaboration with Java Lucene devs) applies to Lucy.

I don't really believe that's a fair characterization of the comments made
in this thread.

The fundemental issue is really wether Lucy, as a project, is "alive".  

This is not about wether progress is being made towards a good C Library
for search, or wether there have been good design discussions to further
that goal, or wether there has been good collaboration amongst various
people towrds common goals that can be implimented in multiple projects --
the answers to all of those questions may be "yes" (and i genuinely
believe that they are) but that doesn't mean that Lucy, as a project, is
alive.

: The proposal remains sound, and there is a deep hunger out there for a solid C
: IR library similar to Lucene.  The KS-then-Lucy progression is the fastest and
: best way to get there.

Marvin, I respect your opinion.  If you believe that the best long term
strategy towards making Lucy into a solid project is to first focus on
KinoSearch, then I have faith in your judgement -- but from my
perspective, that seems like a strong argument in favor of archiving Lucy
at this time and reviving it at a future date when you feel the time is
right to bring he apprpriate code from KS into the apache fold via
software grant.

: >From my perspective, what we have is an optics problem.  I'm working full
: time, and I've been plenty active in the Lucene forums, but you and Grant only
: see a big fat zero.  :(

At this point (from my perspective) Lucy as a project is not "alive" ...
that doesn't mean i don't respect your participation in Lucene as a whole,
and I do recognize that you've been making a lot of progress; but one man
isn't a community, and KS isn't Lucy.  I don't see a zero, I see a large
blank area that can be filled later, but for now it confuses people.

It seems to me that (to borrow a cliche) Lucy was an idea ahead of it's
time, so we should be honest to the world (and ourselves) about the state
of things. If people want to work on a project developing a C search
library with bindings for dynamic langauges let's not frustrate them with
a 3 year old website, and ghost town mailing lists and code base --
instead let's encourage and promote groth in the internals of KinoSearch,
so that at a future point we can revive the Lucy project, with a healthy
developer community.

This discussion isn't about "killing" a project -- it's not an execution
-- it's about acknowledge that Lucy hasn't really been born yet.

-Hoss

Hoss:

> The fundemental issue is really wether Lucy, as a project, is "alive".

There is one extremely active participant, and as this thread attests, there
is both a small community and demand for the product.  Futhermore, the work
that is being done to move Lucy forward is benefitting Java Lucene.

In my view, the amount of raw activity dedicated to advancing Lucy clearly
exceeds the threshold that would justify a mothballing.  It's not abandoned.
Things are moving more slowly than they would if Balmain were actively
contributing, but *plenty* of important work is being accomplished.
Nevertheless, I understand why the project has an appearance of dormancy --
please see my reply to Jukka for an explanation.

It is my belief that the most useful thing we can do for Lucy right now is to
release an API-stable library that implements some extensibility enhancements
that further Lucy's high-level mission of empowering the users of its
bindings, and study how those enhancements perform in the real world.  To
reach this goal as quickly as possible, I have performed a torso transplant on
KinoSearch -- arguably to KinoSearch's *detriment* as a CPAN module, since the
massive code churn has introduced bugs (e.g. the Highlighter broke) and
siphoned dev time away from other priorities.  It's true that "KS isn't Lucy",
but my top priority is Lucy, not KS.

Additionally, rather than foster discussion within Lucy forums, I went abroad,
and without my voice as a catalyst, the forums stagnated.  I calculated that
contributing to other lucene.apache.org forums would suffice.  This was
an error.

To remedy the situation, the main thing we need to figure out is how to move
the extensibility prototyping work under the ASF umbrella.  If those commits
had been flowing into the Lucy repository, we wouldn't be having this
conversation.  

Incubating KinoSearch is one possible approach, but the point is to advance
Lucy, not bless KinoSearch.  Is that really the right way to go?

> If you believe that the best long term strategy towards making Lucy into a
> solid project is to first focus on KinoSearch, then I have faith in your
> judgement -- but from my perspective, that seems like a strong argument in
> favor of archiving Lucy at this time and reviving it at a future date when
> you feel the time is right to bring he apprpriate code from KS into the
> apache fold via software grant.

The point of focusing on something named "KinoSearch" in the short term is to
test an API-stable release which implements Lucy extensibility enhancements
without spoiling the "Lucy" namespace.  The API stability is important because
it allows people to migrate on their own schedule from "KinoSearch" to "Lucy"
and gives them the confidence that their elaborate extensions won't suddenly
get blasted by a "Lucy 2.0" core update that breaks backwards compatibility.

If sane versioning was supported by all of Lucy's target platforms, this
two-step maneuver wouldn't be necessary: we could release a short-lived Lucy
1.0, then follow it up with a solid Lucy 2.0.  Unfortunately, that's not the
case, so I think it makes sense to hijack KS and use it as a vehicle.

Marvin Humphrey

Boy, I wish I had said it this well, Hoss.  +1.

I totally agree.  I very much respect what Marvin is doing with KS and  
the contributions to Lucene he has made and I'm definitely not trying  
to cast out the  people who feel strongly about a C-based search  
library.  I just know that Lucy doesn't meet the standard of what an  
Apache project should be and it has been given ample time to become  
that.   It is very much a tough thing for me to even suggest, because  
I know how much Marvin cares about it and I'd be happy to see a port  
of Lucene in every programming language there is if it is useful to  
people.

However, as PMC chair, I've had to fill out the board reports for a  
good while now with what amounts to "Lucy: No activity to report" (at  
best it says "minimal activity"), see [1], [2], [3], [4] and [5] (and  
I could probably keep going, see http://www.apache.org/foundation/board/calendar.html)
.  In fact, in [2] (Sept. '08), the report was:

"LUCY

Lucy will develop a shared C-based core for ports of Lucene to other
languages, such as Perl, Python and Ruby.  No progress has been made
this quarter, but we have been in contact with the committers and
they are still interested in the project and plan to be more
active in the near future."

So, it is not like my concern (or other that of other PMC members) is  
all of a sudden news.
However, since September, little has happened since the PMC contacted  
Dave and Marvin (Doug was obviously contacted since he's a member of  
the PMC, and that accounts for all the committers).

 From http://svn.apache.org/viewvc/lucene/lucy/, there hasn't been a  
commit in 6 months.  The website itself is still showing only the  
original news item announcing the project back in 2006.  Since Sept of  
'08, there has been a sum total of 39 messages on the mailing list.  
While it is clear Marvin very much has some notion of Lucy alive  
somewhere, it is, unfortunately, not alive at Apache.

However, what about some type of interim probation, for lack of a  
better word?  Marvin and Nathan (and whoever else), how about putting  
forth a plan for going forward with some reasonable milestones that we  
can all point to and see the progress?  And don't feel like it has to  
be some huge leap, either.   While this may seem like it is just  
"jumping through hoops", I don't think it is.  Lucy will never attract  
other developers if all the discussion of how to implement Lucy takes  
place on the KS and Lucene Java mailing lists.

Hope this helps,
Grant

[1] http://www.apache.org/foundation/records/minutes/2008/board_minutes_2008_12_17.txt
[2] http://www.apache.org/foundation/records/minutes/2008/board_minutes_2008_09_17.txt
[3] http://www.apache.org/foundation/records/minutes/2008/board_minutes_2008_06_25.txt
[4] http://www.apache.org/foundation/records/minutes/2008/board_minutes_2008_03_19.txt
[5] http://www.apache.org/foundation/records/minutes/2007/board_minutes_2007_12_19.txt
On Mar 8, 2009, at 6:17 PM, Chris Hostetter wrote:

>
> : It *is* actively progressing.  It's just that neither you nor  
> Grant are
> : willing to acknowledge that any of the design work I just did (in  
> happy
> : collaboration with Java Lucene devs) applies to Lucy.
>
> I don't really believe that's a fair characterization of the  
> comments made
> in this thread.
>
> The fundemental issue is really wether Lucy, as a project, is "alive".
>
> This is not about wether progress is being made towards a good C  
> Library
> for search, or wether there have been good design discussions to  
> further
> that goal, or wether there has been good collaboration amongst various
> people towrds common goals that can be implimented in multiple  
> projects --
> the answers to all of those questions may be "yes" (and i genuinely
> believe that they are) but that doesn't mean that Lucy, as a  
> project, is
> alive.
>
> : The proposal remains sound, and there is a deep hunger out there  
> for a solid C

On Mar 7, 2009, at 3:54 AM, Marvin Humphrey wrote:

> On Sat, Mar 07, 2009 at 03:02:22AM +0100, Jukka Zitting wrote:
>>> Please give me two to three months to make the next dev release of  
>>> KinoSearch.
>>
>> What will happen then?
>
> The next dev release of KS will present real world implementations  
> of many
> designs that have been discussed in Lucene and Lucy forums over the  
> last year.
>
> Some might see that as "progress". ;)

That is progress on KS, not Lucy.
>
>
>> When and how is Lucy development going to start?
>
> It *is* actively progressing.  It's just that neither you nor Grant  
> are
> willing to acknowledge that any of the design work I just did (in  
> happy
> collaboration with Java Lucene devs) applies to Lucy.
>
> Please go read <https://issues.apache.org/jira/browse/LUCENE-1458>  
> and see if
> you can still assert after you read it that no work is being done on  
> Lucy.  I
> warn you, it is a long thread. :)
>
>> You mention that in many cases other forums have been better for
>> discussing related design issues. What's the benefit of keeping the
>> Lucy project alive if there's next to no code or even discussion
>> there?
>
> The proposal remains sound, and there is a deep hunger out there for  
> a solid C
> IR library similar to Lucene.  The KS-then-Lucy progression is the  
> fastest and
> best way to get there.
>
> Things would have gone more smoothly and quickly if Dave Balmain had  
> been able
> to contribute more, but even with that setback, we will still reach  
> the
> finish.
>
>> I'm sure that everyone here would love to see Lucy become more  
>> active.
>> How could we help make that happen?
>
> Help Mike McCandless and Jason Rutherglen finish up their work on  
> the designs
> we've all been discussing.  This is a multi-way collaboration, and  
> Lucy
> benefits when I'm able to study alternatate implementations, just as  
> Java
> Lucene benefits from being able to see what other projects have done.
>
> Cross-pollination has worked very well in the past.  The indexing  
> speedups a
> while back started with McCandless riffing on the KinoSearch merge  
> model.  (He
> followed that up with plenty of interesting innovating on his own.)
>
>> As a wild idea: would there be interest in bringing the KinoSearch
>> codebase over to Apache through incubation?
>
> My main reservation is that I really want to see KS and Lucy play out
> sequentially, because I want Lucy to benefit from having seen how  
> the features
> now in KS work in the real world.  There's no sane versioning under  
> Perl/CPAN.
> You can't move from Lucy version 1 to Lucy version 2 without  
> screwing over
> your users, and therefore I don't want to merge the two projects  
> into one
> namespace.  If we did that, the unified project has to stay as an  
> "alpha" for
> that much longer, and it never really gets the benefit of seeing how a
> real-world release goes.
>
> If, then, we're proceeding sequentially as I recommend, I don't see  
> how
> putting KS through incubation does anything but slow us down.  All  
> we're doing
> is adding extra hoops to jump through.  It might be politically  
> expedient, but
> the engineer in me rebels at the waste, as does the loyal employee.
>
> From my perspective, what we have is an optics problem.  I'm working  
> full
> time, and I've been plenty active in the Lucene forums, but you and  
> Grant only
> see a big fat zero.  :(

It's not that you aren't doing good stuff Marvin.  You are.  I love  
your discussions on Lucene Java and you are no doubt doing interesting  
stuff on KS.  It's just that, as Jukka, has pointed out, Lucy has seen  
absolutely zero of this effort for over three years.  How long do we  
hang on to your promise?  And, it is only you doing so.  That is not a  
community and not a definition of an Apache project.  It's fine for KS  
b/c that doesn't live in Apache.  It's not fine for Lucy, IMO.  
Sorry.  Heck, you and Nathan having a two person discussion on Lucy-
dev on a regular basis would be enough to satisfy this.

-Grant

Hi,

On Sat, Mar 7, 2009 at 9:54 AM, Marvin Humphrey <[EMAIL PROTECTED]> wrote:
> From my perspective, what we have is an optics problem.  I'm working full
> time, and I've been plenty active in the Lucene forums, but you and Grant only
> see a big fat zero.  :(

Try to look at our end of the telescope: If all KinoSearch development
had happened at Lucy and vice versa, would you consider KinoSearch an
active and healthy project?

As Grant says, you and others could start fixing this by bringing at
least a part of the design discussions and related work to the Lucy
forums. Without that there's not much point in the project. An Apache
project can't be just a placefolder for a future contribution.

BR,

Jukka Zitting

Jukka Zitting:

> Try to look at our end of the telescope

I understand the metric you are applying, and I understand and support the
Apache community activity threshold test.  It is clear to me why this audit
has been triggered, and that by pursuing the matter the members of the PMC are
acting in the best interests of Apache.  Nevertheless, I believe that the
outcome of the audit will be something other than a simple shelving.

There are two main factors that have brought us to today's state of affairs.

First, after a promising and productive start, Dave Balmain became (mostly)
unavailable.  If Balmain had been able to maintain his participation, we
wouldn't be having this conversation.

Second, I became absolutely determined to improve the extensibility situation.
I haven't been able to give up on pluggable index components, flexible
indexing, bindings that facilitate rapid prototyping in the host language, a
robust ABI for compiled extensions, loose coupling between the library and its
file format, and human readable metadata and simple binary formats that make
index files easier to grok and debug.

That ambitious agenda is entirely in the spirit of Lucy.  Lucy aims to empower
diverse communities and feed off the dynamism of varied approaches.  It looks
to harness "loose port" energy and encourage cross-pollination.  By making it
as easy as possible to prototype powerful extensions in the host language, the
greater community benefits from being able to evaluate different perspectives.
Then, by making it as easy as possible to port the prototype to a compiled
extension, we encourage ideas to seep across language boundaries.

However, completing all those agenda items requires esoteric, difficult core
architectural work, boring and out of reach for most people.  The elements in
that list aren't the kind of thing where someone just gets an itch they want
to scratch and submits a patch -- they're the kind of thing that allows people
to scratch itches.  My colleagues at Eventful find the Lucy-originated
subclassing abilities in KS pretty convenient, but when I get all amped about
"installing callbacks into dynamically generated vtables", their eyes don't
exactly light up.

If Balmain had been around, we would have been designing these things jointly.
Instead, I've been seeking out collaborators where they can be found.  The OO
hierarchy that supports pluggable indexing components grew out of discussions
with Jason Rutherglen and Mike McCandless.  Nate Kurz was an early and
relentless advocate of mmap.  Peter Karman strongly influenced the design of
Schema and the implementation of human-readable index metadata; his experience
slinging Swish configuration files around came in handy.

When Lucy launched, we didn't have any of those features in either Ferret or
KinoSearch.  Now, prototype implementations are all either done or nearing
completion.  We are moving forward.

> As Grant says, you and others could start fixing this by bringing at
> least a part of the design discussions and related work to the Lucy
> forums.

The way this started off was with me maintaining duplicate code in the
KinoSearch and Lucy repositories.  (I have a little script I use to swap out
names and copyright notices.)  Code that was developed for Lucy was inserted
into KS so that it would get aired out in real-world situations.  Over time, I
expected Lucy code to gradually take over the KS repository.  (This is where
Peter, Nate and others acquired the impression that KS would ultimately become
Lucy.)

When Balmain was forced to curtail his participation, I gradually began to
develop features needed by Lucy in KS proper and put off the formal discussions
and the cross-commits.  Code started to accumulate, but it was always my
expectation that the material would need to be evaluated before going into
Lucy.

The sticking point is the full implementation of the "Boilerplater" code
generator that Balmain and I sketched out, which is now a small compiler,
along with five core utility classes: Obj, VTable, CharBuf, VArray and Hash.
I've been reluctant to commit those to Lucy without review, but there aren't
very many people out there who have both the expertise and the energy to
review it.  Yet since Boilerplater is a build tool prerequisite, without it
little else can go into the Lucy repository.  It's technically an
implementation detail, since it generates C header files which form a public
API, rather than exposes a public API itself -- but it's a pretty important
implementation detail.

As time has passed, we've gotten closer to the point where KS could publish a
public C API using tools that were built for Lucy.  Real-world feedback from
users would substitute for Balmain's formal review, and even if no single user
had Dave's level of expertise, we'd have enough to satisfy the community
participation requirement and Boilerplater could go in.

Unfortunately, while we're close to a public C API for KS, we aren't quite
there yet; too much time has passed, and the lack of visible activity on Lucy
has triggered an audit.  

Marvin Humphrey

For better or worse, Apache demands / expects an active community in
its projects and sub-projects.  The health of the community is far
more important than individuals on the project, or the code itself, or
net innovation/progress to the world, for Apache.  It's not clear this
is the "right" way, but it is the Apache way, for better or worse.

Fundamentally, the Apache way is simply not congruent with the "lone
innovator" way.  I know Lucy/KS are making awesome innovotions, and
that's thanks to your relentless passion & drive.  Lucy/KS innovates
at a much faster pace than Lucene and I for one am very much looking
forward to its eventual realization (plus I want to see how well mmap
works ;).

There's nothing "wrong" with either approach.  They are simply
different.  It's just like some investors prefer seed or early stage
investments, but others invest much later when the business is more
clearly "established".

I completely agree that Lucy/KS and Lucene have all moved forward
through very healthy cross-fertilization, carried out on java-dev.
That has clearly resulted in awesome innovations, for both Lucene and
Lucy/KS.  Lucene finally in 3.0 will finish the transition to wired
autoCommit=false for IndexWriter, something KS has had from the start,
for example (there are many more examples).

I also feel achieving a strong C core, friendly to dynamic languages,
is very important for Lucene's future.  Some day more people may use
it than Lucene (religion: I think dynamic languages, not Java, are the
eventual future).

Lucy/KS living elsewhere should not change that cross-fertilization,
and Lucy/KS will clearly live on even if in the short term it's not
under the Apache Lucene umbrella.

Mike

Marvin Humphrey wrote:

> On Sat, Mar 07, 2009 at 03:02:22AM +0100, Jukka Zitting wrote:
>>> Please give me two to three months to make the next dev release of  
>>> KinoSearch.
>>
>> What will happen then?
>
> The next dev release of KS will present real world implementations  
> of many
> designs that have been discussed in Lucene and Lucy forums over the  
> last year.
>
> Some might see that as "progress". ;)
>
>> When and how is Lucy development going to start?
>
> It *is* actively progressing.  It's just that neither you nor Grant  
> are
> willing to acknowledge that any of the design work I just did (in  
> happy
> collaboration with Java Lucene devs) applies to Lucy.
>
> Please go read <https://issues.apache.org/jira/browse/LUCENE-1458>  
> and see if
> you can still assert after you read it that no work is being done on  
> Lucy.  I
> warn you, it is a long thread. :)
>
>> You mention that in many cases other forums have been better for
>> discussing related design issues. What's the benefit of keeping the
>> Lucy project alive if there's next to no code or even discussion
>> there?
>
> The proposal remains sound, and there is a deep hunger out there for  
> a solid C
> IR library similar to Lucene.  The KS-then-Lucy progression is the  
> fastest and
> best way to get there.
>
> Things would have gone more smoothly and quickly if Dave Balmain had  
> been able
> to contribute more, but even with that setback, we will still reach  
> the
> finish.
>
>> I'm sure that everyone here would love to see Lucy become more  
>> active.
>> How could we help make that happen?
>
> Help Mike McCandless and Jason Rutherglen finish up their work on  
> the designs
> we've all been discussing.  This is a multi-way collaboration, and  
> Lucy
> benefits when I'm able to study alternatate implementations, just as  
> Java
> Lucene benefits from being able to see what other projects have done.
>
> Cross-pollination has worked very well in the past.  The indexing  
> speedups a
> while back started with McCandless riffing on the KinoSearch merge  
> model.  (He
> followed that up with plenty of interesting innovating on his own.)
>
>> As a wild idea: would there be interest in bringing the KinoSearch

Dear all,

I have been following this discussion from afar and basically by mere
luck because I happened to be subscribed to the general lucene list.
I do not understand much about the main topic of this thread, because I
know next to nothing about how open source development really works, and
even less about Apache projects per se.
I doubt that an expression of interest by a mere user will change much
to the debate, but I wanted to nevertheless express a strong interest in
what I perceive as being an underlying topic.

We are a small company providing services in the area of document
management and textual information retrieval. We are extremely
interested in a package such as Lucene, but which would have a Perl
binding.  In searching for such things over the years, I have
encountered the names Clucene, Plucene, Lucy, and Kinosearch (as well as
Lucene itself and Solr, and more distantly Xapian).  I had however
almost given up, because it seemed to me that not very much was
happening regarding the first four named, nor in terms of a
unicode-capable C (and consequently Perl) interface.  I had a bit the
impression that these things were quite dead.
The current thread would seem to indicate that this is not really so,
and to us that in itself is very good news.  By reading the messages on
this thread, I still don't really understand what is really going on
however.
Would someone care to explain what the current status is, in simple words ?

As a small company, we would be interested in contributing within the
limits of our capacities. Unfortunately, those do not include a high
level of C programming skills.  But we are specialists in "unstructured,
textual" information indexing and retrieval, so at the very least we
could help in testing with real-world data.  We also have applications
largely based on Apache and Perl, currently interfacing a high-quality
proprietary text indexing and retrieval system through a self-developed
Perl interface.  Our basic interest lies in, over time, finding an
alternative to that proprietary system.
But we are much less interested in a Java-based solution, because that
would really mean a huge investment for us, to rewrite much of our
existing code into Java, which we are not so fond of.

On Fri, Mar 6, 2009 at 6:02 PM, Jukka Zitting <[EMAIL PROTECTED]> wrote:
> What will happen then? KinoSearch is not Lucy, so I'm not sure how
> this is relevant. Are you then planning to start working on Lucy, or
> to contribute some of the existing KinoSearch code into Lucy?

I don't know the exact distinction, but to a reasonable approximation
KinoSearch can be viewed as the Perl bindings on top of the alpha
version of Lucy.  The vast majority of the work is done in the C
portion of the code, and my understanding is that once this separation
is complete and the API is more final, the C codebase will be renamed
Lucy.  Marvin is thus essentially employed full time to work on Lucy.

> Are there other people who are planning to start working on Lucy?

I'm currently distracted with other projects, but I'm very interested
in the long-term development of Lucy.  I've been corresponding with
Marvin for the past couple years, and am glad to see the project
gradually moving in the direction I want it to go: a light, fast,
flexible framework that can be used as the base for all variety of
search applications.   I'm hoping to contribute more actively in the
future.

Nathan Kurz
[EMAIL PROTECTED]

Hi,

On Sat, Mar 7, 2009 at 7:34 AM, Nathan Kurz <[EMAIL PROTECTED]> wrote:
> I don't know the exact distinction, but to a reasonable approximation
> KinoSearch can be viewed as the Perl bindings on top of the alpha
> version of Lucy.

That's not Lucy code until it's being developed in Lucy. And without
Lucy code it's pretty pointless to have a Lucy project.

> The vast majority of the work is done in the C portion of the code,
> and my understanding is that once this separation is complete and
> the API is more final, the C codebase will be renamed Lucy.

Any larger externally developed codebase can only be imported to
Apache through incubation or a software grant, and the software grant
can only be used if there exists an active community that wants to
take over the development of that code. The latter is not true (there
is no active community) at the moment in Lucy, so at this point any
code from KinoSearch would need to go through full incubation. This is
which is why I suggested the idea of KinoSearch being brought to the
Incubator.

BR,

Jukka Zitting

Marvin Humphrey wrote:

> The conversations that we had in JIRA and on java-dev were  
> beneficial to both
> Lucene and Lucy; should I have posted to the Lucy dev list instead  
> simply to
> demonstrate activity, which would have been less useful to Mike, to  
> me, to
> Lucy, and to Lucene?  To my mind, the Lucene community is also part  
> of the
> Lucy community.

This is a good point... it is unfortunate that the large healthy
community of java-user/dev indeed does tend to draw discussions away
from its "offshots".  Much like most acorns dropping from a large
healthy oak tree will fail to grow because they get no sun.

EG you recently posted about Matcher, to lucy-dev.  You then posted to
java-dev, pointing to that post (which was a great idea).  I pondered
whether to respond via java-dev or lucy-dev.  I knew java-dev would
get more discussion, but I also really, really want Lucy to grow &
succeed, so I responded on lucy-dev.  But then the discussion stopped.

I do think these discussions (and it's not just LUCENE-1483, there
were a number of other issues & just plain discussions on java-dev) do
indeed "count" as good Lucy activity.

Mike

On Mar 6, 2009, at 4:58 PM, Marvin Humphrey wrote:

> Grant,
>
> I am currently employed by Eventful, Inc, in San Diego, CA.  They  
> are paying
> me to work full-time on KinoSearch and Lucy.
>
> I went out of my way when we negotiated the terms of my employment  
> to ensure
> that there was no way my contract could hamper or compromise  
> progress towards
> Lucy.  The actual document is confidential of course, but I feel  
> comfortable
> saying that first, our lawyers hammered out the legal nuts and bolts  
> to my
> satisfaction, and second, Eventful is fully on board with regards to  
> Lucy.  By
> way of illustration, my boss regularly hassles me about publishing a  
> Lucy C
> API, even though since Eventful uses the Perl bindings the benefits  
> would be
> indirect.
>

This just further underscores my point.  Lucy cannot be just about you  
(and your employer) contributing code that you develop in-house at  
Eventful.  A project must be able to survive any single committer  
leaving the project and simply put, Lucy does not meet that criteria.  
In the early stages, yes, often one committer gets things going, but  
Lucy's been around for a fairly long time on life support and you only  
seem to pop up on the list when nudged by the PMC.
> In my opinion, it is not in the best interests of the Apache Lucene  
> project to
> make it more difficult for my employer and myself to contribute.

I agree, but unfortunately, it is Lucy that has languished for a good  
long time.

>
>
>> It is fairly apparent to me that the Lucy project is not making any
>> progress community-wise or code-wise.  Neither Marvin, Dave or Doug
>> are active at all on it, and that accounts for all three committers.
>> There has been very little mailing list traffic,
>
> You may have noticed that up until about three weeks ago (when I  
> dove back
> into the code cave), I was quite active on java-
> [EMAIL PROTECTED] and in
> the Lucene JIRA forums.  Significant design innovations were realized,
> particularly in the area of real-time search.
>
> In the past, many designs have been hashed out cooperatively on the  
> KinoSearch
> and Lucy mailing lists: the Schema class, revisions to QueryParser  
> and the
> boolean Query hierarchy, the implementation of human-readable index  
> metadata,
> C configuration probing, the OO model, index designs which exploit  
> memory
> mapping, and so on.
>
> In this particular case, however, I was assigned the task of solving  
> real-time
> search, for which the Lucy and KinoSearch forums were not ideal.  
> There is a
> very limited number of people who have both the familiarity with the
> Lucene/Lucy segment-based inverted index model and the interest to  
> discuss
> real-time search at the level I desired, where concepts like  
> "segment-centric
> search" could be bandied about.  Basically, I needed Mike McCandless  
> -- so I
> went to where he could be found.
>
> The conversations that we had in JIRA and on java-dev were  
> beneficial to both
> Lucene and Lucy; should I have posted to the Lucy dev list instead  
> simply to
> demonstrate activity, which would have been less useful to Mike, to  
> me, to
> Lucy, and to Lucene?  To my mind, the Lucene community is also part  
> of the
> Lucy community.  Mike's insights were welcome and useful, and it  
> didn't seem
> important to me which specific mailing list they wound up on --  
> they're all
> under the domain lucene.apache.org, after all.  Weren't we all  
> moving forward
> together, and wouldn't that be apparent to members of the PMC such as
> yourself?
>
> Or is this a zero-sum game where design innovations which help Lucy  
> don't
> count as "progress" if they also help Lucene?

That's all fine, but none of it adds up to people looking at Lucy and  
saying "Gee, I want to contribute to Lucy"
>
>
>> Furthermore, I have my doubts about the development process being  
>> employed,
>> which seems to be the notion that KinoSearch is going to be donated  

What's that got to do with anything?  Give me a break.  I'm not  
attacking you.  I'm just stating that Lucy has not had any code or any  
community built for over three years.
Again, this is all great, but it just further demonstrates that you  
are doing this on your own and not as a part of the Lucy community (or  
really, even the Lucene community).  It's not a judgment of you or of  
KS.  I really like what you are doing.  It's merely a statement that  
this is not how Apache works.   There are plenty of other places to  
host code that do not have these requirements.

On Mar 7, 2009, at 7:14 AM, Grant Ingersoll wrote:
> Again, this is all great, but it just further demonstrates that you  
> are doing this on your own and not as a part of the Lucy community  
> (or really, even the Lucene community).  It's not a judgment of you  
> or of KS.  I really like what you are doing.  It's merely a  
> statement that this is not how Apache works.   There are plenty of  
> other places to host code that do not have these requirements.
I should back off a little bit.  I really do not want Lucy to go  
away.  I do want a C port of Lucene (even a loose one).  So, let's  
figure out how to build a community around it.

I think Jukka's suggestion is the best:  Donate KS and or some portion  
of it to the incubator that you think will make for Lucy and we will  
put it through incubation.  I'd even be happy to mentor it and I would  
bet the PMC would sponsor it.

-Grant

Grant Ingersoll wrote:

> I really do not want Lucy to go away.

I also certainly do not want Lucy to go away; I want the innovations
Lucy/KS is pursuing to continue, and the cross-feritilization with
Lucene to continue.  I would think that they could and should even if
Lucy is not under the Lucene umbrella in the short term.

>   I do want a C port of Lucene (even a loose one).
BTW, I think "loose" ports are far more interesting than strict ports,
since they have the freedom to introduce neat innovations, enabling
two-way cross fertilization.  Lucene grows much more due to loose
ports.  I've learned so much more from Marvin, and Lucene has
benefited far more from Marvin's work, than from the strict Lucene
ports.

Mike

Grant Ingersoll wrote:
> Therefore, it is with some hesitation that I suggest we mothball Lucy.

When committers are inactive for a year, we ask them if they'd like to
be made emeritus.  Usually they either don't respond or they say yes.
If they say "no" we give them the benefit of the doubt for a while
longer.  A similar process is followed for Apache members who've been
inactive.  Inactivity should not be punished: we're all volunteers.

If there's a decent chance that Lucy will become active soon then we
don't want to further burden that.  Marvin has argued that there's a
strong chance Lucy will become active soon.  I'm inclined to give him
the benefit of the doubt for another six months or so before we do
anything that would be nontrivial to reverse.

So if "mothballing" would be easy to reverse, it might be okay.  We
could just, e.g., change the website to say, "inactive", remove the
website from the TLP's website and remove commit privileges, but not
remove accounts, mailing lists or JIRA instances.  Then reversal would
take about 10 minutes of the PMC chair's time.  Marvin or others could
petition this list to have it reversed.

But I'd also be fine with putting the project on notice for a few
months, with the understanding that if activity doesn't pick up soon, it
will be mothballed, as outlined above.  Then, after mothballing, if
nothing happens for a while longer, we remove it altogether.  At each
transition we should invite discussion.  I don't see much point in
rushing this.

This path does risk dragging out the pain, so we shouldn't go down it
too lightly.  Marvin, do you expect you'll be actively committing code
to Lucy over the next six months?  If so, I think we should give him
that, if not, we should mothball now as outlined above.

Another analysis to consider: If Marvin came to us today and pitched
Lucy, would we accept it as a single-committer subproject?  If not, then
we should mothball now.  If so, then we can give him the benefit of the
doubt for a bit longer.

Note that, if we give notice now, and nothing happens in six months,
that's likely to considerably reduce our patience.

Doug

P.S. I hereby resign as a Lucy committer.

+1.  I don't see any point in rushing it either, but I don't feel like  
we are, other than the sense that some decision needs to be made based  
on this thread.  Like I said before, this conversation isn't news to  
those who are involved.  I'm totally fine w/ giving the benefit of the  
doubt, I trust Marvin is working on it, but likely needs a reminder of  
how Apache projects work.

I'd _suggest_ a few other things beyond just code commits, however:
1. When discussing ideas on java-dev, at least send a message to lucy-
dev saying "See thread X on java-dev" (however, please don't cross-
post).  Do this religiously.  At some point, after you start seeing  
Lucy members commenting on java-dev, you will realize you have enough  
of a group to sustain the conversations on lucy (or, even general;  
General is likely a good place for cross-fertilization too) at which  
point you will likely be sending a msg to java-dev saying, "hey check  
out lucy-dev for a great conversation on X"
2. Update the website to show a current status and/or post some news.
3. Move beyond the "Eventful and I" are figuring out this stuff and  
start thinking in terms of how you can build community in Lucy.  As  
should be clear now, Apache is more than just code.  Code can be  
stored in a number of places (SF, Google, etc.).  Apache is about  
building community around code.
4. Related to #3: Actively start searching for replacements for Doug  
and Dave (hint, I think you have two volunteers on this thread  
already).   Keep in mind the litmus test for an Apache project: Would  
this project survive a committer leaving?  In the short term, we know  
the answer is no, but in the long term the answer must be yes.  Try to  
figure out some specific areas you could get help.  Sign up to be a  
mentor for GSOC and try to get a student to help out.  Blog/Twitter/
Whatever about Lucy and what you are up to.  In short, start promoting  
Lucy as Lucy, not as some offshoot of KS.
5. Try to think in terms of how KS can leverage Lucy and less in terms  
of how Lucy can be extracted from KS at some point in the future  
(which is very much the sense I get from reading the responses).  In  
other words, even if you think people aren't capable of doing the low-
level grunt work you allude to in prior posts, assume that any and  
everyone on the Lucy list does grok that stuff and discuss it there.  
I often don't understand everything some of the guys on Mahout talk  
about, but it is a great learning place and eventually it gets through  
my skull.

Finally, six months or so does sound like the right time frame.

HTH,
Grant
On Mar 9, 2009, at 6:06 PM, Doug Cutting wrote:

> Grant Ingersoll wrote:
>> Therefore, it is with some hesitation that I suggest we mothball  
>> Lucy.
>
> When committers are inactive for a year, we ask them if they'd like  
> to be made emeritus.  Usually they either don't respond or they say  
> yes. If they say "no" we give them the benefit of the doubt for a  
> while longer.  A similar process is followed for Apache members  
> who've been inactive.  Inactivity should not be punished: we're all  
> volunteers.
>
> If there's a decent chance that Lucy will become active soon then we  
> don't want to further burden that.  Marvin has argued that there's a  
> strong chance Lucy will become active soon.  I'm inclined to give  
> him the benefit of the doubt for another six months or so before we  
> do anything that would be nontrivial to reverse.
>
> So if "mothballing" would be easy to reverse, it might be okay.  We  
> could just, e.g., change the website to say, "inactive", remove the  
> website from the TLP's website and remove commit privileges, but not  
> remove accounts, mailing lists or JIRA instances.  Then reversal  
> would take about 10 minutes of the PMC chair's time.  Marvin or  
> others could petition this list to have it reversed.
>
> But I'd also be fine with putting the project on notice for a few  

Grant Ingersoll wrote:
> I'd _suggest_ a few other things beyond just code commits, however:

+1 for all these.

> Finally, six months or so does sound like the right time frame.

So we can consider this fair warning: if there have only been a few
token commits, and there isn't activity consistent with building a
community, we mothball then.

Doug

+1, this seems like the right next step.

A few more observations on building a community... it can sometimes be
counterintuitive:

For example, the fact that Marvin is so incredibly responsive whenever
issues are raised on the KS lists can prevent community growth.
If he held back a bit, giving others the chance & motivation to answer
other user's questions, this'd help spread things out and grow the
community.

Also: code need not and should not be perfect before entering
SVN.  It's fine to accept some dirt, as long as the code is net/net an
overall step in the right direction -- it's progress not perfection.  
Once
it's in the limelight, people will see the dirt and fix it with time,
which grows the community.  If it comes in nearly perfect there's
nothing to do... and the more perfect the code is, the less people
have to come out and ask for help.

Lucene has all sorts of interesting warts, odd defaults, a wide
plethora of APIs and packages that have changed with time, and are
scattered between core/contrib and are at different levels of
maturity, etc., etc., all of which provides great "fodder" for a
community.

When a new user makes a contribution you should bend over backwards to
help get that in and then keep them involved.  Respond quickly, be
energetic, don't worry about the dirt, just get it in (if it's overall
a step forward)... because that's a chance to grow the community which
is far more valuable than growing the code.

I for one would love the see the current Lucy code checked in, even if
it's not done yet, has some dirt, etc.  Put big comments at the top
with the current status and your known TODOs.  Then make fixes over
time incrementally, publicly.

Mike

Grant Ingersoll wrote:

> +1.  I don't see any point in rushing it either, but I don't feel  
> like we are, other than the sense that some decision needs to be  
> made based on this thread.  Like I said before, this conversation  
> isn't news to those who are involved.  I'm totally fine w/ giving  
> the benefit of the doubt, I trust Marvin is working on it, but  
> likely needs a reminder of how Apache projects work.
>
> I'd _suggest_ a few other things beyond just code commits, however:
> 1. When discussing ideas on java-dev, at least send a message to  
> lucy-dev saying "See thread X on java-dev" (however, please don't  
> cross-post).  Do this religiously.  At some point, after you start  
> seeing Lucy members commenting on java-dev, you will realize you  
> have enough of a group to sustain the conversations on lucy (or,  
> even general; General is likely a good place for cross-fertilization  
> too) at which point you will likely be sending a msg to java-dev  
> saying, "hey check out lucy-dev for a great conversation on X"
> 2. Update the website to show a current status and/or post some news.
> 3. Move beyond the "Eventful and I" are figuring out this stuff and  
> start thinking in terms of how you can build community in Lucy.  As  
> should be clear now, Apache is more than just code.  Code can be  
> stored in a number of places (SF, Google, etc.).  Apache is about  
> building community around code.
> 4. Related to #3: Actively start searching for replacements for Doug  
> and Dave (hint, I think you have two volunteers on this thread  
> already).   Keep in mind the litmus test for an Apache project:  
> Would this project survive a committer leaving?  In the short term,  
> we know the answer is no, but in the long term the answer must be  
> yes.  Try to figure out some specific areas you could get help.  
> Sign up to be a mentor for GSOC and try to get a student to help  
> out.  Blog/Twitter/Whatever about Lucy and what you are up to.  In  
> short, start promoting Lucy as Lucy, not as some offshoot of KS.
> 5. Try to think in terms of how KS can leverage Lucy and less in  
> terms of how Lucy can be extracted from KS at some point in the  
> future (which is very much the sense I get from reading the  
> responses).  In other words, even if you think people aren't capable  

: Another analysis to consider: If Marvin came to us today and pitched Lucy,
: would we accept it as a single-committer subproject?  If not, then we should
: mothball now.  If so, then we can give him the benefit of the doubt for a bit
: longer.

speaking for myself: I would be -1 for any new subproject that didn't have
*more* then 3 people interested in being initial committers ... there's
always some level of attrition, and a project really needs at least 3
people actively involved to get anywhere.

I'm fine with Doug's proposal to wait a while, but as other people have
expressed: i'm more concerend with seeing Lucy build up a community of
people interested in maintaining the code then i am with seeing code get
committed.  Even if thousands of lines of production quality code gets
committed in the next 6 months that doesn't mean Lucy will have a
community to support/improve that code moving forward beyond that.
-Hoss

On Mon, Mar 09, 2009 at 03:06:45PM -0700, Doug Cutting wrote:

> Marvin, do you expect you'll be actively committing code
> to Lucy over the next six months?

Yes.  I can commit Boilerplater and its support classes.

The general design of Boilerplater was hashed out by myself and Dave Balmain.
All of the code is new, and was written specifically for Lucy.

To the extent that Boilerplater extends or diverges from what Balmain and I
designed, I can post rationales to lucy-dev or to JIRA.

Four of the support classes are nothing controversial: Obj, CharBuf (a Unicode
string buffer class), Hash and VArray are all basic CompSci projects.

VTable's design is intimately tied to Boilerplater.  I would have preferred
that Boilerplater and VTable undergo review by someone of Balmain's level of
expertise prior to committing, but that's not absolutely essential.  If it
were itself a public API, then I think review would be more important;
however, it's more of an implementation that's used to generate header code
that adheres to a pre-existing design.  It can be changed later if weaknesses
are discovered.

Next up after that would presumably be the contents of Lucy::Store.

Marvin Humphrey

Torsten Curdt wrote:

>>>  I do want a C port of Lucene (even a loose one).
>>
>>
>> BTW, I think "loose" ports are far more interesting than strict  
>> ports,
>> since they have the freedom to introduce neat innovations, enabling
>> two-way cross fertilization.  Lucene grows much more due to loose
>> ports.  I've learned so much more from Marvin, and Lucene has
>> benefited far more from Marvin's work, than from the strict Lucene
>> ports.
>
> Talking about that - have you guys seen this?
>
> http://vafer.org/blog/20090107014544
>
> LuceneKit is a Objective-C port. I think it would be great to have it
> part of the Lucene community. But I fear it has similar problems that
> Lucy has.

Excellent!  I hadn't seen LuceneKit, thanks.

Mike

Grant wrote:
Therefore, it is with some hesitation that I suggest we mothball
Lucy.  Mostly, I hesitate, because I hate to see any project be
archived on the hope that someone will come in and pick it up.
However, I just don't see that happening.  If Marvin wishes to
resurrect it, he can donate KS (or whatever core part of it is Lucy)
and go through incubation and prove there is a community and then we
can turn it back on.
Hi Grant,

I just subscribed to this list because I saw your post to the lucy-dev
list, where I have been subscribed since the project started. Thanks for
initiating this conversation.

It's true that Lucy still has no code. I am seeing at least a half-dozen
daily commits to KS svn however, and I know that Marvin still plans to
donate the core C pieces of that project to Lucy. KS has gained a lot of
traction in the Perl user community, even if the number of committers is
low (e.g., I have a commit bit on KS svn but haven't contributed much in
the last year).

So I would hate to see the idea of Lucy mothballed. Whether KS still
bears enough resemblance to Lucene to be called a 'loose port' is
probably debate-able, and so for reasons of [political] identity perhaps
it doesn't belong under the Lucene umbrella anymore. But I know the
cross-fertilization of ideas between the Lucene and KS developers has
been significant, and the idea of a core C IR library with the features
that KS has is very appealing (at least to me).

I'll leave it to Marvin to comment on whether he's ready to commit code
to Lucy and/or if that's still the appropriate vehicle for his vision.

pek
--
Peter Karman  .  [EMAIL PROTECTED]  .  http://peknet.com/

On Mar 6, 2009, at 11:06 AM, Peter Karman wrote:
>
> So I would hate to see the idea of Lucy mothballed. Whether KS still
> bears enough resemblance to Lucene to be called a 'loose port' is
> probably debate-able, and so for reasons of [political] identity  
> perhaps
> it doesn't belong under the Lucene umbrella anymore. But I know the
> cross-fertilization of ideas between the Lucene and KS developers has
> been significant, and the idea of a core C IR library with the  
> features
> that KS has is very appealing (at least to me).

My suggestion is, then, that you start making yourself heard on the  
Lucy mailing lists and start discussing ideas there more.

>
>
> I'll leave it to Marvin to comment on whether he's ready to commit  
> code
> to Lucy and/or if that's still the appropriate vehicle for his vision.

My understanding of the ASF is such that I don't believe it is proper  
for him to do so without either filling out a Software Grant or going  
through incubation.  I'd be happy to be corrected and I'd be happy to  
help guide the process if I am right and Marvin is willing to donate  
some large chunk of code that has been developed elsewhere.

Thus, if there are core parts of KS that are going to come from Lucy,  
then those parts need to be developed in the Lucy community by Lucy  
community members and not in KS first and not by one person in a  
(near) complete vacuum (unless we go the donation route.)

There is nothing preventing the further cross-fertilization of ideas  
between Lucene and KS, though.  We always welcome Marvin's input on  
discussions about how best to implement ideas (and anyone elses, for  
that matter) and I see no reason why mothballing Lucy would prevent  
the discussion of ideas that so often takes place on the Lucene lists.

-Grant