Many Lucene/Solr committers think that merging development would be a
benefit to both projects.
Separate downloads would remain (among other things), so end users
would not be impacted (except for higher quality products over time).
Since this is a change to Lucene/Solr project development, I'd like to
get a format vote from the committers of both projects.
If there are 3 +1s and more +1s than -1s, we can pass this to the
Lucene PMC to ratify.

-Yonik

Discussion thread:
http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene

On 03/03/2010 05:42 PM, Yonik Seeley wrote:
> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>    

+1

--
- Mark

http://www.lucidimagination.com

+1

Mike

On Wed, Mar 3, 2010 at 5:42 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>

What does merging development mean?  I think we probably should be more
specific since that can mean different things to different folks.  We should
be all voting on the same thing.

Bill

On Wed, Mar 3, 2010 at 5:42 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:

> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
>
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>

See the linked to discussion. (I know its pretty long)

I think Mike covered the broad strokes here:
http://search.lucidimagination.com/search/document/477615025eaa0e9d/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene

Though I'm sure its open to tuning. I think the discussion gives the
gist of what the pro mergers want though (though many responders
appeared to be confused as to what most of us think the merge means, so
who knows)

- Mark

On 03/03/2010 06:48 PM, Bill Au wrote:
> What does merging development mean?  I think we probably should be more
> specific since that can mean different things to different folks.  We should
> be all voting on the same thing.
>
> Bill
>
> On Wed, Mar 3, 2010 at 5:42 PM, Yonik Seeley<[EMAIL PROTECTED]>  wrote:
>
>    
>> Many Lucene/Solr committers think that merging development would be a
>> benefit to both projects.
>> Separate downloads would remain (among other things), so end users
>> would not be impacted (except for higher quality products over time).
>> Since this is a change to Lucene/Solr project development, I'd like to
>> get a format vote from the committers of both projects.
>> If there are 3 +1s and more +1s than -1s, we can pass this to the
>> Lucene PMC to ratify.
>>
>> -Yonik
>>
>> Discussion thread:
>>
>> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>>
>>      
>    
--
- Mark

http://www.lucidimagination.com

I also feel like this vote is too broad. E.g. I'm ok with merging
dev-lists and committers, but unhappy about aligning releases. What
points exactly are we voting here on?

  Michael

On 3/3/10 3:54 PM, Mark Miller wrote:
> See the linked to discussion. (I know its pretty long)
>
> I think Mike covered the broad strokes here:
> http://search.lucidimagination.com/search/document/477615025eaa0e9d/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>
>
> Though I'm sure its open to tuning. I think the discussion gives the
> gist of what the pro mergers want though (though many responders
> appeared to be confused as to what most of us think the merge means,
> so who knows)
>
> - Mark
>
> On 03/03/2010 06:48 PM, Bill Au wrote:
>> What does merging development mean?  I think we probably should be more
>> specific since that can mean different things to different folks.  We
>> should
>> be all voting on the same thing.
>>
>> Bill
>>
>> On Wed, Mar 3, 2010 at 5:42 PM, Yonik Seeley<[EMAIL PROTECTED]>  wrote:
>>
>>> Many Lucene/Solr committers think that merging development would be a
>>> benefit to both projects.
>>> Separate downloads would remain (among other things), so end users
>>> would not be impacted (except for higher quality products over time).
>>> Since this is a change to Lucene/Solr project development, I'd like to
>>> get a format vote from the committers of both projects.
>>> If there are 3 +1s and more +1s than -1s, we can pass this to the
>>> Lucene PMC to ratify.
>>>
>>> -Yonik
>>>
>>> Discussion thread:
>>>
>>> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>>>
>>>
>
>

Agreed - a better description of what we are voting on should have been
included.

I'm only for the merge with aligned releases - its the only way Solr can
really stay on Lucene trunk happily. Aligned releases are also my
biggest worry (and part of why I initially leaned against such a merge),
but without it, there goes all of the larger reasons I'm into the merge
now - Solr can be on trunk and we can have better sharing / less
duplication between the projects - which I personally think requires
Solr being on Lucene trunk - or it won't really work at all. And Solr
being on trunk really needs aligned releases

So we should probably spell out what the vote would mean a bit more
formally. I assumed we were essentially voting on the proposal Mike put
out - but that is certainly an assumption.

- Mark

On 03/03/2010 07:00 PM, Michael Busch wrote:
> I also feel like this vote is too broad. E.g. I'm ok with merging
> dev-lists and committers, but unhappy about aligning releases. What
> points exactly are we voting here on?
>
>  Michael
>
> On 3/3/10 3:54 PM, Mark Miller wrote:
>> See the linked to discussion. (I know its pretty long)
>>
>> I think Mike covered the broad strokes here:
>> http://search.lucidimagination.com/search/document/477615025eaa0e9d/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>>
>>
>> Though I'm sure its open to tuning. I think the discussion gives the
>> gist of what the pro mergers want though (though many responders
>> appeared to be confused as to what most of us think the merge means,
>> so who knows)
>>
>> - Mark
>>
>> On 03/03/2010 06:48 PM, Bill Au wrote:
>>> What does merging development mean?  I think we probably should be more
>>> specific since that can mean different things to different folks.  
>>> We should
>>> be all voting on the same thing.
>>>
>>> Bill
>>>
>>> On Wed, Mar 3, 2010 at 5:42 PM, Yonik Seeley<[EMAIL PROTECTED]>  wrote:
>>>
>>>> Many Lucene/Solr committers think that merging development would be a
>>>> benefit to both projects.
>>>> Separate downloads would remain (among other things), so end users
>>>> would not be impacted (except for higher quality products over time).
>>>> Since this is a change to Lucene/Solr project development, I'd like to
>>>> get a format vote from the committers of both projects.
>>>> If there are 3 +1s and more +1s than -1s, we can pass this to the
>>>> Lucene PMC to ratify.
>>>>
>>>> -Yonik
>>>>
>>>> Discussion thread:
>>>>
>>>> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>>>>
>>>>
>>
>>
>
--
- Mark

http://www.lucidimagination.com

On Wed, Mar 3, 2010 at 7:41 PM, Mark Miller <[EMAIL PROTECTED]> wrote:
> I'm only for the merge with aligned releases - its the only way Solr can
> really stay on Lucene trunk happily. Aligned releases are also my biggest
> worry (and part of why I initially leaned against such a merge), but without
> it, there goes all of the larger reasons I'm into the merge now - Solr can
> be on trunk and we can have better sharing / less duplication between the
> projects - which I personally think requires Solr being on Lucene trunk - or
> it won't really work at all. And Solr being on trunk really needs aligned
> releases

Correct - I believe most who agreed with a merge essentially agreed on
all the points about what a merge meant.  Merged dev list, committers,
and releases.  Maintain separate downloads, user lists.

 I wanted to avoid the lawyers... if we need to hammer out all the
little things that aren't as important (sync release numbers, etc)
discussion will be endless and we'll never get anywhere.

-Yonik

On 3/3/10 4:41 PM, Mark Miller wrote:
> Agreed - a better description of what we are voting on should have
> been included.
>
> I'm only for the merge with aligned releases - its the only way Solr
> can really stay on Lucene trunk happily. Aligned releases are also my
> biggest worry (and part of why I initially leaned against such a
> merge), but without it, there goes all of the larger reasons I'm into
> the merge now - Solr can be on trunk and we can have better sharing /
> less duplication between the projects - which I personally think
> requires Solr being on Lucene trunk - or it won't really work at all.
> And Solr being on trunk really needs aligned releases

So if it seems like that most people are concerned about releases (even
those you are generally in favor of this proposal), then maybe we should
discuss exactly this point. We haven't really discussed alternatives
about the release alignment. This vote feels rushed.

  Michael

On Wed, Mar 3, 2010 at 8:10 PM, Michael Busch <[EMAIL PROTECTED]> wrote:
> On 3/3/10 4:41 PM, Mark Miller wrote:
>>
>> Agreed - a better description of what we are voting on should have been
>> included.
>>
>> I'm only for the merge with aligned releases - its the only way Solr can
>> really stay on Lucene trunk happily. Aligned releases are also my biggest
>> worry (and part of why I initially leaned against such a merge), but without
>> it, there goes all of the larger reasons I'm into the merge now - Solr can
>> be on trunk and we can have better sharing / less duplication between the
>> projects - which I personally think requires Solr being on Lucene trunk - or
>> it won't really work at all. And Solr being on trunk really needs aligned
>> releases
>
> So if it seems like that most people are concerned about releases (even
> those you are generally in favor of this proposal), then maybe we should
> discuss exactly this point. We haven't really discussed alternatives about
> the release alignment. This vote feels rushed.

It's been discussed for a week, and I'm with Mark - I'm only for a
real merge of development, and that includes release schedules.

-Yonik

+1

On Mar 3, 2010 5:43 PM, "Yonik Seeley" <[EMAIL PROTECTED]> wrote:

Many Lucene/Solr committers think that merging development would be a
benefit to both projects.
Separate downloads would remain (among other things), so end users
would not be impacted (except for higher quality products over time).
Since this is a change to Lucene/Solr project development, I'd like to
get a format vote from the committers of both projects.
If there are 3 +1s and more +1s than -1s, we can pass this to the
Lucene PMC to ratify.

-Yonik

Discussion thread:
http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene

On 3/3/10 6:00 PM, Yonik Seeley wrote:
> On Wed, Mar 3, 2010 at 8:10 PM, Michael Busch<[EMAIL PROTECTED]>  wrote:
>    
>> So if it seems like that most people are concerned about releases (even
>> those you are generally in favor of this proposal), then maybe we should
>> discuss exactly this point. We haven't really discussed alternatives about
>> the release alignment. This vote feels rushed.
>>      
> It's been discussed for a week, and I'm with Mark - I'm only for a
> real merge of development, and that includes release schedules.
>
> -Yonik
>
>    

How will we merge release policies? (or are they exactly the same?) Does
Solr use the same release numbering? Does it have the same
backwards-compatibility requirements as Lucene?

If we release Solr 1.5.0 with Lucene 3.1.0, and we find a bug in the
Lucene jar and want to release a 3.1.1 bugfix release, will we also
release a Solr 1.5.1, even though all Solr jars would be identical to
the 1.5.0?
Or will we just release Solr/Lucene 4.0.0 next and always use same
release numbers?

How will we avoid longer release cycles? Solr had had very infrequent
releases. What were the reasons for that? Are we comfortable with saying
we'll just try to be disciplined enough or is there a way to somehow
enforce release frequency? It seems certain that if more people work on
the code, there will at any given point be more patches/new features
under development and things need to be coordinated in a way that allows
frequent releases.

In an earlier mail I gave the following example: If we had a separate
analyzer module, and we add an analyzer in a new language to it, it
would be cool to release it soon, without having to wait until
Lucene/Solr are ready for a release. The pace here can be faster,
because I imagine in such an Analyzer module it's much less common that
a patch "touches everything". What do people think about this? Maybe
it's just a nice wish, but not realizable, because there'd be a lot of
version management overhead. But maybe not?

I'm not against this whole thing and I'm not trying to be difficult
here, and I dislike endless discussions just as everyone else. But I
honestly don't know right now how to vote, because I have those open
questions and know that a lot of other people here are concerned about
the release alignment as well.

  Michael

> In an earlier mail I gave the following example: If we had a separate
> analyzer module, and we add an analyzer in a new language to it, it would be
> cool to release it soon, without having to wait until Lucene/Solr are ready
> for a release. The pace here can be faster, because I imagine in such an
> Analyzer module it's much less common that a patch "touches everything".
> What do people think about this? Maybe it's just a nice wish, but not
> realizable, because there'd be a lot of version management overhead. But
> maybe not?
>

Hi Michael, this is a great question. In my opinion, it would be nice
for any analyzer to have some time to sit around in trunk before being
released. In general we have been trying to treat contrib/analyzers
with some special care and also maintain more backwards compatibility,
etc. It helps if we can somehow shake out any problems before
releasing them.

But there is a a slight problem: I feel that new
analyzers/improvements to existing ones in lucene trunk are really not
being tested by many people, because Solr is not using lucene trunk. I
think if they were available to the many Solr trunk users, then this
would give us more feedback and better quality releases, just from an
analysis-only perspective.
--
Robert Muir
[EMAIL PROTECTED]

In the case where changes are in Lucene only I think it is OK to increment
the Solr release number since even though the Solr jars are unchanged
because the new release of Solr will contain the new Lucene  jars.  But what
about the case where changes are in Solr only?  Would we still increment the
Lucene release number even though everything in the Lucene download is the
same as before?

Bill

On Wed, Mar 3, 2010 at 10:13 PM, Michael Busch <[EMAIL PROTECTED]> wrote:

> On 3/3/10 6:00 PM, Yonik Seeley wrote:
>
>> On Wed, Mar 3, 2010 at 8:10 PM, Michael Busch<[EMAIL PROTECTED]>  wrote:
>>
>>
>>> So if it seems like that most people are concerned about releases (even
>>>
>>> those you are generally in favor of this proposal), then maybe we should
>>> discuss exactly this point. We haven't really discussed alternatives
>>> about
>>> the release alignment. This vote feels rushed.
>>>
>>>
>> It's been discussed for a week, and I'm with Mark - I'm only for a
>> real merge of development, and that includes release schedules.
>>
>> -Yonik
>>
>>
>>
>
> How will we merge release policies? (or are they exactly the same?) Does
> Solr use the same release numbering? Does it have the same
> backwards-compatibility requirements as Lucene?
>
> If we release Solr 1.5.0 with Lucene 3.1.0, and we find a bug in the Lucene
> jar and want to release a 3.1.1 bugfix release, will we also release a Solr
> 1.5.1, even though all Solr jars would be identical to the 1.5.0?
> Or will we just release Solr/Lucene 4.0.0 next and always use same release
> numbers?
>
> How will we avoid longer release cycles? Solr had had very infrequent
> releases. What were the reasons for that? Are we comfortable with saying
> we'll just try to be disciplined enough or is there a way to somehow enforce
> release frequency? It seems certain that if more people work on the code,
> there will at any given point be more patches/new features under development
> and things need to be coordinated in a way that allows frequent releases.
>
> In an earlier mail I gave the following example: If we had a separate
> analyzer module, and we add an analyzer in a new language to it, it would be
> cool to release it soon, without having to wait until Lucene/Solr are ready
> for a release. The pace here can be faster, because I imagine in such an
> Analyzer module it's much less common that a patch "touches everything".
> What do people think about this? Maybe it's just a nice wish, but not
> realizable, because there'd be a lot of version management overhead. But
> maybe not?
>
> I'm not against this whole thing and I'm not trying to be difficult here,
> and I dislike endless discussions just as everyone else. But I honestly
> don't know right now how to vote, because I have those open questions and
> know that a lot of other people here are concerned about the release
> alignment as well.
>
>  Michael
>
>

Hi,

-1 on the current VOTE, as I am thinking the same like Michael Busch and Bill Au:

- I am fine with merging development mailing lists (not user mailing lists).
- But I do not want to enforce releases to appear at the same time, so there must be some coordination with the fact that "Solr depends on Lucene but NOT Lucene depends on Solr".
- A modularization is needed: Lucene-Core (with no analyzers at all, only abstract classes), Lucene-Analysis, Lucene-Facetting, Lucene-FunctionQueries, Lucene-Foobar, Solr-Core, Solr-Foobar,...
- No requirement for Lucene Committers to work on Solr Tests or that Solr tests must pass when Lucene Changes. I would like to have it more in a way that the issue tracker would do that like it is now: Lucene is enhanced, BW layer still alive (so solr tests should work), so open issue against solr referring to lucene issue to fix solr and remove usage of deprecated methods or fix other problems.
- And last but not least the whole merge should be done *after* the current code bases are again closer to each other, especially Flex is in and Solr is at least on Lucene 3.0.1.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Michael Busch [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, March 04, 2010 4:13 AM
> To: [EMAIL PROTECTED]
> Subject: Re: [VOTE] merge lucene/solr development
>
> On 3/3/10 6:00 PM, Yonik Seeley wrote:
> > On Wed, Mar 3, 2010 at 8:10 PM, Michael Busch<[EMAIL PROTECTED]>
> wrote:
> >
> >> So if it seems like that most people are concerned about releases
> (even
> >> those you are generally in favor of this proposal), then maybe we
> should
> >> discuss exactly this point. We haven't really discussed alternatives
> about
> >> the release alignment. This vote feels rushed.
> >>
> > It's been discussed for a week, and I'm with Mark - I'm only for a
> > real merge of development, and that includes release schedules.
> >
> > -Yonik
> >
> >
>
> How will we merge release policies? (or are they exactly the same?)
> Does
> Solr use the same release numbering? Does it have the same
> backwards-compatibility requirements as Lucene?
>
> If we release Solr 1.5.0 with Lucene 3.1.0, and we find a bug in the
> Lucene jar and want to release a 3.1.1 bugfix release, will we also
> release a Solr 1.5.1, even though all Solr jars would be identical to
> the 1.5.0?
> Or will we just release Solr/Lucene 4.0.0 next and always use same
> release numbers?
>
> How will we avoid longer release cycles? Solr had had very infrequent
> releases. What were the reasons for that? Are we comfortable with
> saying
> we'll just try to be disciplined enough or is there a way to somehow
> enforce release frequency? It seems certain that if more people work on
> the code, there will at any given point be more patches/new features
> under development and things need to be coordinated in a way that
> allows
> frequent releases.
>
> In an earlier mail I gave the following example: If we had a separate
> analyzer module, and we add an analyzer in a new language to it, it
> would be cool to release it soon, without having to wait until
> Lucene/Solr are ready for a release. The pace here can be faster,
> because I imagine in such an Analyzer module it's much less common that
> a patch "touches everything". What do people think about this? Maybe
> it's just a nice wish, but not realizable, because there'd be a lot of
> version management overhead. But maybe not?
>
> I'm not against this whole thing and I'm not trying to be difficult
> here, and I dislike endless discussions just as everyone else. But I
> honestly don't know right now how to vote, because I have those open
> questions and know that a lot of other people here are concerned about
> the release alignment as well.
>
>   Michael

On Thu, Mar 4, 2010 at 2:19 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> Hi,
>
> -1 on the current VOTE, as I am thinking the same like Michael Busch and Bill Au:
>
> - I am fine with merging development mailing lists (not user mailing
>   lists).

OK.

> - But I do not want to enforce releases to appear at the same time,
>   so there must be some coordination with the fact that "Solr
>   depends on Lucene but NOT Lucene depends on Solr".

Maybe we could release Lucene but not necessarily Solr?  EG say with
segment-based search, we cutover, and Solr tests all pass (because the
change was technically back-compat), but Solr has poor performance
because it still sometimes searches at the MultiReader level, yet,
Lucene is stable.  So we release Lucene, but only release Solr once it has
more fully cutover?  Would this be possible?

Likewise in the analyzers example -- if contrib/analyzers why not
separately release it as important bug fixes happen?

I'm not saying we'd do this across the board for all modules, but it's
a possibility.

Also, remember that as a combined dev community, we all want what's
best for users (faster releases), so we all can brainstorm together
about how to make that possible.  I've not heard anybody wanting
slower releases.  This is why I see release frequency as really a
matter of discipline.  If we will take it seriously, we should appoint
a release czar on each cycle.

> - A modularization is needed: Lucene-Core (with no analyzers at all,
>   only abstract classes), Lucene-Analysis, Lucene-Facetting,
>   Lucene-FunctionQueries, Lucene-Foobar, Solr-Core, Solr-Foobar,...

I'm a huge fan of doing this as well, but, I see merging Solr/Lucene
dev as a huge step towards making this modularization possible.

Maybe we can do this up front, when we merge?  EG moving concrete
queries, analyzers out of core (keeping abstract base classes), moving
queryParser out, seem like obvious first steps.  Hey then Lucene's JAR
will be less than 1.0 MB again!

> - No requirement for Lucene Committers to work on Solr Tests or that
>   Solr tests must pass when Lucene Changes. I would like to have it
>   more in a way that the issue tracker would do that like it is now:
>   Lucene is enhanced, BW layer still alive (so solr tests should
>   work), so open issue against solr referring to lucene issue to fix
>   solr and remove usage of deprecated methods or fix other problems.

First off, not breaking tests seems like a simple win?  Ie, if we're
not breaking back compat, the tests all pass.  If we accidentally
break back compat, the tests fail, and protect us, which is only good?

I think your real concern might be that you (and other "I want to
focus only on core Lucene devs") might be forced to spend alot of time
working on Solr when new changes happen in Lucene?

I would argue that this won't happen.  Or, it will only happen if you
want to work on Solr.  Think about the segment based search example --
Lucene can cutover to segment based search, pass all tests.  But then
a separate issue would be opened and likely a separate person (say
Yonik) would work on it, to have Solr properly take advantage of /
expose the new feature.

Now some features will be driven by someone wearing a Solr hat.  EG
Chris/Grant's (& other's) push to get spatial working well with Solr.
If this push were done in with shared development then it would be
created/iterated in both a Solr and Lucene friendly way, with a single
source.

And take our work in flex -- if we're doing our jobs, landing flex will
already pass all Solr tests.  I *really* want to have Solr tests confirm
that we didn't break back-compat.  But I can't, now (Solr's not on trunk).

Does that mean I must go and fix Solr to make it possible to specify a
custom codec through schema.xml?  No.  Exposing flex in Solr, NRT in
Solr, is always going to be a separate step.

The combined dev community would have no requirement/expectation that
if someone adds something cool to Lucene they must also expose it in
Solr.  There will still be devs that wear mostly Solr vs most Lucene
hats.  There will also be devs that comfortably wear both.  There will
be devs that focus on analyzers and do amazing things ;)
Well, this really is a logistical question (ie not really a reason for
casting a -1 vote).

But, yes, we have to get Solr upgraded to Lucene trunk before we can
merge development -- there's no way around that.

On requiring flex to land first... I'm now agnostic.  I suspect landing
flex won't be that much harder before vs after.  So if we can't finish
up flex in time (I have ~88 nocommits left ;), I don't think that
should block merging Solr/Lucene dev.

Mike

> > - A modularization is needed: Lucene-Core (with no analyzers at all,
> >   only abstract classes), Lucene-Analysis, Lucene-Facetting,
> >   Lucene-FunctionQueries, Lucene-Foobar, Solr-Core, Solr-Foobar,...
>
> I'm a huge fan of doing this as well, but, I see merging Solr/Lucene
> dev as a huge step towards making this modularization possible.
>
> Maybe we can do this up front, when we merge?  EG moving concrete
> queries, analyzers out of core (keeping abstract base classes), moving
> queryParser out, seem like obvious first steps.  Hey then Lucene's JAR
> will be less than 1.0 MB again!

Yeah, QueryParser should go away from core (foget that).

> > - No requirement for Lucene Committers to work on Solr Tests or that
> >   Solr tests must pass when Lucene Changes. I would like to have it
> >   more in a way that the issue tracker would do that like it is now:
> >   Lucene is enhanced, BW layer still alive (so solr tests should
> >   work), so open issue against solr referring to lucene issue to fix
> >   solr and remove usage of deprecated methods or fix other problems.
>
> First off, not breaking tests seems like a simple win?  Ie, if we're
> not breaking back compat, the tests all pass.  If we accidentally
> break back compat, the tests fail, and protect us, which is only good?
>
> I think your real concern might be that you (and other "I want to
> focus only on core Lucene devs") might be forced to spend alot of time
> working on Solr when new changes happen in Lucene?

I am fine with fixing bugs in solr that are there before the change but only appear because of the change. My problem is more such things like the per-segment-mega-problem because Solr was simply using Lucene incorrectly (I hope this is said too hard). We did not break backwards. But if we would had to repair the whole solr (which is still not finished) after the per-segment change, we were still not having per-segment search. And fixing this is surely not easy possible for non-solrcore developers like me or you. So even if development goes together we should still have the possibility to update lucene and if its not a backwards break but incorrect usage of Lucene's API (or assumptions on behavior of Lucene's API that are not documented or are obvious from API design - like for Filters have never to work on Top-Level Searchers and *only* use the passed in IndexReader), I would simply break solr and let the solr devs fix it in a separate issue.

> And take our work in flex -- if we're doing our jobs, landing flex will
> already pass all Solr tests.  I *really* want to have Solr tests
> confirm
> that we didn't break back-compat.  But I can't, now (Solr's not on
> trunk).

OK

> > - And last but not least the whole merge should be done *after* the
> >   current code bases are again closer to each other, especially Flex
> >   is in and Solr is at least on Lucene 3.0.1.
>
> Well, this really is a logistical question (ie not really a reason for
> casting a -1 vote).

The -1 was for the vote in general as it is not clear about what we are voting.

Uwe

On Thu, Mar 4, 2010 at 5:51 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:

>> > - And last but not least the whole merge should be done *after* the
>> >   current code bases are again closer to each other, especially Flex
>> >   is in and Solr is at least on Lucene 3.0.1.
>>
>> Well, this really is a logistical question (ie not really a reason for
>> casting a -1 vote).
>
> The -1 was for the vote in general as it is not clear about what we
> are voting.

We are voting on this:

 * Merging the dev lists into a single list.

 * Merging committers.

 * When any change is committed (to a module that "belongs to" Solr or
   to Lucene), all tests must pass

 * Release both at once (but the specific logistics is still up for
   discussion)

 * Modularlize the sources: pull things out of Lucene's core (break
   out query parser, move all core queries & analyzers under their
   contrib counterparts), pull things out of Solr's core (analyzers,
   queries)

These things would not change:

 * Besides modularizing (above), the source code would remain factored
   into separate dirs/modules the way it is now.

 * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX issues)

 * User's lists remain separate.

 * Web sites remain separate.

 * Release artifacts/jars remain separate

> I am fine with fixing bugs in solr that are there before the change
> but only appear because of the change.

OK

> My problem is more such things like the per-segment-mega-problem
> because Solr was simply using Lucene incorrectly (I hope this is
> said too hard).

You know, Lucene also used those APIs "incorrectly" until we cutover
Lucene's search to be per-segment ;)  We "got lucky" in that the APIs
were at best "ambiguous" about whether the incoming reader was
per-segment or not.

> We did not break backwards.

Right, and so Solr's tests should have passed.

> But if we would had to repair the whole solr (which is still not
> finished) after the per-segment change, we were still not having
> per-segment search.

But you won't have to fix Solr from such a change.  Others (people
wearing Solr hats) will.

> And fixing this is surely not easy possible for non-solrcore
> developers like me or you.

Right.

> So even if development goes together we should still have the
> possibility to update lucene and if its not a backwards break but
> incorrect usage of Lucene's API (or assumptions on behavior of
> Lucene's API that are not documented or are obvious from API design
> - like for Filters have never to work on Top-Level Searchers and
> *only* use the passed in IndexReader), I would simply break solr and
> let the solr devs fix it in a separate issue.

There would no longer be solr devs -- just "devs" who sometimes wear
Solr hats, sometimes wear Lucene hats, sometimes both, at different
times.

Uwe, this is in fact the proposal -- you can break Solr (but you must
pass its tests), and devs with Solr hats will fix it.  It's a separate
issue.

Mike

On Thu, Mar 4, 2010 at 2:40 PM, Michael McCandless
<[EMAIL PROTECTED]> wrote:
> On Thu, Mar 4, 2010 at 5:51 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
>
>>> > - And last but not least the whole merge should be done *after* the
>>> >   current code bases are again closer to each other, especially Flex
>>> >   is in and Solr is at least on Lucene 3.0.1.
>>>
>>> Well, this really is a logistical question (ie not really a reason for
>>> casting a -1 vote).
>>
>> The -1 was for the vote in general as it is not clear about what we
>> are voting.
>
> We are voting on this:
>
>  * Merging the dev lists into a single list.
>
>  * Merging committers.
>
>  * When any change is committed (to a module that "belongs to" Solr or
>   to Lucene), all tests must pass

I don't know why this is such a big problem except of the consumed
time by running the tests of both. Yet, for a lucene committer it
takes about 5-8 min to run all the test, no idea how long solr tests
take. Aside of this it would give me a way better feeling to commit
changes if all those test pass. BWCompat policy will force you to not
break stuff in a large scale and the solr tests are a good judge for
that!
>
>  * Release both at once (but the specific logistics is still up for
>   discussion)
>
>  * Modularlize the sources: pull things out of Lucene's core (break
>   out query parser, move all core queries & analyzers under their
>   contrib counterparts), pull things out of Solr's core (analyzers,
>   queries)
That is one of the things I would love to see. I worked with lucene on
mobile devices and each 1kb is valuable if you just need a small
subset of lucene. The core should be as small as possible IMO.
>
> These things would not change:
>
>  * Besides modularizing (above), the source code would remain factored
>   into separate dirs/modules the way it is now.
>
>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX issues)
>
>  * User's lists remain separate.
>
>  * Web sites remain separate.
>
>  * Release artifacts/jars remain separate
>
>> I am fine with fixing bugs in solr that are there before the change
>> but only appear because of the change.
>
> OK
>
>> My problem is more such things like the per-segment-mega-problem
>> because Solr was simply using Lucene incorrectly (I hope this is
>> said too hard).
>
> You know, Lucene also used those APIs "incorrectly" until we cutover
> Lucene's search to be per-segment ;)  We "got lucky" in that the APIs
> were at best "ambiguous" about whether the incoming reader was
> per-segment or not.
>
>> We did not break backwards.
>
> Right, and so Solr's tests should have passed.
Absolutely true!
>
>> But if we would had to repair the whole solr (which is still not
>> finished) after the per-segment change, we were still not having
>> per-segment search.
>
> But you won't have to fix Solr from such a change.  Others (people
> wearing Solr hats) will.
>
>> And fixing this is surely not easy possible for non-solrcore
>> developers like me or you.
>
> Right.
I agree but with solr we will have a whole bunch of guys who will be able too
>
>> So even if development goes together we should still have the
>> possibility to update lucene and if its not a backwards break but
>> incorrect usage of Lucene's API (or assumptions on behavior of
>> Lucene's API that are not documented or are obvious from API design
>> - like for Filters have never to work on Top-Level Searchers and
>> *only* use the passed in IndexReader), I would simply break solr and
>> let the solr devs fix it in a separate issue.
>
> There would no longer be solr devs -- just "devs" who sometimes wear
> Solr hats, sometimes wear Lucene hats, sometimes both, at different
> times.
>
> Uwe, this is in fact the proposal -- you can break Solr (but you must
> pass its tests), and devs with Solr hats will fix it.  It's a separate
> issue.
>
> Mike
>

Are we still voting or is this already a discussion?
If we are voting, +1 from me!
Simon

+1

On Wed, Mar 3, 2010 at 5:42 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>

On Thu, Mar 4, 2010 at 7:10 PM, Michael McCandless <
[EMAIL PROTECTED]> wrote:

>
> We are voting on this:
>
>  * Merging the dev lists into a single list.
>
>  * Merging committers.
>
>  * When any change is committed (to a module that "belongs to" Solr or
>   to Lucene), all tests must pass
>
>  * Release both at once (but the specific logistics is still up for
>   discussion)
>
>  * Modularlize the sources: pull things out of Lucene's core (break
>   out query parser, move all core queries & analyzers under their
>   contrib counterparts), pull things out of Solr's core (analyzers,
>   queries)
>
> These things would not change:
>
>  * Besides modularizing (above), the source code would remain factored
>   into separate dirs/modules the way it is now.
>
>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX issues)
>
>  * User's lists remain separate.
>
>  * Web sites remain separate.
>
>  * Release artifacts/jars remain separate
>
>
+1

--
Regards,
Shalin Shekhar Mangar.

On 03/04/2010 09:21 AM, Simon Willnauer wrote:
> Are we still voting or is this already a discussion?
>    
As far as I'm concerned, a vote is on - the vote was started and people
are voting.
If the majority vote +1, the vote would still conceivably pass - if
people think its not
right to vote now and their should be more discussion, I'd assume they
would vote -1
on the current vote, and perhaps we would have another after further
discussion.
But while there can still be discussion as people decide which way to
vote, there is
def a vote occurring.

--
- Mark

http://www.lucidimagination.com

Hi,

On Thu, Mar 4, 2010 at 3:33 PM, Mark Miller <[EMAIL PROTECTED]> wrote:
> On 03/04/2010 09:21 AM, Simon Willnauer wrote:
>> Are we still voting or is this already a discussion?
>
> As far as I'm concerned, a vote is on - the vote was started and people are
> voting.

I haven't followed this too closely, but the large amount of confusion
seems to indicate that calling a vote on this may be premature.

As an outside observer (I'm not actively involved in Solr or Lucene
Java development) this proposal seems like a pretty monumental change
for the projects. Would it make sense to rather start with smaller
steps that everyone can agree on?

BR,

Jukka Zitting

On 2010-03-03 23:42, Yonik Seeley wrote:
> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>
>

Tentative +1, if the consensus from the discussion is written down
somewhere as "bylaws" that we can refer to when solving unexpected
complications resulting from this merge (project mgmt, committer access,
test reqs, rolling out releases etc...)
--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Hi All,

I'll preface my vote in this by mentioning I'm neither a Lucene(-java) or
Solr committer, but I am -1 for this proposal is in its current form, per my
comments referenced in the below URL during the existing discussions.

Cheers,
Chris

On 3/3/10 3:42 PM, "Yonik Seeley" <[EMAIL PROTECTED]> wrote:

> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out
> _a_standalone_shared_analysis_package_for_nutch_solr_lucene
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On 03/04/2010 10:30 AM, Jukka Zitting wrote:
> Hi,
>
> On Thu, Mar 4, 2010 at 3:33 PM, Mark Miller<[EMAIL PROTECTED]>  wrote:
>    
>> On 03/04/2010 09:21 AM, Simon Willnauer wrote:
>>      
>>> Are we still voting or is this already a discussion?
>>>        
>> As far as I'm concerned, a vote is on - the vote was started and people are
>> voting.
>>      
> I haven't followed this too closely, but the large amount of confusion
> seems to indicate that calling a vote on this may be premature.
>
> As an outside observer (I'm not actively involved in Solr or Lucene
> Java development) this proposal seems like a pretty monumental change
> for the projects. Would it make sense to rather start with smaller
> steps that everyone can agree on?
>
> BR,
>
> Jukka Zitting
>    
Hey Jukka - I can see your point for sure -

but with more people voting than saying they are confused, that seems to
say something ?

Since Yonik started the vote, I'd defer to him on all this - its only my
opinion.

But it seems that anyone could argue any vote is too monumental or
confusing at any time. But if the majority of people
are not confused and are voting (most now voting for the merge), I'm not
sure it makes sense to say the vote is premature?

If 90%+ of the committers are voting +1 so far, I'm not sure that says
we should start with smaller steps either?

My simple opinion though.

--
- Mark

http://www.lucidimagination.com

On Wed, Mar 03, 2010 at 06:54:33PM -0500, Mark Miller wrote:
> (though many responders appeared to be confused as to what most of us think
> the merge means, so who knows)

I went back to the post where Yonik broached the issue to try to understand
where my confusion arose.

I read this passage and took away the notion that Lucene would be absorbed
into Solr:

    Some in Lucene development have expressed a desire to make Lucene more
    of a complete solution, rather than just a core full-text search
    library... things like a data schema, faceting, etc.  The Lucene
    project already has an enterprise search platform with these
    features... that's Solr.  Trying to pull popular pieces out of Solr
    makes life harder for Solr developers, brings our projects into
    conflict, and is often unsuccessful (witness the largely failed
    migration of FunctionQueries from Solr to Lucene).  For Lucene to
    achieve the ultimate in usability for users, it can't require Java
    experience... it needs higher level abstractions provided by Solr.
    
    The other benefit to Lucene would be to bring features to developers
    much sooner... Solr has had features years before they were developed
    in Lucene, and currently has more developers working with it.  Esp
    with Solr not using Lucene trunk, if a Solr developer wants a feature
    quickly, they cannot add it to Lucene (even if it might make sense
    there) since that introduces a big unpredictable lag - when that
    version of Lucene make it's way into Solr.
    
    The current divide is a bit unnatural.  For maximum benefit of both
    projects, it seems like Solr and Lucene should essentially merge.

Yonik's email was followed up shortly by this, from Mike McCandless:

    I think this is a good idea!  LuSolr ;) (kidding)

I figured Mike was "kidding" about the name but +1 on absorbing.  Sigh.

Ah, but here's the sentence I missed from Yonik's original email:

    Lucene core would essentially remain as it is, but:

Feh.  Sloppy reading on my part.  Still, I don't think it's hard to understand
why some of us got the wrong idea.

Marvin Humphrey

On Wed, Mar 03, 2010 at 06:54:33PM -0500, Mark Miller wrote:

> >  (though many responders appeared to be confused as to what most of us think
> >  the merge means, so who knows)
>    
On 03/04/2010 10:43 AM, Marvin Humphrey wrote:
>
> Feh.  Sloppy reading on my part.  Still, I don't think it's hard to understand
> why some of us got the wrong idea.
>
>
>    

Heh - heck no - I certainly didn't mean it that way. I can easily see
why people would be confused with that long thread. Didn't mean to imply
there is no way they should be.

But I think to those of us that are pro merge, it's just very clear that
Lucene would have to remain a completely separate library. I'd bet my
life savings that none of us would even remotely support Solr gobbling
up Lucene. This is only a merge on the dev side - the consumer side
would remain relatively the same.

--
- Mark

http://www.lucidimagination.com

+1, if we are voting on this:

Michael McCandless wrote:

 > We are voting on this:
 >
 >  * Merging the dev lists into a single list.
 >
 >  * Merging committers.
 >
 >  * When any change is committed (to a module that "belongs to" Solr or
 >    to Lucene), all tests must pass
 >
 >  * Release both at once (but the specific logistics is still up for
 >    discussion)
 >
 >  * Modularlize the sources: pull things out of Lucene's core (break
 >    out query parser, move all core queries & analyzers under their
 >    contrib counterparts), pull things out of Solr's core (analyzers,
 >    queries)
 >
 > These things would not change:
 >
 >  * Besides modularizing (above), the source code would remain factored
 >    into separate dirs/modules the way it is now.
 >
 >  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX issues)
 >
 >  * User's lists remain separate.
 >
 >  * Web sites remain separate.
 >
 >  * Release artifacts/jars remain separate

Koji

--
http://www.rondhuit.com/en/

If we vote on what Mike says, I revise my vote and simply vote +/-0 to not stop progress. I have some problem with the construct but in general I am fine with merging dev lists, splitting into modules, merged committers - but not the requirement on tests pass always. In my opinion, if anything changed in lucene breaks some tests we could open an issue in solr.

One idea: If we really make solr depend on the new lucene lib, solr should not have lucene jars in its lib folder, but instead the nightly build should fetch the jars from the lucene hudson build. For committers working in svn, maybe some relation to rev numbers (like we do for lucene backwards tests) can be put into solrs common-build.xml so the ant script of solr can checkout the correct lucene rev and build it on they fly.

Uwe

> We are voting on this:
>
>  * Merging the dev lists into a single list.
>
>  * Merging committers.
>
>  * When any change is committed (to a module that "belongs to" Solr or
>    to Lucene), all tests must pass
>
>  * Release both at once (but the specific logistics is still up for
>    discussion)
>
>  * Modularlize the sources: pull things out of Lucene's core (break
>    out query parser, move all core queries & analyzers under their
>    contrib counterparts), pull things out of Solr's core (analyzers,
>    queries)
>
> These things would not change:
>
>  * Besides modularizing (above), the source code would remain factored
>    into separate dirs/modules the way it is now.
>
>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX issues)
>
>  * User's lists remain separate.
>
>  * Web sites remain separate.
>
>  * Release artifacts/jars remain separate
>
> > I am fine with fixing bugs in solr that are there before the change
> > but only appear because of the change.
>
> OK
>
> > My problem is more such things like the per-segment-mega-problem
> > because Solr was simply using Lucene incorrectly (I hope this is
> > said too hard).
>
> You know, Lucene also used those APIs "incorrectly" until we cutover
> Lucene's search to be per-segment ;)  We "got lucky" in that the APIs
> were at best "ambiguous" about whether the incoming reader was
> per-segment or not.
>
> > We did not break backwards.
>
> Right, and so Solr's tests should have passed.
>
> > But if we would had to repair the whole solr (which is still not
> > finished) after the per-segment change, we were still not having
> > per-segment search.
>
> But you won't have to fix Solr from such a change.  Others (people
> wearing Solr hats) will.
>
> > And fixing this is surely not easy possible for non-solrcore
> > developers like me or you.
>
> Right.
>
> > So even if development goes together we should still have the
> > possibility to update lucene and if its not a backwards break but
> > incorrect usage of Lucene's API (or assumptions on behavior of
> > Lucene's API that are not documented or are obvious from API design
> > - like for Filters have never to work on Top-Level Searchers and
> > *only* use the passed in IndexReader), I would simply break solr and
> > let the solr devs fix it in a separate issue.
>
> There would no longer be solr devs -- just "devs" who sometimes wear
> Solr hats, sometimes wear Lucene hats, sometimes both, at different
> times.
>
> Uwe, this is in fact the proposal -- you can break Solr (but you must
> pass its tests), and devs with Solr hats will fix it.  It's a separate
> issue.
>
> Mike

On Mar 4, 2010, at 7:30 AM, Jukka Zitting wrote:

> Hi,
>
> On Thu, Mar 4, 2010 at 3:33 PM, Mark Miller <[EMAIL PROTECTED]> wrote:
>> On 03/04/2010 09:21 AM, Simon Willnauer wrote:
>>> Are we still voting or is this already a discussion?
>>
>> As far as I'm concerned, a vote is on - the vote was started and people are
>> voting.
>
> I haven't followed this too closely, but the large amount of confusion
> seems to indicate that calling a vote on this may be premature.
>
> As an outside observer (I'm not actively involved in Solr or Lucene
> Java development) this proposal seems like a pretty monumental change
> for the projects. Would it make sense to rather start with smaller
> steps that everyone can agree on?
>

I don't see that smaller steps would work.  For instance, I'm -1 on just the analyzers move, b/c I don't think it will work in practice and it is just something that will end up gutting Solr over time as more things are moved out.  It will require Solr to always have to be exactly in sync with the same versions of Lucene otherwise there will never be agreement on releasing the Analyzer artifact between Lucene and Solr committers, which is what this bigger vote is about.

My only concern with the current vote is the lockstep releases, but I don't know if that is actually something that has to be voted on now anyway.  I don't see any benefit from deciding that now.  I suspect that the committers can work that out when the time comes.

Taking a step back, my view as PMC Chair and given the Board's penchant for "no umbrella" projects, I also think the merge makes sense at the ASF level.  Mahout will be spinning out to a TLP after 0.3 (assuming the board approves it) and I think Tika likely makes sense as a TLP too.  One could make a case for Nutch being spun out due to it being more focused on crawling these days (but I'm not pushing it either, as it still fits well in Lucene IMO too), but I think Solr, Lucene and the Ports are intimately connected and are solely focused on one thing: search.  This, in my mind, is a nicely focused community with a few subprojects with a very large set of overlapping committers, a well focused PMC and an aligned community.   At the same time, the PMC needs to be respectful of the very large community of users who have already built Lucene implementations.  I think this proposal does that.

So, in the end, +1 on the vote, w/ the exception of the lockstep releases, which I think we should defer to the community once it's fully formed to work out the details.

-Grant

On Mar 4, 2010, at 8:19 AM, Uwe Schindler wrote:

> If we vote on what Mike says, I revise my vote and simply vote +/-0 to not stop progress. I have some problem with the construct but in general I am fine with merging dev lists, splitting into modules, merged committers - but not the requirement on tests pass always. In my opinion, if anything changed in lucene breaks some tests we could open an issue in solr.

I'm not sure why this is such a big deal to you, Uwe.  Just as when tests break today w/in Lucene, say in a contrib module that you don't personally spend time on, other committers step up to help and get it fixed and you work with them to see it through.  That's how the community works.  Why should it be any different after this merge?  No Lucene/Solr committer, upon seeing you broke something in Solr is going to let those things stay broken, just like how you and Michael worked to get all the new Analyzer stuff working in 2.9.   I also suspect that you, being the conscientious developer you are, will either a) make sure someone w/ the appropriate expertise is made aware of it and work with them to fix it or b) you'll figure it out yourself and fix it too, thus adding to your skillset and making you a better developer.  In fact, you've already done this on a number of occasions across the Lucene/Solr boundary as it is now (Trie fields, analyzers, etc.)

-Grant

On Thu, 4 Mar 2010, Michael McCandless wrote:

> We are voting on this:
>
> * Merging the dev lists into a single list.
>
> * Merging committers.
>
> * When any change is committed (to a module that "belongs to" Solr or
>   to Lucene), all tests must pass
>
> * Release both at once (but the specific logistics is still up for
>   discussion)
>
> * Modularlize the sources: pull things out of Lucene's core (break
>   out query parser, move all core queries & analyzers under their
>   contrib counterparts), pull things out of Solr's core (analyzers,
>   queries)
>
> These things would not change:
>
> * Besides modularizing (above), the source code would remain factored
>   into separate dirs/modules the way it is now.
>
> * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX issues)
>
> * User's lists remain separate.
>
> * Web sites remain separate.
>
> * Release artifacts/jars remain separate

+1 with the following amendment:
   - A Solr release implies a Lucene release but the other way around.

Andi..

+1
Great idea! :-)

-Yonik

On Wed, Mar 3, 2010 at 5:42 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene
>

On 03/04/2010 11:24 AM, Grant Ingersoll wrote:
>
> So, in the end, +1 on the vote, w/ the exception of the lockstep releases, which I think we should defer to the community once it's fully formed to work out the details.
>
> -Grant

I don't think you can really cherry pick what you +1 on though. Like I
said, I'm not +1 without sync'd releases - I'm open to how they happen
and the specifics, but my vote is only valid for the full proposal. Some
sort of sync'd releasing is a linchpin for me.

--
- Mark

http://www.lucidimagination.com

-1

This would be a serious mistake.
- Neal

-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Yonik Seeley
Sent: Wednesday, March 03, 2010 4:43 PM
To: [EMAIL PROTECTED]
Subject: [VOTE] merge lucene/solr development

Many Lucene/Solr committers think that merging development would be a
benefit to both projects.
Separate downloads would remain (among other things), so end users
would not be impacted (except for higher quality products over time).
Since this is a change to Lucene/Solr project development, I'd like to
get a format vote from the committers of both projects.
If there are 3 +1s and more +1s than -1s, we can pass this to the
Lucene PMC to ratify.

-Yonik

Discussion thread:
http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene

For those trying to keep track of the current voting, here is a list of
the Lucene/Solr committers and what I have seen of their votes thus far:

Bill Au
Doug Cutting
Otis Gospodnetić
Erik Hatcher
Chris Hostetter
Grant Ingersoll - +1*
Mike Klaas
Shalin Shekhar Mangar - +1
Ryan McKinley
Mark Miller - +1
Noble Paul - +1
Yonik Seeley - +1
Koji Sekiguchi - +1
Michael Busch
Doron Cohen
Mike McCandless - +1
Bernhard Messer
Robert Muir - +1
Uwe Schindler - +/-0
Wolfgang Hoschek
Patrick O'Leary
Andi Vajda - +1*
Karl Wettin
Simon Willnauer - +1

* vote with some kind of clause (is that legal :) )

--
- Mark

http://www.lucidimagination.com

: Subject: RE: [VOTE] merge lucene/solr development

-1

If this is the direction the dev community wants to go in then so be it --
FWIW: My current opinion is that this direction does make sense in teh
long run -- but the vote as it stands feels way to broad.  It seems like
people are trying vote on a "goal" instead of on specific actions, and
treating that goal as a driver to make several changes all at the same
time.

i would much rather attempt specific changes first (individually when
possible), and see how we progress.

As Uwe says alludes: merging releases, and trying to enforce that changes
can't be made to core that might break Solr are things that could prove
very challenging, particularly in the "next" release, and I don't see why
we should try to make all of this happen at once.

Why don't we just start by attempting to have a common dev list and
merging committers, in the hopes that it will promote better communication
about features up and down the stack, and better bug
fixing/refactoring/modularization -- then see if that leads us to a point
where it makes sense to more tightly couple the build systems and
releases?

: -1 on the current VOTE, as I am thinking the same like Michael Busch and Bill Au:
:
: - I am fine with merging development mailing lists (not user mailing lists).
: - But I do not want to enforce releases to appear at the same time, so there must be some coordination with the fact that "Solr depends on Lucene but NOT Lucene depends on Solr".
: - A modularization is needed: Lucene-Core (with no analyzers at all, only abstract classes), Lucene-Analysis, Lucene-Facetting, Lucene-FunctionQueries, Lucene-Foobar, Solr-Core, Solr-Foobar,...
: - No requirement for Lucene Committers to work on Solr Tests or that Solr tests must pass when Lucene Changes. I would like to have it more in a way that the issue tracker would do that like it is now: Lucene is enhanced, BW layer still alive (so solr tests should work), so open issue against solr referring to lucene issue to fix solr and remove usage of deprecated methods or fix other problems.
: - And last but not least the whole merge should be done *after* the current code bases are again closer to each other, especially Flex is in and Solr is at least on Lucene 3.0.1.
-Hoss

+1

this is software.  let's try it.  if it doesn't work out, we know what to do.

Otis

----- Original Message ----
> From: Yonik Seeley <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wed, March 3, 2010 5:42:38 PM
> Subject: [VOTE] merge lucene/solr development
>
> Many Lucene/Solr committers think that merging development would be a
> benefit to both projects.
> Separate downloads would remain (among other things), so end users
> would not be impacted (except for higher quality products over time).
> Since this is a change to Lucene/Solr project development, I'd like to
> get a format vote from the committers of both projects.
> If there are 3 +1s and more +1s than -1s, we can pass this to the
> Lucene PMC to ratify.
>
> -Yonik
>
> Discussion thread:
> http://search.lucidimagination.com/search/document/c7817932400808ad/factor_out_a_standalone_shared_analysis_package_for_nutch_solr_lucene

Extending this with a few more specific steps:

a) merge dev list and committers

b) have lucene build trigger solr build and test

c) have lucene release trigger solr release shortly afterwards

d) completely synchronize builds

If these steps (as steps, not as a bundle) are what the proposal really is
about, then it might win over some of the negative votes.  Surely there is
value in the first two steps and little controversy.  Surely also (c) is in
the hands of the solr-focused devs and would be a natural outgrowth of (b).
If (d) winds up in the far future because people are happy, then how can
that be objectionable.

A phrase I haven't heard yet is "progress, not perfection".    I thought
that was the Lucene development motto.   Can't it be applied here?
On Thu, Mar 4, 2010 at 9:41 AM, Chris Hostetter <[EMAIL PROTECTED]>wrote:

> Why don't we just start by attempting to have a common dev list and
> merging committers, in the hopes that it will promote better communication
> about features up and down the stack, and better bug
> fixing/refactoring/modularization -- then see if that leads us to a point
> where it makes sense to more tightly couple the build systems and
> releases?
>

--
Ted Dunning, CTO
DeepDyve

On Thu, Mar 4, 2010 at 12:41 PM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:

> Why don't we just start by attempting to have a common dev list and
> merging committers, in the hopes that it will promote better
> communication about features up and down the stack, and better bug
> fixing/refactoring/modularization -- then see if that leads us to a
> point where it makes sense to more tightly couple the build systems
> and releases?

I'm all for being iterative / taking baby steps, when the problem
naturally can be solved that way -- progress not perfection.  Many
problems decompose like this.

But I don't think this problem does.

In particular, how would the above baby step address the code
duplication (my goal in the original opening) -- eg the 3 places where
concrete analyzers/queries are today.  How would it lead to making
facets work with pure Lucene?  To developing spatial in one place?

Mike

On Thu, Mar 4, 2010 at 12:43 PM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
> +1
>
> this is software.  let's try it.  if it doesn't work out, we know what to do.

+1 ;)

"it's just software" is another quote I love ...

Mike

On 03/04/2010 01:04 PM, Michael McCandless wrote:
> On Thu, Mar 4, 2010 at 12:41 PM, Chris Hostetter
> <[EMAIL PROTECTED]>  wrote:
>
>    
>> Why don't we just start by attempting to have a common dev list and
>> merging committers, in the hopes that it will promote better
>> communication about features up and down the stack, and better bug
>> fixing/refactoring/modularization -- then see if that leads us to a
>> point where it makes sense to more tightly couple the build systems
>> and releases?
>>      
> I'm all for being iterative / taking baby steps, when the problem
> naturally can be solved that way -- progress not perfection.  Many
> problems decompose like this.
>
> But I don't think this problem does.
>
> In particular, how would the above baby step address the code
> duplication (my goal in the original opening) -- eg the 3 places where
> concrete analyzers/queries are today.  How would it lead to making
> facets work with pure Lucene?  To developing spatial in one place?
>
> Mike
>    

Agreed - merging committers is just one piece - by itself, it doesn't
make much sense. Why would we just make all Lucene devs Solr committers
or vice versa? That's not a goal of mine. If they should be a commiter,
they would contribute to the project and become one naturally. To me its
just a requirement of the other goals (Solr on Lucene trunk, less
duplication/sharing between the projects) - not a baby step we can "try
out".

--
- Mark

http://www.lucidimagination.com

: In particular, how would the above baby step address the code
: duplication (my goal in the original opening) -- eg the 3 places where
: concrete analyzers/queries are today.  How would it lead to making
: facets work with pure Lucene?  To developing spatial in one place?

Having a shared dev list and a unified committer base should (in theory)
help improve the situation of duplicated functionality and promote
refactoring (ie: all commiters are on the same page, and can add/move
classes/tests arround)

If those changes don't have that effect, then i really don't see how
integrating the build processes or releases cycles will ever improve the
situation.
-Hoss

: "it's just software" is another quote I love ...

There are a lot of situations where i 100% agree with that sentiment.  
But we aren't just voting on a code chagne here, we're talking about
community and process -- which isn't just software, it's also people.

People can be messy, they don't always behave the way the spec says they
should and you can't write unit-tests for them.
-Hoss

On 03/04/2010 01:16 PM, Chris Hostetter wrote:
> : In particular, how would the above baby step address the code
> : duplication (my goal in the original opening) -- eg the 3 places where
> : concrete analyzers/queries are today.  How would it lead to making
> : facets work with pure Lucene?  To developing spatial in one place?
>
> Having a shared dev list and a unified committer base should (in theory)
> help improve the situation of duplicated functionality and promote
> refactoring (ie: all commiters are on the same page, and can add/move
> classes/tests arround)
>
> If those changes don't have that effect, then i really don't see how
> integrating the build processes or releases cycles will ever improve the
> situation.
>
>
> -Hoss
>
>    
If Solr is not on Lucene trunk, that's not going to happen.

Solr on Lucene trunk is key. Release cycles is key to that.

Build process is key to getting devs that stick in one world to branch
into the other just a bit.

--
- Mark

http://www.lucidimagination.com

: If Solr is not on Lucene trunk, that's not going to happen.
:
: Solr on Lucene trunk is key. Release cycles is key to that.
:
: Build process is key to getting devs that stick in one world to branch into
: the other just a bit.

Then approach the problem top down instead of bottom up: If this solr-dev
community is in general on board with the "merge" end goal, it shouldn't
be hard to get a lot of solr commiters to work on modifying the Solr
build process so it starts using Lucene trunk (or Lucene nightlies) and
commiting hte changes to Solr to get that to compile/run properly -- if
there are Lucene-Java changes neccessay to make that work, then those
Solr-devs can submit patches back.

As you said in your last email "If they should be a commiter, they would
contribute to the project and become one naturally" ... so why not just
let nature take it's course?  If the Solr community moves it's build
process to be tightly bound with the LUcene one *then* it can make sense
to talk about lock step releases, and shared dev lists, etc...

I honestly don't care which direction we go, but we should be able do this
incrementally -- if we can't, then we probably can't do it in one fell
swoop either.

As I said: The whole thing feels like we're voting on a "goal" -- Goals
are nice in that they give us an end point to aim for, but I can't
remember any time we've ever voted on a "goal" ... this situation (saying
"it would be really great to eventually be/have _____" so let's just set
out to do that") doesn't feel like any change that's ever turned out well
in Lucene.  

Things that have worked well in the past have been incremental and
iterative.
-Hoss

Wouldn't a vote with some kind of a clause be actually a 0 or -1 since the
voter is not agreeing with the current vote in its entirety?

My vote is -1.  I am actually for merging Lucene/Solr development but I feel
that the current vote is too broad.  I would vote for syncing Solr releases
to Lucene releases but not the other way around.

Bill

On Thu, Mar 4, 2010 at 12:37 PM, Mark Miller <[EMAIL PROTECTED]> wrote:

> For those trying to keep track of the current voting, here is a list of the
> Lucene/Solr committers and what I have seen of their votes thus far:
>
> Bill Au
> Doug Cutting
> Otis Gospodnetić
> Erik Hatcher
> Chris Hostetter
> Grant Ingersoll - +1*
> Mike Klaas
> Shalin Shekhar Mangar - +1
> Ryan McKinley
> Mark Miller - +1
> Noble Paul - +1
> Yonik Seeley - +1
> Koji Sekiguchi - +1
> Michael Busch
> Doron Cohen
> Mike McCandless - +1
> Bernhard Messer
> Robert Muir - +1
> Uwe Schindler - +/-0
> Wolfgang Hoschek
> Patrick O'Leary
> Andi Vajda - +1*
> Karl Wettin
> Simon Willnauer - +1
>
> * vote with some kind of clause (is that legal :) )
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>

Who knows - this isn't the official count - just a gauge of what has
happened.

What the true votes of the *'s are remains to be seen - I wouldn't
default them either way
as the voters seemed to think they can vote with a clause - we don't
know what they would vote without. But right now they vote +1 with an
asterisk.

Again this is not the official count - and when we get an official
count, this would still have to then be ratified by the PMC.

On 03/04/2010 01:48 PM, Bill Au wrote:
> Wouldn't a vote with some kind of a clause be actually a 0 or -1 since the
> voter is not agreeing with the current vote in its entirety?
>
> My vote is -1.  I am actually for merging Lucene/Solr development but I feel
> that the current vote is too broad.  I would vote for syncing Solr releases
> to Lucene releases but not the other way around.
>
> Bill
>
> On Thu, Mar 4, 2010 at 12:37 PM, Mark Miller<[EMAIL PROTECTED]>  wrote:
>
>    
>> For those trying to keep track of the current voting, here is a list of the
>> Lucene/Solr committers and what I have seen of their votes thus far:
>>
>> Bill Au
>> Doug Cutting
>> Otis Gospodnetić
>> Erik Hatcher
>> Chris Hostetter
>> Grant Ingersoll - +1*
>> Mike Klaas
>> Shalin Shekhar Mangar - +1
>> Ryan McKinley
>> Mark Miller - +1
>> Noble Paul - +1
>> Yonik Seeley - +1
>> Koji Sekiguchi - +1
>> Michael Busch
>> Doron Cohen
>> Mike McCandless - +1
>> Bernhard Messer
>> Robert Muir - +1
>> Uwe Schindler - +/-0
>> Wolfgang Hoschek
>> Patrick O'Leary
>> Andi Vajda - +1*
>> Karl Wettin
>> Simon Willnauer - +1
>>
>> * vote with some kind of clause (is that legal :) )
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>>      
>    
--
- Mark

http://www.lucidimagination.com

Hi Hoss,

>
> : If Solr is not on Lucene trunk, that's not going to happen.
> :
> : Solr on Lucene trunk is key. Release cycles is key to that.
> :
> : Build process is key to getting devs that stick in one world to branch into
> : the other just a bit.
>
> Then approach the problem top down instead of bottom up: If this solr-dev
> community is in general on board with the "merge" end goal, it shouldn't
> be hard to get a lot of solr commiters to work on modifying the Solr
> build process so it starts using Lucene trunk (or Lucene nightlies) and
> commiting hte changes to Solr to get that to compile/run properly -- if
> there are Lucene-Java changes neccessay to make that work, then those
> Solr-devs can submit patches back.

Agreed, +1, why does there need to be some type of (potentially) sweeping
memorandum in place to ensure this happens?

>
> As you said in your last email "If they should be a commiter, they would
> contribute to the project and become one naturally" ... so why not just
> let nature take it's course?  If the Solr community moves it's build
> process to be tightly bound with the LUcene one *then* it can make sense
> to talk about lock step releases, and shared dev lists, etc...

+1 again and this is also one of my main objections to the proposal as it
stands. Let the community decide who the committers are, and let each
community decide what version of software it depends on. I'm not convinced
that they are the same communities for that matter, and statements made
earlier about Solr being the biggest user of Lucene(-java) may be true from
a Lucene ecosystem perspective, but I disagree it's true (or even provable
for that matter) from an external perspective given Lucene(-java)'s
widespread use predating Solr as an Apache project, and Lucene's infection
into other communities (e.g., PHP with Zend, etc.)

>
> I honestly don't care which direction we go, but we should be able do this
> incrementally -- if we can't, then we probably can't do it in one fell
> swoop either.
>
> As I said: The whole thing feels like we're voting on a "goal" -- Goals
> are nice in that they give us an end point to aim for, but I can't
> remember any time we've ever voted on a "goal" ... this situation (saying
> "it would be really great to eventually be/have _____" so let's just set
> out to do that") doesn't feel like any change that's ever turned out well
> in Lucene.
>
> Things that have worked well in the past have been incremental and
> iterative.

+1.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Thu, 4 Mar 2010, Mark Miller wrote:

> Who knows - this isn't the official count - just a gauge of what has
> happened.
>
> What the true votes of the *'s are remains to be seen - I wouldn't default
> them either way as the voters seemed to think they can vote with a clause
> - we don't know what they would vote without. But right now they vote +1
> with an asterisk.

Indeed. In my case, if the hard synchronisation of releases in both
directions were to become a hard requirement, I would probably vote -1.

I'm +1 with making it a requirement that in order to release Solr we'd have
to release Lucene too.

I am -1 making it a requirement that in order to release Lucene we'd
have to release Solr too.

Infrequent releases have been a problem on Lucene and, apparently, an even
worse one on Solr. Until Solr has shown that it can release as frequently as
Lucene I don't think slowing down Lucene releases even more is a good move.

I think that merging the projects as proposed would make it a easier for
Solr to improve its release frequency, though. This my +1 vote on the rest
of the proposal.

Andi..

On Mar 4, 2010, at 11:09 AM, Mattmann, Chris A (388J) wrote:

>
> +1 again and this is also one of my main objections to the proposal as it
> stands. Let the community decide who the committers are, and let each
> community decide what version of software it depends on. I'm not convinced
> that they are the same communities for that matter, and statements made
> earlier about Solr being the biggest user of Lucene(-java) may be true from
> a Lucene ecosystem perspective, but I disagree it's true (or even provable
> for that matter) from an external perspective given Lucene(-java)'s
> widespread use predating Solr as an Apache project, and Lucene's infection
> into other communities (e.g., PHP with Zend, etc.)

So, when those projects donate themselves to Apache, we can manage them, until then what we are proposing would have absolutely no effect on them.  They'll still get their Lucene JARs just like they always do.  I don't hear anyone proposing to gut those projects.  The fact is, Solr is managed by the Lucene PMC and it is our responsibility to make sure we produce good software under the ASF model.  That doesn't mean we have to merge, but, there are a good chunk of committers and others who feel some improvement on the current situation is necessary.  Moreover, many people want what is in Solr, but don't ever work to keep it whole.  From a PMC perspective, that's not fair to Solr even if it benefits Lucene and is perfectly "legal" and vice versa, so as a PMC member I don't like that situation.  Furthermore, as a committer on both projects, I don't like the duplication of effort and I don't like having to choose between the two when I know that my changes would benefit both communities but am forced to do so solely on the arbitrary fact that way back in 2006 when CNET donated Solr they said "Let's make this a subproject" instead of saying "Let's make this a contrib/feature of Lucene".    I often choose Solr these days solely b/c it means I can get access to it sooner from an end use perspective and nothing else.

The fact is, the PMC is who releases software, not individual committers or even individual subprojects.  The fact that their is a Solr and a Lucene is almost an arbitrary distinction, at least in the early days.  Nowadays, it's obviously grown, but it still is the PMC that releases both.

-Grant

Hi Grant,

>>
>> +1 again and this is also one of my main objections to the proposal as it
>> stands. Let the community decide who the committers are, and let each
>> community decide what version of software it depends on. I'm not convinced
>> that they are the same communities for that matter, and statements made
>> earlier about Solr being the biggest user of Lucene(-java) may be true from
>> a Lucene ecosystem perspective, but I disagree it's true (or even provable
>> for that matter) from an external perspective given Lucene(-java)'s
>> widespread use predating Solr as an Apache project, and Lucene's infection
>> into other communities (e.g., PHP with Zend, etc.)
>
> So, when those projects donate themselves to Apache, we can manage them, until
> then what we are proposing would have absolutely no effect on them.  They'll
> still get their Lucene JARs just like they always do.

Actually it will have a profound affect on them, as it'll define when, how,
where and why they get those JARs as that is dictated by those that release
and prepare them, the entity on the table that is being proposed to be
merged.
> I don't hear anyone
> proposing to gut those projects.  The fact is, Solr is managed by the Lucene
> PMC and it is our responsibility to make sure we produce good software under
> the ASF model.  That doesn't mean we have to merge, but, there are a good
> chunk of committers and others who feel some improvement on the current
> situation is necessary.  Moreover, many people want what is in Solr, but don't
> ever work to keep it whole.

So why does that require "merging" the projects in the way proposed? I'm
sorry I still don't buy into that? Why do we need to merge the committers of
both projects into 1? So Solr committers can touch Lucene code, and update
it? If that's the case, then propose the set of Solr committers that should
be Lucene committers and call a vote on that. Or, vice versa. That way you
can gauge feedback from the community on whether or not those being voted on
have made enough contribution to be approved in. That's the Apache way, as I
understand it.
> From a PMC perspective, that's not fair to Solr
> even if it benefits Lucene and is perfectly "legal" and vice versa, so as a
> PMC member I don't like that situation.  Furthermore, as a committer on both
> projects, I don't like the duplication of effort and I don't like having to
> choose between the two when I know that my changes would benefit both
> communities but am forced to do so solely on the arbitrary fact that way back
> in 2006 when CNET donated Solr they said "Let's make this a subproject"
> instead of saying "Let's make this a contrib/feature of Lucene".    I often
> choose Solr these days solely b/c it means I can get access to it sooner from
> an end use perspective and nothing else.

This is why I was suggesting that perhaps a new TLP might make sense then
for Solr, or potentially even Lucene. Why not start there instead of this
approach being proposed? If the Lucene(-java) committers and Solr committers
want a shared project between them, based off of 2 existing ASF projects, in
essence to merge the code base and ensure that both are aligned and not
duplicated, propose that and let's vote on that. That's not what I'm hearing
though.

>
> The fact is, the PMC is who releases software, not individual committers or
> even individual subprojects.  The fact that their is a Solr and a Lucene is
> almost an arbitrary distinction, at least in the early days.  Nowadays, it's
> obviously grown, but it still is the PMC that releases both.

Understood, I hear ya.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

-0.02 as I'm no committer (but I was considering to involve myself more..)

I'm very concerned about this point:
"Release both at once"

All other ideas sound reasonable, but this one is dangerous: why
enforce upon yourself such a binding limitation?
There might be plenty of good reasons to release a new version of
Lucene while Solr is not ready yet, and still other products might
begin working with a new Lucene version or integrate some urgent fix.
I love Solr, but there's more out there: Alfresco, JIRA, liferay,
hibernate search, elastic search, mahout, CQ5, confluence, Hippo, and
many more ... that means plenty of communities which might want to
contribute to Lucene but don't have a priority to dive into Solr.

It's possible you'll have to release one of the two without any
change, just to release the other for some urgent fix: this will have
people waste time upgrading for nothing, or read an empty changelog
and wonder about the madness in the world.
Not to mention your own effort as time for releasing ?

It's all fine for me, I just suggest to not make this a binding rule
as it will likely hurt yourself.

regards,
Sanne

2010/3/4 Grant Ingersoll <[EMAIL PROTECTED]>:
>
>
> On Mar 4, 2010, at 11:09 AM, Mattmann, Chris A (388J) wrote:
>
>>
>> +1 again and this is also one of my main objections to the proposal as it
>> stands. Let the community decide who the committers are, and let each
>> community decide what version of software it depends on. I'm not convinced
>> that they are the same communities for that matter, and statements made
>> earlier about Solr being the biggest user of Lucene(-java) may be true from
>> a Lucene ecosystem perspective, but I disagree it's true (or even provable
>> for that matter) from an external perspective given Lucene(-java)'s
>> widespread use predating Solr as an Apache project, and Lucene's infection
>> into other communities (e.g., PHP with Zend, etc.)
>
> So, when those projects donate themselves to Apache, we can manage them, until then what we are proposing would have absolutely no effect on them.  They'll still get their Lucene JARs just like they always do.  I don't hear anyone proposing to gut those projects.  The fact is, Solr is managed by the Lucene PMC and it is our responsibility to make sure we produce good software under the ASF model.  That doesn't mean we have to merge, but, there are a good chunk of committers and others who feel some improvement on the current situation is necessary.  Moreover, many people want what is in Solr, but don't ever work to keep it whole.  From a PMC perspective, that's not fair to Solr even if it benefits Lucene and is perfectly "legal" and vice versa, so as a PMC member I don't like that situation.  Furthermore, as a committer on both projects, I don't like the duplication of effort and I don't like having to choose between the two when I know that my changes would benefit both communities but am forced to do so solely on the arbitrary fact that way back in 2006 when CNET donated Solr they said "Let's make this a subproject" instead of saying "Let's make this a contrib/feature of Lucene".    I often choose Solr these days solely b/c it means I can get access to it sooner from an end use perspective and nothing else.
>
> The fact is, the PMC is who releases software, not individual committers or even individual subprojects.  The fact that their is a Solr and a Lucene is almost an arbitrary distinction, at least in the early days.  Nowadays, it's obviously grown, but it still is the PMC that releases both.
>
> -Grant

As far as releases - I'm not sure Lucene has released major versions
appreciably faster than Solr.  Lucene released many more bugfix
releases... but those are orders of magnitude easier - not an issue.

It is precisely the intent, goals, and commitments that matter to me.
A single development team.
I'm not sweating the small stuff - it will be worked out.  The
*intent* to try and develop and release together is what's important.

If we merge, and Solr's not on Lucene's trunk yet, and Lucene is close
to a release - of course it makes sense for Lucene to go ahead and
release.  It's all these things we *don't* have to specify because
we'll use our heads and not try to practice contract law ;-)
That's even true inside Solr itself - when it was felt that some
components weren't quite ready for "prime time", they didn't make it
into the release.

-Yonik

On 03/04/2010 02:13 PM, Andi Vajda wrote:
>
>
> I am -1 making it a requirement that in order to release Lucene we'd
> have to release Solr too.

It would seem to make sense though - if Lucene is releases, it makes
sense that Solr would release too with the Lucene improvements/fixes.

As yonik says, we would just have to do more major Solr dev on branches
so that its easier to release Solr more often.

--
- Mark

http://www.lucidimagination.com

----- Original Message ----
> From: Andi Vajda <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Thu, March 4, 2010 2:13:22 PM
> Subject: Re: [VOTE] merge lucene/solr development
>
>
> On Thu, 4 Mar 2010, Mark Miller wrote:
>
> > Who knows - this isn't the official count - just a gauge of what has happened.
> >
> > What the true votes of the *'s are remains to be seen - I wouldn't default
> them either way as the voters seemed to think they can vote with a clause - we
> don't know what they would vote without. But right now they vote +1 with an
> asterisk.
>
> Indeed. In my case, if the hard synchronisation of releases in both directions
> were to become a hard requirement, I would probably vote -1.

I don't think we want hard release syncing.  We have a clear unidirectional dependency.  Thus:

> I'm +1 with making it a requirement that in order to release Solr we'd have to
> release Lucene too.

If a bug in a Solr release is found, it should get a new bugfix release (off the x.x branch), even if there is no Lucene release.  That x.x.x release of Solr would include the same Lucene jars as the x.x version.

> I am -1 making it a requirement that in order to release Lucene we'd have to
> release Solr too.

I think everyone is in agreement with that.
I think we all realize that while we don't want Solr releases to trail Lucene releases a lot, there will be some delay.  It's just that that delay could be smaller if Solr used either Lucene trunk or if we had a process that updated Lucene jars in Solr lib/ dir in Solr svn repo on a nightly basis (call it near-real-time trunk ;)).

Otis

> Infrequent releases have been a problem on Lucene and, apparently, an even worse
> one on Solr. Until Solr has shown that it can release as frequently as Lucene I
> don't think slowing down Lucene releases even more is a good move.
>
> I think that merging the projects as proposed would make it a easier for Solr to
> improve its release frequency, though. This my +1 vote on the rest of the
> proposal.
>
> Andi..

----- Original Message ----

> From: Uwe Schindler <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Thu, March 4, 2010 11:19:47 AM
> Subject: RE: [VOTE] merge lucene/solr development
>
> If we vote on what Mike says, I revise my vote and simply vote +/-0 to not stop
> progress. I have some problem with the construct but in general I am fine with
> merging dev lists, splitting into modules, merged committers - but not the
> requirement on tests pass always. In my opinion, if anything changed in lucene
> breaks some tests we could open an issue in solr.

I think that's what's being proposed.  With this proposal people wearing Solr dev hats will know they need to fix Solr sooner than they know now - Hudson will tell them on a regular basis, even if you don't spot the Solr test failing or even if you spot it, but don't enter it into JIRA because you know Hudson will tell Solr guys something in Lucene trunk changed very recenly and broke Solr.

Guys, is this interpretation correct?

> One idea: If we really make solr depend on the new lucene lib, solr should not
> have lucene jars in its lib folder, but instead the nightly build should fetch
> the jars from the lucene hudson build. For committers working in svn, maybe some
> relation to rev numbers (like we do for lucene backwards tests) can be put into
> solrs common-build.xml so the ant script of solr can checkout the correct lucene
> rev and build it on they fly.

I was wondering the same thing.  That way svn repos don't need to be reorganized.  Or maybe there is some svn repo linking trickery that's possible.

Otis

> > We are voting on this:
> >
> >  * Merging the dev lists into a single list.
> >
> >  * Merging committers.
> >
> >  * When any change is committed (to a module that "belongs to" Solr or
> >    to Lucene), all tests must pass
> >
> >  * Release both at once (but the specific logistics is still up for
> >    discussion)
> >
> >  * Modularlize the sources: pull things out of Lucene's core (break
> >    out query parser, move all core queries & analyzers under their
> >    contrib counterparts), pull things out of Solr's core (analyzers,
> >    queries)
> >
> > These things would not change:
> >
> >  * Besides modularizing (above), the source code would remain factored
> >    into separate dirs/modules the way it is now.
> >
> >  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX issues)
> >
> >  * User's lists remain separate.
> >
> >  * Web sites remain separate.
> >
> >  * Release artifacts/jars remain separate
> >
> > > I am fine with fixing bugs in solr that are there before the change
> > > but only appear because of the change.
> >
> > OK
> >
> > > My problem is more such things like the per-segment-mega-problem
> > > because Solr was simply using Lucene incorrectly (I hope this is
> > > said too hard).
> >
> > You know, Lucene also used those APIs "incorrectly" until we cutover
> > Lucene's search to be per-segment ;)  We "got lucky" in that the APIs
> > were at best "ambiguous" about whether the incoming reader was
> > per-segment or not.
> >
> > > We did not break backwards.
> >
> > Right, and so Solr's tests should have passed.
> >
> > > But if we would had to repair the whole solr (which is still not
> > > finished) after the per-segment change, we were still not having
> > > per-segment search.
> >
> > But you won't have to fix Solr from such a change.  Others (people
> > wearing Solr hats) will.
> >
> > > And fixing this is surely not easy possible for non-solrcore
> > > developers like me or you.
> >
> > Right.
> >
> > > So even if development goes together we should still have the
> > > possibility to update lucene and if its not a backwards break but
> > > incorrect usage of Lucene's API (or assumptions on behavior of
> > > Lucene's API that are not documented or are obvious from API design
> > > - like for Filters have never to work on Top-Level Searchers and
> > > *only* use the passed in IndexReader), I would simply break solr and

----- Original Message ----

> From: Bill Au <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Wed, March 3, 2010 11:29:33 PM
> Subject: Re: [VOTE] merge lucene/solr development
>
> In the case where changes are in Lucene only I think it is OK to increment
> the Solr release number since even though the Solr jars are unchanged
> because the new release of Solr will contain the new Lucene  jars.  But what
> about the case where changes are in Solr only?  Would we still increment the
> Lucene release number even though everything in the Lucene download is the
> same as before?

I don't think Lucene version would change in this case.

Why do/should release numbers have to remain in sync?  I missed that part...

Otis
> On Wed, Mar 3, 2010 at 10:13 PM, Michael Busch wrote:
>
> > On 3/3/10 6:00 PM, Yonik Seeley wrote:
> >
> >> On Wed, Mar 3, 2010 at 8:10 PM, Michael Busch  wrote:
> >>
> >>
> >>> So if it seems like that most people are concerned about releases (even
> >>>
> >>> those you are generally in favor of this proposal), then maybe we should
> >>> discuss exactly this point. We haven't really discussed alternatives
> >>> about
> >>> the release alignment. This vote feels rushed.
> >>>
> >>>
> >> It's been discussed for a week, and I'm with Mark - I'm only for a
> >> real merge of development, and that includes release schedules.
> >>
> >> -Yonik
> >>
> >>
> >>
> >
> > How will we merge release policies? (or are they exactly the same?) Does
> > Solr use the same release numbering? Does it have the same
> > backwards-compatibility requirements as Lucene?
> >
> > If we release Solr 1.5.0 with Lucene 3.1.0, and we find a bug in the Lucene
> > jar and want to release a 3.1.1 bugfix release, will we also release a Solr
> > 1.5.1, even though all Solr jars would be identical to the 1.5.0?
> > Or will we just release Solr/Lucene 4.0.0 next and always use same release
> > numbers?
> >
> > How will we avoid longer release cycles? Solr had had very infrequent
> > releases. What were the reasons for that? Are we comfortable with saying
> > we'll just try to be disciplined enough or is there a way to somehow enforce
> > release frequency? It seems certain that if more people work on the code,
> > there will at any given point be more patches/new features under development
> > and things need to be coordinated in a way that allows frequent releases.
> >
> > In an earlier mail I gave the following example: If we had a separate
> > analyzer module, and we add an analyzer in a new language to it, it would be
> > cool to release it soon, without having to wait until Lucene/Solr are ready
> > for a release. The pace here can be faster, because I imagine in such an
> > Analyzer module it's much less common that a patch "touches everything".
> > What do people think about this? Maybe it's just a nice wish, but not
> > realizable, because there'd be a lot of version management overhead. But
> > maybe not?
> >
> > I'm not against this whole thing and I'm not trying to be difficult here,
> > and I dislike endless discussions just as everyone else. But I honestly
> > don't know right now how to vote, because I have those open questions and
> > know that a lot of other people here are concerned about the release
> > alignment as well.
> >
> >  Michael
> >
> >

On Thu, Mar 4, 2010 at 4:06 PM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
> Why do/should release numbers have to remain in sync?  I missed that part...

Again, these are just distracting details that we can get buried in.
They will be figured out later.
Questions like this exist even w/o a merge - when Solr upgrades to a
new lucene major release, does it have to make a new major release
also?  No - please don't answer that - it's just an example of stuff
we *shouldn't* have to figure out up front.

-Yonik

On Thu, Mar 4, 2010 at 2:13 PM, Andi Vajda <[EMAIL PROTECTED]> wrote:

> Indeed. In my case, if the hard synchronisation of releases in both
> directions were to become a hard requirement, I would probably vote
> -1.
>
> I'm +1 with making it a requirement that in order to release Solr
> we'd have to release Lucene too.
>
> I am -1 making it a requirement that in order to release Lucene we'd
> have to release Solr too.

I think these sort of logistics can be hashed out by the dev
community... but it's apparently concerning enough people (that
releases are "synchronized").... so I'll call a new vote making this
change explicit (ie, that Lucene can release w/o Solr).

Mike

Uwe Schindler wrote:
> One idea: If we really make solr depend on the new lucene lib, solr
> should not have lucene jars in its lib folder, but instead the
> nightly build should fetch the jars from the lucene hudson build.

If Solr trunk is meant to always be based on Lucene trunk, then
shouldn't they both be under a single trunk?  A Solr release would then
not include an independent version of Lucene, but would rather be an
alternate avenue for releasing the Lucene code.  One could tag, branch
and release them separately or together.  If separately, then a Solr
release tag might include a version of Lucene that's never released
independently.

Doug

Hey Doug,

Doesn't this move towards having a shared code base, and more so the criteria for that of a TLP? Thoughts?

Cheers,
Chris

On 3/4/10 3:08 PM, "Doug Cutting" <[EMAIL PROTECTED]> wrote:

Uwe Schindler wrote:
> One idea: If we really make solr depend on the new lucene lib, solr
> should not have lucene jars in its lib folder, but instead the
> nightly build should fetch the jars from the lucene hudson build.

If Solr trunk is meant to always be based on Lucene trunk, then
shouldn't they both be under a single trunk?  A Solr release would then
not include an independent version of Lucene, but would rather be an
alternate avenue for releasing the Lucene code.  One could tag, branch
and release them separately or together.  If separately, then a Solr
release tag might include a version of Lucene that's never released
independently.

Doug

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Mattmann, Chris A (388J) wrote:
> Doesn't this move towards having a shared code base,

Yes.  The desire seems to be to have a shared code base, no?

> and more so the criteria for that of a TLP? Thoughts?

A shared codebase with a single pool of committers are canonical TLP
attributes, if that's what you mean.

Doug

Apoligies in advance for calling yet another vote, but I just wanted
to make sure this was official.
Mike's second VOTE thread could probably technically stand on it's own
(since it included PMC votes), but given that I said in my previous
VOTE thread that I was just polling Lucene/Solr committers and would
call a second PMC vote, that may have acted to suppress PMC votes on
Mike's thread also.

Please vote for the proposal quoted below to merge lucene/solr development.
Here's my +1

-Yonik

Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
> Subject: Merge the development of Solr/Lucene (take 2)
> A new vote, that slightly changes proposal from last vote (adding only
> that Lucene can cut a release even if Solr doesn't):
>
>  * Merging the dev lists into a single list.
>
>  * Merging committers.
>
>  * When any change is committed (to a module that "belongs to" Solr or
>    to Lucene), all tests must pass.
>
>  * Release details will be decided by dev community, but, Lucene may
>    release without Solr.
>
>  * Modulariize the sources: pull things out of Lucene's core (break
>    out query parser, move all core queries & analyzers under their
>    contrib counterparts), pull things out of Solr's core (analyzers,
>    queries).
>
> These things would not change:
>
>  * Besides modularizing (above), the source code would remain factored
>    into separate dirs/modules the way it is now.
>
>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>    issues).
>
>  * User's lists remain separate.
>
>  * Web sites remain separate.
>
>  * Release artifacts/jars remain separate.

+1

This tally is not official, but here is my count of how people voted on
this merge. A couple people that voted +1 on the first vote did not take
the time to vote again. I've counted them as +1 as I assume they did not
change their mind. Take that for what you will. This is not an official
tally of the vote. If you think your counted wrong, fell free to correct
me. In my mind, its hard to believe that all of these people on the
front lines of Lucene/Solr development don't know what they are doing in
regards to the project.

Bill Au : +1
Doug Cutting
Otis Gospodnetić : +1
Erik Hatcher
Chris Hostetter : -1
Grant Ingersoll : +1
Mike Klaas
Shalin Shekhar Mangar : +1
Ryan McKinley : +1
Mark Miller : +1
Noble Paul : +1
Yonik Seeley : +1
Koji Sekiguchi : +1
Michael Busch : +1
Doron Cohen
Mike McCandless : +1
Bernhard Messer
Robert Muir : +1
Uwe Schindler : +1
Wolfgang Hoschek
Patrick O'Leary
Andi Vajda : +1
Karl Wettin
Simon Willnauer : +1
On 03/08/2010 09:11 PM, Yonik Seeley wrote:
> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>    
>> Subject: Merge the development of Solr/Lucene (take 2)
>> A new vote, that slightly changes proposal from last vote (adding only
>> that Lucene can cut a release even if Solr doesn't):
>>
>>   * Merging the dev lists into a single list.
>>
>>   * Merging committers.
>>
>>   * When any change is committed (to a module that "belongs to" Solr or
>>     to Lucene), all tests must pass.
>>
>>   * Release details will be decided by dev community, but, Lucene may
>>     release without Solr.
>>
>>   * Modulariize the sources: pull things out of Lucene's core (break
>>     out query parser, move all core queries&  analyzers under their
>>     contrib counterparts), pull things out of Solr's core (analyzers,
>>     queries).
>>
>> These things would not change:
>>
>>   * Besides modularizing (above), the source code would remain factored
>>     into separate dirs/modules the way it is now.
>>
>>   * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>     issues).
>>
>>   * User's lists remain separate.
>>
>>   * Web sites remain separate.
>>
>>   * Release artifacts/jars remain separate.
>>      
--
- Mark

http://www.lucidimagination.com

For completeness from the VOTE on private@

PMC votes:
=======================+1

Mark Miller
Michael McCandless
Yonik Seely
Ryan McKinley

-0

Doug Cutting

-1

Dennis Kubes
Scott Ganyo
Chris Mattmann

Cheers,
Chris

On 3/8/10 6:11 PM, "Yonik Seeley" <[EMAIL PROTECTED]> wrote:

Apoligies in advance for calling yet another vote, but I just wanted
to make sure this was official.
Mike's second VOTE thread could probably technically stand on it's own
(since it included PMC votes), but given that I said in my previous
VOTE thread that I was just polling Lucene/Solr committers and would
call a second PMC vote, that may have acted to suppress PMC votes on
Mike's thread also.

Please vote for the proposal quoted below to merge lucene/solr development.
Here's my +1

-Yonik

Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
> Subject: Merge the development of Solr/Lucene (take 2)
> A new vote, that slightly changes proposal from last vote (adding only
> that Lucene can cut a release even if Solr doesn't):
>
>  * Merging the dev lists into a single list.
>
>  * Merging committers.
>
>  * When any change is committed (to a module that "belongs to" Solr or
>    to Lucene), all tests must pass.
>
>  * Release details will be decided by dev community, but, Lucene may
>    release without Solr.
>
>  * Modulariize the sources: pull things out of Lucene's core (break
>    out query parser, move all core queries & analyzers under their
>    contrib counterparts), pull things out of Solr's core (analyzers,
>    queries).
>
> These things would not change:
>
>  * Besides modularizing (above), the source code would remain factored
>    into separate dirs/modules the way it is now.
>
>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>    issues).
>
>  * User's lists remain separate.
>
>  * Web sites remain separate.
>
>  * Release artifacts/jars remain separate.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Mon, Mar 8, 2010 at 9:22 PM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:
> For completeness from the VOTE on private@

It's called private for a reason.

-Yonik

Yet the information we were voting on is public information really and this doesn't really count as "sensitive" IMO. I'd venture to guess not all the PMC is subscribed to general@, and may miss this vote, so it's important their votes be counted.

Chris

On 3/8/10 6:23 PM, "Yonik Seeley" <[EMAIL PROTECTED]> wrote:

On Mon, Mar 8, 2010 at 9:22 PM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:
> For completeness from the VOTE on private@

It's called private for a reason.

-Yonik

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On 03/08/2010 09:32 PM, Mattmann, Chris A (388J) wrote:
> Yet the information we were voting on is public information really and this doesn't really count as "sensitive" IMO.
Any thing I send to private@, I kind of count on not being public. I'd
rather you not decide that for me. In this case, I'm not terribly upset
that my private vote has gotten out - but again, I'd prefer that you
didn't make that call for me. I gave reasons for me vote, and you have
taken the liberty of taking my vote apart from them - I don't like that
either.

> I'd venture to guess not all the PMC is subscribed to general@, and may miss this vote, so it's important their votes be counted.
>    
If you concerned about this, the best way to handle it is to send them a
note. Or send a followup to private@ with a reminder
that a vote is happening on general@. Get permission to take their
private votes public for them. Anything would be better than making our
private communications public - whether we decided the vote should have
been done in public or not.

> Chris
>
>
>
> On 3/8/10 6:23 PM, "Yonik Seeley"<[EMAIL PROTECTED]>  wrote:
>
> On Mon, Mar 8, 2010 at 9:22 PM, Mattmann, Chris A (388J)
> <[EMAIL PROTECTED]>  wrote:
>    
>> For completeness from the VOTE on private@
>>      
> It's called private for a reason.
>
> -Yonik
>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: [EMAIL PROTECTED]
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>    
--
- Mark

http://www.lucidimagination.com

+0

I know I had voted +1 in the second vote, because I was happy about the
fact that Lucene can release w/o Solr. But I spent more time thinking
about this last weekend. I still don't really WANT this change, but can
live with the current proposal. Hence, a +0 in this "official" vote
summarizes probably more accurately how I feel about it.

Question: Is it sufficient to have more +1s than -1s for this vote to
pass? I thought for votes as significant as this one a -1 veto is a
showstopper?

  Michael

On 3/8/10 6:11 PM, Yonik Seeley wrote:
> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>    
>> Subject: Merge the development of Solr/Lucene (take 2)
>> A new vote, that slightly changes proposal from last vote (adding only
>> that Lucene can cut a release even if Solr doesn't):
>>
>>   * Merging the dev lists into a single list.
>>
>>   * Merging committers.
>>
>>   * When any change is committed (to a module that "belongs to" Solr or
>>     to Lucene), all tests must pass.
>>
>>   * Release details will be decided by dev community, but, Lucene may
>>     release without Solr.
>>
>>   * Modulariize the sources: pull things out of Lucene's core (break
>>     out query parser, move all core queries&  analyzers under their
>>     contrib counterparts), pull things out of Solr's core (analyzers,
>>     queries).
>>
>> These things would not change:
>>
>>   * Besides modularizing (above), the source code would remain factored
>>     into separate dirs/modules the way it is now.
>>
>>   * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>     issues).
>>
>>   * User's lists remain separate.
>>
>>   * Web sites remain separate.
>>
>>   * Release artifacts/jars remain separate.
>>      
>

>> Yet the information we were voting on is public information really and this
>> doesn't really count as "sensitive" IMO.
> Any thing I send to private@, I kind of count on not being public. I'd
> rather you not decide that for me. In this case, I'm not terribly upset
> that my private vote has gotten out - but again, I'd prefer that you
> didn't make that call for me. I gave reasons for me vote, and you have
> taken the liberty of taking my vote apart from them - I don't like that
> either.

If you're unhappy I've passed along your private vote, sorry about that. I
think that the natural interpretation of the comments (and subsequent
actions) is that this is a vote best held in public, so I imagined since no
one's vote really changed from the public version we've held 2 times already
(nor did the rationale seemingly) it was a liberty to take. In any case,
sorry to take your liberty away from providing your own private vote, won't
happen again.

>
>> I'd venture to guess not all the PMC is subscribed to general@, and may miss
>> this vote, so it's important their votes be counted.
>>  
> If you concerned about this, the best way to handle it is to send them a
> note. Or send a followup to private@ with a reminder
> that a vote is happening on general@.

That's one way, sure. If it's not a sensitive matter, it's typical Apache
policy for the PMC to conduct as much non-sensitive business in public as
possible. Passing along non-sensitive information for completeness on a VOTE
with impact is a just cause IMO. But, everyone is free to their own
interpretation and those other folks can feel free to re-chime in with their
votes (I've CC'ed them on this message -- please reply to general@).

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Mon, Mar 8, 2010 at 9:49 PM, Michael Busch <[EMAIL PROTECTED]> wrote:
> Question: Is it sufficient to have more +1s than -1s for this vote to pass?
3 +1s and more +1s than -1s is sufficient.

> I thought for votes as significant as this one a -1 veto is a showstopper?
It's not really tied to significance - releases, acceptance to
incubate, etc, all require more +1s than -1s.

-Yonik

On 03/08/2010 09:49 PM, Michael Busch wrote:
>
> Question: Is it sufficient to have more +1s than -1s for this vote to
> pass? I thought for votes as significant as this one a -1 veto is a
> showstopper?
>

Hey Michael - its a good question. And I think the answer is, this is
not a vote that can be vetoed with a single -1. Though I'm not sure its
completely clear.

Quoting the Apache page on voting:

There are essentially three types of vote:

   1. Code modifications,
   2. Package releases
   3. Procedural

Votes on procedural issues follow the common format of majority rule
unless otherwise stated.

Likewise, package release cannot be vetoed. Only Code modifications with
a valid technical reason can be vetoed.

So from what I gather, this is closest to procedural and would be
majority. But that's not entirely clear. But it does appear that Apache
favors for majority for this type of thing (or enough +1's), and saves
vetoes for code changes.

--
- Mark

On 03/08/2010 09:50 PM, Mattmann, Chris A (388J) wrote:
>
> won't
> happen again.
>    

Thanks - I'm obviously not terribly upset about it, but I'd like to
feel  that things I send to private won't go public without me making it
so, since I will write those emails thinking such.

--
- Mark

Hey Mark,

I hear ya on that. No worries.

Cheers,
Chris

On 3/8/10 6:58 PM, "Mark Miller" <[EMAIL PROTECTED]> wrote:

On 03/08/2010 09:50 PM, Mattmann, Chris A (388J) wrote:
>
> won't
> happen again.
>

Thanks - I'm obviously not terribly upset about it, but I'd like to
feel  that things I send to private won't go public without me making it
so, since I will write those emails thinking such.

--
- Mark
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Hmm, Right now I consider Solr too far from lucene to see a merge succeed,
if you asked me 2 years ago I would have said merge merge merge.

I also think Solr hasn't worked out a good roadmap or schedule for releases,
which I'm sure will impact the lucene world.
So

-1

On Mon, Mar 8, 2010 at 6:11 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:

> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
>
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
> > Subject: Merge the development of Solr/Lucene (take 2)
> > A new vote, that slightly changes proposal from last vote (adding only
> > that Lucene can cut a release even if Solr doesn't):
> >
> >  * Merging the dev lists into a single list.
> >
> >  * Merging committers.
> >
> >  * When any change is committed (to a module that "belongs to" Solr or
> >    to Lucene), all tests must pass.
> >
> >  * Release details will be decided by dev community, but, Lucene may
> >    release without Solr.
> >
> >  * Modulariize the sources: pull things out of Lucene's core (break
> >    out query parser, move all core queries & analyzers under their
> >    contrib counterparts), pull things out of Solr's core (analyzers,
> >    queries).
> >
> > These things would not change:
> >
> >  * Besides modularizing (above), the source code would remain factored
> >    into separate dirs/modules the way it is now.
> >
> >  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
> >    issues).
> >
> >  * User's lists remain separate.
> >
> >  * Web sites remain separate.
> >
> >  * Release artifacts/jars remain separate.
>

Hi Michael,

It¹s a good question. I think each side of the fence on this issue has their
own interpretation. Here¹s the Apache page on voting:

http://www.apache.org/foundation/voting.html

I think parts of Mike¹s 2nd proposal [1] (what we¹re voting on) include
elements that are procedural:

 * Merging the dev lists into a single list.
 * Release details will be decided by dev community, but, Lucene may
   release without Solr. (though I¹m not sure why this is included, b/c it¹s
the way that the communities work now?)

But others aren¹t:

 * Merging committers.
And these relate directly to code and will effect change:

 * When any change is committed (to a module that "belongs to" Solr or
   to Lucene), all tests must pass.
 * Modulariize the sources: pull things out of Lucene's core (break
   out query parser, move all core queries & analyzers under their
   contrib counterparts), pull things out of Solr's core (analyzers,
   queries).

So, there are parts of this proposal that I believe VETO does in fact apply
to.

Cheers,
Chris

[1] http://bit.ly/bUJAee

On 3/8/10 6:49 PM, "Michael Busch" <[EMAIL PROTECTED]> wrote:

> Question: Is it sufficient to have more +1s than -1s for this vote to
> pass? I thought for votes as significant as this one a -1 veto is a
> showstopper?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

you should x-post this on solr-dev.

(not a committer so no vote for me)
On 3/9/10 1:11 PM, Yonik Seeley wrote:
> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>    
>> Subject: Merge the development of Solr/Lucene (take 2)
>> A new vote, that slightly changes proposal from last vote (adding only
>> that Lucene can cut a release even if Solr doesn't):
>>
>>   * Merging the dev lists into a single list.
>>
>>   * Merging committers.
>>
>>   * When any change is committed (to a module that "belongs to" Solr or
>>     to Lucene), all tests must pass.
>>
>>   * Release details will be decided by dev community, but, Lucene may
>>     release without Solr.
>>
>>   * Modulariize the sources: pull things out of Lucene's core (break
>>     out query parser, move all core queries&  analyzers under their
>>     contrib counterparts), pull things out of Solr's core (analyzers,
>>     queries).
>>
>> These things would not change:
>>
>>   * Besides modularizing (above), the source code would remain factored
>>     into separate dirs/modules the way it is now.
>>
>>   * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>     issues).
>>
>>   * User's lists remain separate.
>>
>>   * Web sites remain separate.
>>
>>   * Release artifacts/jars remain separate.
>>      
>

I alerted java-dev and solr-dev of the committer vote - we are now onto
a PMC vote.

On 03/08/2010 10:18 PM, Ian Holsman wrote:
> you should x-post this on solr-dev.
>
> (not a committer so no vote for me)
> On 3/9/10 1:11 PM, Yonik Seeley wrote:
>> Apoligies in advance for calling yet another vote, but I just wanted
>> to make sure this was official.
>> Mike's second VOTE thread could probably technically stand on it's own
>> (since it included PMC votes), but given that I said in my previous
>> VOTE thread that I was just polling Lucene/Solr committers and would
>> call a second PMC vote, that may have acted to suppress PMC votes on
>> Mike's thread also.
>>
>> Please vote for the proposal quoted below to merge lucene/solr
>> development.
>> Here's my +1
>>
>> -Yonik
>>
>> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
>> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>>
>>> Subject: Merge the development of Solr/Lucene (take 2)
>>> A new vote, that slightly changes proposal from last vote (adding only
>>> that Lucene can cut a release even if Solr doesn't):
>>>
>>>   * Merging the dev lists into a single list.
>>>
>>>   * Merging committers.
>>>
>>>   * When any change is committed (to a module that "belongs to" Solr or
>>>     to Lucene), all tests must pass.
>>>
>>>   * Release details will be decided by dev community, but, Lucene may
>>>     release without Solr.
>>>
>>>   * Modulariize the sources: pull things out of Lucene's core (break
>>>     out query parser, move all core queries&  analyzers under their
>>>     contrib counterparts), pull things out of Solr's core (analyzers,
>>>     queries).
>>>
>>> These things would not change:
>>>
>>>   * Besides modularizing (above), the source code would remain factored
>>>     into separate dirs/modules the way it is now.
>>>
>>>   * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>>     issues).
>>>
>>>   * User's lists remain separate.
>>>
>>>   * Web sites remain separate.
>>>
>>>   * Release artifacts/jars remain separate.
>
--
- Mark

http://www.lucidimagination.com

I don't think any of it's a showstopper, but at the same time we should try to address the concerns of those who voted -1 and see if a better solution can be developed so that they hopefully can become at least a 0 if not a +1.  The whole point of the move is to build a stronger community, not a weaker one.  At the same time, we should also remember that a large part of the motivation for this move comes from people wanting things that are in Solr to be moved to Lucene in the first place (Analyzers, Faceting, Function Queries, Open Bit Set, Spatial, Schema to name a few past and present ones;  these constitute a lot of Solr's functionality, BTW.)  If there are baby steps that bring the two together, we should consider them.  Personally, I think the proposal contains said baby steps, but perhaps some would prefer smaller ones to begin with so they should outline them.

It should also be noted that a good chunk of the Solr committers are already Lucene committers, and of the remaining there are: Bill Au, Mike Klaas, Ryan McKinley, Shalin and Noble.  Mike has been inactive for quite some time (and has elected to go emeritus even though it's not marked on the page) and and Ryan, Shalin and Noble already contribute to Lucene in various parts (AFAICT), so to me it's not a big stretch to say bring them into the fold.  I haven't tracked Bill's involvement, but I also know Bill and trust he knows what it means to be a committer, i.e. he knows as much what not to touch as what to touch.  Of course, we can do a separate vote on that if that helps satisfy Chris' issue on the committers.  

In the end, for me anyway, the current separation hurts Lucene a good deal as much as it hurts Solr, if not more.  Likewise, I wish some of the Nutch committers would speak up, as I'm sure there are some pieces of Nutch that are "core" too, but for a lack of visibility down lower in Lucene committer wise, especially as Nutch as looking to refactor into more components.  Obviously not the crawling stuff, but perhaps some of Nutch's analyzer and low level Lucene stuff would make sense to be pushed lower in the stack.

In the end, I'm still +1 on the current move.  We can consider the other moves separately if the community wishes.
On Mar 8, 2010, at 9:52 PM, Yonik Seeley wrote:

> On Mon, Mar 8, 2010 at 9:49 PM, Michael Busch <[EMAIL PROTECTED]> wrote:
>> Question: Is it sufficient to have more +1s than -1s for this vote to pass?
> 3 +1s and more +1s than -1s is sufficient.
>
>> I thought for votes as significant as this one a -1 veto is a showstopper?
> It's not really tied to significance - releases, acceptance to
> incubate, etc, all require more +1s than -1s.
>
> -Yonik

On 3/8/10 8:24 PM, Grant Ingersoll wrote:
> I don't think any of it's a showstopper,

I'm surprised here after reading the Apache voting page. This proposal
contains points that involve code restructurings.

> but at the same time we should try to address the concerns of those who voted -1 and see if a better solution can be developed so that they hopefully can become at least a 0 if not a +1.  The whole point of the move is to build a stronger community, not a weaker one.
>    

Yes I totally agree and thanks for saying that, Grant.  We should try to
make a decision so that the general mood in this community is good at
the end and that everyone considers the outcome as beneficial for the
whole project. This means that both sides have to be open for compromises.

  Michael

On Mon, Mar 8, 2010 at 11:59 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote:
> I read the previous discussions on general (although I missed the original
> email by Yonik which I have since read) and I think all of this discussion
> should be happening there, so I copied general to this response. But since
> this vote is occurring on private I thought it most appropriate to respond
> where it occurred.

We've had tons of discussion and two votes on general@ already amongst
the lucene/solr committers.

I called the final PMC vote on private@ to ratify the decision of the
lucene/solr committers.  Doug correctly pointed out that the PMC vote
doesn't belong on private@ so I moved it back here.  So the vote is no
longer being held on private@.

It would be odd indeed if the PMC voted to overrule the committers.

-Yonik

On 03/09/2010 12:14 AM, Michael Busch wrote:
> On 3/8/10 8:24 PM, Grant Ingersoll wrote:
>> I don't think any of it's a showstopper,
>
> I'm surprised here after reading the Apache voting page. This proposal
> contains points that involve code restructurings.
The veto is reserved for "code modifications" not reorganizations of
development. And the veto requires a valid technical reason against a
specific code change.

Also, we have decided on no code restructurings - the hope is to allow
them (and in the past you have championed some of the ones we hope to
see), but there are no restructurings that are part of the vote. The
change says nothing about what will happen regarding the code - the
community would decide that as we go. If you have to pick one of the 3
buckets, this is procedural.

http://www.apache.org/foundation/voting.html

--
- Mark

On 3/8/10 9:26 PM, "Mark Miller" <[EMAIL PROTECTED]> wrote:

> Also, we have decided on no code restructurings - the hope is to allow
> them (and in the past you have championed some of the ones we hope to
> see), but there are no restructurings that are part of the vote.

Ummm, that's not true.

Mike's last proposal listed these points:

 * When any change is committed (to a module that "belongs to" Solr or
   to Lucene), all tests must pass.
 * Modulariize the sources: pull things out of Lucene's core (break
   out query parser, move all core queries & analyzers under their
   contrib counterparts), pull things out of Solr's core (analyzers,
   queries).

If those don't have to do with code changes, then I'm not sure what they are
and would appreciate clarification.

Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Just to clarify - reading back, I see Mike put some examples of what
could be pulled into Lucene core - I personally don't see that as a
binding part of the vote - they are what we hope to do and are examples
of things that are not controversial. I think quite obviously, anything
after the merge would be taken as we normally take it - we would make
issues, someone would have to actually do the work, etc. Mike listed
some things that make sense to go into Lucene, and I doubt anyone is
going to argue, but it would be weird to say the result of this Vote
demands we move all queries from Solr into Lucene. It will just allow us
to do so and it would make sense to do so.

Mike took that from an earlier email proposing the merge - its not
really a "road plan" that we are voting for in terms of what goes where
- we are voting for the merge of development. What goes where should be
determined like we normally do that stuff.

- Mark

On 03/09/2010 12:26 AM, Mark Miller wrote:
> On 03/09/2010 12:14 AM, Michael Busch wrote:
>> On 3/8/10 8:24 PM, Grant Ingersoll wrote:
>>> I don't think any of it's a showstopper,
>>
>> I'm surprised here after reading the Apache voting page. This
>> proposal contains points that involve code restructurings.
>
>
> The veto is reserved for "code modifications" not reorganizations of
> development. And the veto requires a valid technical reason against a
> specific code change.
>
> Also, we have decided on no code restructurings - the hope is to allow
> them (and in the past you have championed some of the ones we hope to
> see), but there are no restructurings that are part of the vote. The
> change says nothing about what will happen regarding the code - the
> community would decide that as we go. If you have to pick one of the 3
> buckets, this is procedural.
>
> http://www.apache.org/foundation/voting.html
>
> --
> - Mark
>
>    

Hey Chris,

see my response to Michael.

But quickly,

the first star is not a code change. Its procedural.

the second star, and I'm sure youll have arguments with this :), is not
something we are specifically voting on. The reason we are merging dev
is obviously so that those changes can occur - but this vote is not to
force those changes. Even those against the merge would like to see
those changes. Putting more queries, querparsers, and analyzers into
Lucene is not a controversial change :)

On 03/09/2010 12:33 AM, Mattmann, Chris A (388J) wrote:
> On 3/8/10 9:26 PM, "Mark Miller"<[EMAIL PROTECTED]>  wrote:
>
>    
>> Also, we have decided on no code restructurings - the hope is to allow
>> them (and in the past you have championed some of the ones we hope to
>> see), but there are no restructurings that are part of the vote.
>>      
> Ummm, that's not true.
>
> Mike's last proposal listed these points:
>
>   * When any change is committed (to a module that "belongs to" Solr or
>     to Lucene), all tests must pass.
>   * Modulariize the sources: pull things out of Lucene's core (break
>     out query parser, move all core queries&  analyzers under their
>     contrib counterparts), pull things out of Solr's core (analyzers,
>     queries).
>
> If those don't have to do with code changes, then I'm not sure what they are
> and would appreciate clarification.
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: [EMAIL PROTECTED]
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>    
--
- Mark

http://www.lucidimagination.com

Frankly, if you guys insist, we could drop the modularize point and take
yet another vote. If that's going to be your veto toehold, we don't need
it cluttering things up. Modularizing Lucene doesn't need to be in there
(though it already is somewhat modularized, and people plan to work
further along those lines regardless of this vote). Specific things we
would like to pull from Solr into Lucene don't need to be in there. All
of a sudden I'm agreeing with Hoss about goals rather than actual steps
;) Because those points are not important to this vote at all - they are
more examples of what we will be able to do than mandates. They are the
goodness that will come, not reasons for vetoes. (nor do I agree they
fall under the "code modication veto for a valid technical reason" anyway)
This is about merging dev so we can put code where it belongs and do
things that can make sense - its not a vote where specific code
refactorings matter at all - we don't develop and organize code with PMC
votes.

On 03/09/2010 12:40 AM, Mark Miller wrote:
> Hey Chris,
>
> see my response to Michael.
>
> But quickly,
>
> the first star is not a code change. Its procedural.
>
> the second star, and I'm sure youll have arguments with this :), is
> not something we are specifically voting on. The reason we are merging
> dev is obviously so that those changes can occur - but this vote is
> not to force those changes. Even those against the merge would like to
> see those changes. Putting more queries, querparsers, and analyzers
> into Lucene is not a controversial change :)
>
> On 03/09/2010 12:33 AM, Mattmann, Chris A (388J) wrote:
>> On 3/8/10 9:26 PM, "Mark Miller"<[EMAIL PROTECTED]>  wrote:
>>
>>> Also, we have decided on no code restructurings - the hope is to allow
>>> them (and in the past you have championed some of the ones we hope to
>>> see), but there are no restructurings that are part of the vote.
>> Ummm, that's not true.
>>
>> Mike's last proposal listed these points:
>>
>>   * When any change is committed (to a module that "belongs to" Solr or
>>     to Lucene), all tests must pass.
>>   * Modulariize the sources: pull things out of Lucene's core (break
>>     out query parser, move all core queries&  analyzers under their
>>     contrib counterparts), pull things out of Solr's core (analyzers,
>>     queries).
>>
>> If those don't have to do with code changes, then I'm not sure what
>> they are
>> and would appreciate clarification.
>>
>> Cheers,
>> Chris
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: [EMAIL PROTECTED]
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>
>
--
- Mark

http://www.lucidimagination.com

I believe this is a question of identity.  What is Lucene?

IMO Lucene is a full text search library, that is it's purpose.  It
isn't trying to be a search server or a search engine.  It is easy to
include as a library and is used on everything from embedded servers to
www search engines.

Quoting from Yonik's previous posting:

 > Some in Lucene development have expressed a desire to make Lucene more
 > of a complete solution, rather than just a core full-text search
 > library... things like a data schema, faceting, etc.  The Lucene
 > project already has an enterprise search platform with these
 > features... that's Solr.

So is Lucene a full text search library or is it something different?
And isn't that something different already Solr?  Why should they be the
same thing when their goals aren't the same?

 > Trying to pull popular pieces out of Solr
 > makes life harder for Solr developers, brings our projects into
 > conflict, and is often unsuccessful (witness the largely failed
 > migration of FunctionQueries from Solr to Lucene).

I feel for you, really.  I remember trying to develop in Nutch on Hadoop
0.04.  But the logic is not correct.  Just because Solr wants X feature
and Solr uses Lucene != everyone who uses Lucene wants X.  Faceting for
example, great feature, but not useful in every full text search.

 > For Lucene to achieve the ultimate in usability for users, it can't
 > require Java experience... it needs higher level abstractions provided
 > by Solr.

I don't believe this to be true.  If the Lucene community had wanted
very general language agnostic search, it would have happened by now.
Lucene is a Java API.  Solr on the other hand is a server and therefore
should be language agnostic.

 > The other benefit to Lucene would be to bring features to developers
 > much sooner... Solr has had features years before they were developed
 > in Lucene, and currently has more developers working with it.

"We have more developers than you do" isn't a valid reason to merge,
especially in open source software.  Maybe in the corporate world.  IMO
if Solr has more developers and want some architecture changed in Lucene
and it is to the benefit of the entire Lucene community, then those
changes can be proposed and voted upon.

 > Esp with Solr not using Lucene trunk, if a Solr developer wants a
 > feature quickly, they cannot add it to Lucene (even if it might make
 > sense there) since that introduces a big unpredictable lag

Solr has the option of not using Lucene.  If something needs to go into
Lucene, it should be voted on and support all of the different uses for
Lucene.  As a friend told me recently, specialization is for insects.

 > 1) Solr would go back to using Lucene's trunk

Use trunk, don't use trunk.  That is up to the Solr project.  It
shouldn't influence Lucene's behavior.

 > 2) For new Solr features, there would be an effort to abstract it such
 > that non-Solr users could use the functionality (faceting, field
 > collapsing, etc)

Can you say that every feature would be applicable to a full text search
library.  If not then it is beyond the core responsibilities of Lucene.

 > 3) For new Lucene features, there would be an effort to integrate it
 > into Solr.

No.  Because by specializing towards Solr, or Nutch, or any of the
hundred other applications that use Lucene, it looses its general
applicability.  Where would Hadoop be if it never made it past Nutch?

 > 4) Releases would be synchronized... Lucene and Solr would release at
 > the same time.

So synchronize your releases.  Communicate.

I am open to listening to your responses, but all of this is to say my
vote is still currently -1.

Dennis

There are scads of features of Lucene that are not useful for all
applications (payloads, for one example, back compatibility for another).

The point is that the option to use faceting or not would be *very* useful
for all search applications.  Power is good, especially since somebody else
has done the work already.

On Mon, Mar 8, 2010 at 10:10 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote:

> Faceting for example, great feature, but not useful in every full text
> search.
>

This logic escapes me.

Nutch hatched Hadoop.  Hadoop was perceived to be of much broader utility
than just for nutch so it was made more general and a separate project was
formed.  Hadoop does not depend on Nutch.

Lucene existed.  Solr was built to make it easier to use Lucene.  The
developers of Solr built a bunch of stuff that was specific to server-ness
and a bunch of stuff that would have general utility to many Lucene
developers.  Solr depends critically on Lucene and can be seen as a Lucene
wrapper.

How does this analogy fit together?  Is it supposed to be Hadoop is to Nutch
as Solr is to Lucene?  That seems so clearly wrong it can't be what you were
saying.

On Mon, Mar 8, 2010 at 10:10 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote:

> > 3) For new Lucene features, there would be an effort to integrate it
> > into Solr.
>
> No.  Because by specializing towards Solr, or Nutch, or any of the hundred
> other applications that use Lucene, it looses its general applicability.
>  Where would Hadoop be if it never made it past Nutch?
>

On Tue, Mar 9, 2010 at 7:41 AM, Yonik Seeley <[EMAIL PROTECTED]> wrote:

>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
>
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
> > Subject: Merge the development of Solr/Lucene (take 2)
> > A new vote, that slightly changes proposal from last vote (adding only
> > that Lucene can cut a release even if Solr doesn't):
> >
> >  * Merging the dev lists into a single list.
> >
> >  * Merging committers.
> >
> >  * When any change is committed (to a module that "belongs to" Solr or
> >    to Lucene), all tests must pass.
> >
> >  * Release details will be decided by dev community, but, Lucene may
> >    release without Solr.
> >
> >  * Modulariize the sources: pull things out of Lucene's core (break
> >    out query parser, move all core queries & analyzers under their
> >    contrib counterparts), pull things out of Solr's core (analyzers,
> >    queries).
> >
> > These things would not change:
> >
> >  * Besides modularizing (above), the source code would remain factored
> >    into separate dirs/modules the way it is now.
> >
> >  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
> >    issues).
> >
> >  * User's lists remain separate.
> >
> >  * Web sites remain separate.
> >
> >  * Release artifacts/jars remain separate.
>

+1

I think that, in the long term, this move will prove beneficial for both the
projects.

--
Regards,
Shalin Shekhar Mangar.

I'm still trying to grok the different points of view and apparent
(mis?) perceptions on what everyone is saying.

Going back to the beginning, the basic problem is that code is
duplicated between solr and lucene and fixing that is difficult with
the current structure.

There is no intention "merge" solr and lucene - just figure out a way
to make their development smoother.
 - Lucene is (and will be) the core full text search library and a
bunch of optional packages to support various things like analysis,
query parsing, etc, etc
 - Solr is a full featured search server -- it strings together many
of the lucene libraries with a simplified API (http)

This proposal would make it easier to keep the general stuff (like
numerics, spatial, analysis) out of solr and into optional lucene
packages.  It would not change the nature of either project.
On Mon, Mar 8, 2010 at 11:59 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote:
> I read the previous discussions on general (although I missed the original
> email by Yonik which I have since read) and I think all of this discussion
> should be happening there, so I copied general to this response. But since
> this vote is occurring on private I thought it most appropriate to respond
> where it occurred.  Wouldn't you agree?
>
> I believe this is a question of identity.  What is Lucene?
>
> IMO Lucene is a full text search library, that is it's purpose.  It isn't
> trying to be a search server or a search engine.  It is easy to include as a
> library and is used on everything from embedded servers to www search
> engines.

absolutely agree.  This proposal would help both projects focus on what they do

>
> Quoting from Yonik's previous posting:
>
>> Some in Lucene development have expressed a desire to make Lucene more
>> of a complete solution, rather than just a core full-text search
>> library... things like a data schema, faceting, etc.  The Lucene
>> project already has an enterprise search platform with these
>> features... that's Solr.
>
> So is Lucene a full text search library or is it something different? And
> isn't that something different already Solr?  Why should they be the same
> thing when their goals aren't the same?

they would not be the same thing.  They would be different packages
(as they are now).  It would just be easier to manage the development
of many optional search features.

>
>> Trying to pull popular pieces out of Solr
>> makes life harder for Solr developers, brings our projects into
>> conflict, and is often unsuccessful (witness the largely failed
>> migration of FunctionQueries from Solr to Lucene).
>
> I feel for you, really.  I remember trying to develop in Nutch on Hadoop
> 0.04.  But the logic is not correct.  Just because Solr wants X feature and
> Solr uses Lucene != everyone who uses Lucene wants X.  Faceting for example,
> great feature, but not useful in every full text search.
>

The real problem is that both solr and lucene have their own versions
of the same thing.  There is no intention to add *every* feature to
the core, rather make sure that development can be focused somewhere.
Right now there are two versions of function queries, numerics and
spatial... uggg.
>> For Lucene to achieve the ultimate in usability for users, it can't
>> require Java experience... it needs higher level abstractions provided
>> by Solr.
>
> I don't believe this to be true.  If the Lucene community had wanted very
> general language agnostic search, it would have happened by now. Lucene is a
> Java API.  Solr on the other hand is a server and therefore should be
> language agnostic.
>

agree

>> The other benefit to Lucene would be to bring features to developers
>> much sooner... Solr has had features years before they were developed
>> in Lucene, and currently has more developers working with it.
>
> "We have more developers than you do" isn't a valid reason to merge,
> especially in open source software.  Maybe in the corporate world.  IMO if
> Solr has more developers and want some architecture changed in Lucene and it

I agree in theory...  but in practice, it means that stuff that
conceptually should live in lucene gets added to solr and later
duplicated in lucene
That would be the lipmus test to know if it should be in the lucene
distribution vs solr.

But yes, I think collapsing and faceting (sans-solr) belong in lucene.

Not sure I follow...  I think this means solr would aim to use the new
lucene features as they are developed.  For example, using the
reusable token streams from the get-go rather then get stuck in 6
months of stale patches that don't apply.
ryan

+1

Mike

On Mon, Mar 8, 2010 at 9:11 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>> Subject: Merge the development of Solr/Lucene (take 2)
>> A new vote, that slightly changes proposal from last vote (adding only
>> that Lucene can cut a release even if Solr doesn't):
>>
>>  * Merging the dev lists into a single list.
>>
>>  * Merging committers.
>>
>>  * When any change is committed (to a module that "belongs to" Solr or
>>    to Lucene), all tests must pass.
>>
>>  * Release details will be decided by dev community, but, Lucene may
>>    release without Solr.
>>
>>  * Modulariize the sources: pull things out of Lucene's core (break
>>    out query parser, move all core queries & analyzers under their
>>    contrib counterparts), pull things out of Solr's core (analyzers,
>>    queries).
>>
>> These things would not change:
>>
>>  * Besides modularizing (above), the source code would remain factored
>>    into separate dirs/modules the way it is now.
>>
>>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>    issues).
>>
>>  * User's lists remain separate.
>>
>>  * Web sites remain separate.
>>
>>  * Release artifacts/jars remain separate.
>

guys.. there is a lot of discussion going on.
and a awful lot of voting.
and as a interesting onlooker, I am really confused about what exactly
the vote is for.. there are so many interjections and clarifications
that I'm not sure
what is going on

can we stop calling for a vote for say 72 hours or a week, and just
discuss it a bit, get the proposal on what you want to be done clarified
and then call for a vote?

it's not like there is a house on fire, and this really seems like it is
getting pushed.
On 3/9/10 5:46 PM, Ted Dunning wrote:
> There are scads of features of Lucene that are not useful for all
> applications (payloads, for one example, back compatibility for another).
>
> The point is that the option to use faceting or not would be *very* useful
> for all search applications.  Power is good, especially since somebody else
> has done the work already.
>
> On Mon, Mar 8, 2010 at 10:10 PM, Dennis Kubes<[EMAIL PROTECTED]>  wrote:
>
>    
>> Faceting for example, great feature, but not useful in every full text
>> search.
>>
>>      
>

Hi,

On Tue, Mar 9, 2010 at 3:11 AM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Please vote for the proposal quoted below to merge lucene/solr development.

+0 with my PMC member hat on, as I think this matter is up to the
Lucene and Solr developers to decide.

That said, I generally think having multiple distinct development
communities under one PMC is a bit troublesome (as we're seeing in
this discussion), so consolidating the community seems like a good
direction given that the technical synergies are there. On the other
hand I share Chris' concern about the massive scope of this vote. All
the proposed changes could IMHO just as well be handled as separate
and more easily reversible steps.

BR,

Jukka Zitting

On 2010-03-09 05:24, Grant Ingersoll wrote:

> In the end, for me anyway, the current separation hurts Lucene a good
> deal as much as it hurts Solr, if not more.  Likewise, I wish some of
> the Nutch committers would speak up, as I'm sure there are some
> pieces of Nutch that are "core" too, but for a lack of visibility
> down lower in Lucene committer wise, especially as Nutch as looking
> to refactor into more components.  Obviously not the crawling stuff,
> but perhaps some of Nutch's analyzer and low level Lucene stuff would
> make sense to be pushed lower in the stack.

With my Nutch hat on, I'm -0 to this current vote.

If the primary devs really insist on going this way, so be it, but I
think that long-term it brings more challenges than it solves, among
them the danger that Lucene ceases to be a general purpose Java search
library (where being Java-centric is nothing wrong) and caters too much
to Solr's needs at the expense of other projects.

Re: Nutch components - those that are reusable in Lucene or Solr
contexts eventually find their way to respective projects, witness e.g.
CommonGrams. Other stuff makes sense only in Nutch and it would be a
mistake to push it by force to become e.g. a contrib module in Lucene if
it's not applicable to a majority of Lucene community. Refactoring to
increase reuse doesn't mean we have to merge Nutch with Lucene, it's
just a cleaner separation of concerns. Anyway, that's not the topic of
the current vote.

--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

On Tue, Mar 9, 2010 at 5:10 AM, Andrzej Bialecki <[EMAIL PROTECTED]> wrote:

> Re: Nutch components - those that are reusable in Lucene or Solr
> contexts eventually find their way to respective projects, witness
> e.g. CommonGrams.

In fact I think this is a great example -- as far as I can tell,
CommonGrams was poached from Nutch, into Solr, and then was
nurtured/improved in both projects separately right?

So.... can/should we freely poach across all our sub projects?

It has obvious downsides (it's essentially a fork that will confuse
those users that use both Solr & Lucene, in the short term, until
things "stabilize" into a clean refactoring; it's double the dev; we
must re-sync with time; etc.).

But it has a massive upside: it means we don't rely only on "push"
(Solr devs to push into Lucene or vice/versa).  We can also use "pull"
(Lucene devs can pull pieces from Nutch/Solr into Lucene).  It becomes
a 2-way street for "properly" factoring our shared code with time.

If we had that freedom ("poaching is perfectly fine"), then,
interested devs could freely "refactor" across sub projects.

Not having this freedom today, and not having merged dev, is stunting
both Solr & Lucene's growth.

Mike

On 2010-03-09 11:40, Michael McCandless wrote:
> On Tue, Mar 9, 2010 at 5:10 AM, Andrzej Bialecki<[EMAIL PROTECTED]>  wrote:
>
>> Re: Nutch components - those that are reusable in Lucene or Solr
>> contexts eventually find their way to respective projects, witness
>> e.g. CommonGrams.
>
> In fact I think this is a great example -- as far as I can tell,
> CommonGrams was poached from Nutch, into Solr, and then was
> nurtured/improved in both projects separately right?

Right. In fact, Nutch would like to eventually delegate indexing solely
to Solr, at which point we will reuse the CommonGrams from Solr.

>
> So.... can/should we freely poach across all our sub projects?

In my opinion: with proper attribution, by all means!

>
> It has obvious downsides (it's essentially a fork that will confuse
> those users that use both Solr&  Lucene, in the short term, until
> things "stabilize" into a clean refactoring; it's double the dev; we
> must re-sync with time; etc.).
>
> But it has a massive upside: it means we don't rely only on "push"
> (Solr devs to push into Lucene or vice/versa).  We can also use "pull"
> (Lucene devs can pull pieces from Nutch/Solr into Lucene).  It becomes
> a 2-way street for "properly" factoring our shared code with time.
>
> If we had that freedom ("poaching is perfectly fine"), then,
> interested devs could freely "refactor" across sub projects.
>
> Not having this freedom today, and not having merged dev, is stunting
> both Solr&  Lucene's growth.

Erhm.. don't we have this freedom already??? Another example is
TimeLimitedCollector - poaching _is_ perfectly fine as far as I'm
concerned. All projects are under the same license, often also share the
same people, so I see no reason not to share freely where it makes sense
from technical POV, though we may sometimes succumb to the NIH syndrome ;)

This push/pull between the projects reminds me of discussions with my
clients when I try to convince them to open-source some generic
functionality: long-term you can only benefit greatly from getting rid
of generic code, if there's a lively community with focus just on that
functionality - you don't have to maintain it and you reap the benefits
of external development, and you can focus on developing of what's
unique to your project.

So, I'm all for the poaching ;) but IMHO this doesn't necessitate the
merge, just refactoring, push/pull and the right mindset.
--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

On Tue, Mar 9, 2010 at 6:09 AM, Andrzej Bialecki <[EMAIL PROTECTED]> wrote:
> On 2010-03-09 11:40, Michael McCandless wrote:
>>
>> On Tue, Mar 9, 2010 at 5:10 AM, Andrzej Bialecki<[EMAIL PROTECTED]>  wrote:
>>
>> So.... can/should we freely poach across all our sub projects?
>
> In my opinion: with proper attribution, by all means!

+1

I think shifting to this ("poaching is fine") would be healthy for all
subs.  Pull & push would let code flow freely two-way across the
projects.

>>> Re: Nutch components - those that are reusable in Lucene or Solr
>>> contexts eventually find their way to respective projects, witness
>>> e.g. CommonGrams.
>>
>> In fact I think this is a great example -- as far as I can tell,
>> CommonGrams was poached from Nutch, into Solr, and then was
>> nurtured/improved in both projects separately right?
>
> Right. In fact, Nutch would like to eventually delegate indexing solely to
> Solr, at which point we will reuse the CommonGrams from Solr.

Exactly -- this is how the refactoring would play out with time.
Begins with poaching but eventually winds up with a single source
again (just "moved" from one sub to another).

Ie, if Lucene poached all Solr analyzers, as well as its own core
analyzers, moving all analyzers into contrib/analyzers, then
eventually Solr/Nutch would just use Lucene's contrib/analyzers as the
single source.

Others (whoever has the itch/time) can poach function queries, facets,
etc.

The freedom to poach gives us a powerful push AND pull tool to make
this refactoring gradually over time.

Something interesting can be born in Solr (just because that's where
the itch first arrived) and then freely poached & refactored later by
someone wearing a Lucene dev hat.

> So, I'm all for the poaching ;) but IMHO this doesn't necessitate the merge,
> just refactoring, push/pull and the right mindset.

In fact in my opinion, if we are free to poach across all subs, we
don't need to merge -- poaching solves the primary pain I feel with
our subs now (splintering of code/dev across the subs,
preventing/frustrating people like Robert who put in tons of effort to
improve our analyzers only to see patches languish).

Mike

On Mar 9, 2010, at 5:40 AM, Michael McCandless wrote:

> On Tue, Mar 9, 2010 at 5:10 AM, Andrzej Bialecki <[EMAIL PROTECTED]> wrote:
>
>> Re: Nutch components - those that are reusable in Lucene or Solr
>> contexts eventually find their way to respective projects, witness
>> e.g. CommonGrams.
>
> In fact I think this is a great example -- as far as I can tell,
> CommonGrams was poached from Nutch, into Solr, and then was
> nurtured/improved in both projects separately right?
>
> So.... can/should we freely poach across all our sub projects?
>
> It has obvious downsides (it's essentially a fork that will confuse
> those users that use both Solr & Lucene, in the short term, until
> things "stabilize" into a clean refactoring; it's double the dev; we
> must re-sync with time; etc.).
>
> But it has a massive upside: it means we don't rely only on "push"
> (Solr devs to push into Lucene or vice/versa).  We can also use "pull"
> (Lucene devs can pull pieces from Nutch/Solr into Lucene).  It becomes
> a 2-way street for "properly" factoring our shared code with time.
>
> If we had that freedom ("poaching is perfectly fine"), then,
> interested devs could freely "refactor" across sub projects.
>

As someone who works on both, I don't think it is fine.  Just look at the function query mess.  Just look at the version mess.  It's very frustrating as a developer and it makes me choose between two projects that I happen to like equally, but for different reasons.  If I worked on Nutch, I would feel the same way.

Also, I do look at Solr/Lucene differently.  There is almost complete overlap in the committer base.  Nutch is not that way, nor is any other project.  I simply don't think Lucene will end up being geared toward Solr because there are so many users of Lucene here they will prevent that from happening.  

-Grant

On 2010-03-09 13:21, Grant Ingersoll wrote:
>
> On Mar 9, 2010, at 5:40 AM, Michael McCandless wrote:

>> If we had that freedom ("poaching is perfectly fine"), then,
>> interested devs could freely "refactor" across sub projects.
>>
>
> As someone who works on both, I don't think it is fine.  Just look at
> the function query mess.  Just look at the version mess.  It's very
> frustrating as a developer and it makes me choose between two
> projects that I happen to like equally, but for different reasons.
> If I worked on Nutch, I would feel the same way.

The mess happened afaik due to a lack of communication and NIH. It's
true that
if Sol had been merged with Lucene then only one version would have won.
This may still happen with enough cooperation on both sides, even
without the merge.

Anyway, my vote hovers near 0 either way, as you said it's different for
Nutch.

--
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

On Tue, Mar 9, 2010 at 7:21 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote:

>> If we had that freedom ("poaching is perfectly fine"), then,
>> interested devs could freely "refactor" across sub projects.
>
> As someone who works on both, I don't think it is fine.  Just look at the function query mess.  Just look at the version mess.  It's very frustrating as a developer and it makes me choose between two projects that I happen to like equally, but for different reasons.  If I worked on Nutch, I would feel the same way.

But... Lucene should poach from external (eg non-Apache) projects, if
the license works?

Ie if some great analyzer is out there, and Robert spots it, and the
license works, we should poach it?  (In fact he just did this w/
Andrzej's Polish stemmer ;) ).

So we have something of a double standard...

And, ironically, I think it's the fact that there's so much committer
overlap between Solr and Lucene that is causing this antagonism
towards poaching.

When in fact I think poaching, at a wider scale (across unrelated
projects) is a very useful means for any healthy open source software
to evolve.

Why should Lucene be prevented from having a useful feature just
because Solr happened to create it first?

Mike

On Mar 9, 2010, at 8:21 AM, Michael McCandless wrote:

> On Tue, Mar 9, 2010 at 7:21 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote:
>
>>> If we had that freedom ("poaching is perfectly fine"), then,
>>> interested devs could freely "refactor" across sub projects.
>>
>> As someone who works on both, I don't think it is fine.  Just look at the function query mess.  Just look at the version mess.  It's very frustrating as a developer and it makes me choose between two projects that I happen to like equally, but for different reasons.  If I worked on Nutch, I would feel the same way.
>
> But... Lucene should poach from external (eg non-Apache) projects, if
> the license works?
>
> Ie if some great analyzer is out there, and Robert spots it, and the
> license works, we should poach it?  (In fact he just did this w/
> Andrzej's Polish stemmer ;) ).

I'd prefer "donate" to poach, but, realize that isn't always the case.
>
> So we have something of a double standard...
>
> And, ironically, I think it's the fact that there's so much committer
> overlap between Solr and Lucene that is causing this antagonism
> towards poaching.
>
> When in fact I think poaching, at a wider scale (across unrelated
> projects) is a very useful means for any healthy open source software
> to evolve.
>
> Why should Lucene be prevented from having a useful feature just
> because Solr happened to create it first?

But why should I be forced to maintain two versions due to some arbitrary code separation?  And why should you force a good chunk of us to do a whole lot of extra work simply because of some arbitrary code separation?  Here, it is the Lucene PMC that releases code and it is just silly that with all of this overlap at the committer level we still have this duplication.   I can't speak for the external projects (I don't believe any of them have even responded here other than Jackrabbit), but if they don't like it, they should get more involved in the community and work to be committers.  

At any rate, this is exactly why merging makes sense.  You would no longer have this issue of "first".  I would no longer have to choose where to add my spatial work based on some arbitrary line that someone drew in the sand that isn't all that pertinent anymore given the desires of most in the community to blur that line.  It would be available to everyone.

For that matter, why do we even need to have this discussion at all?  Most of us Solr committers are Lucene committers.  We can simply start committing Solr code to Lucene such that in 6 months the whole discussion is moot and the three committers on Solr who aren't Lucene committers can earn their Lucene merit very quickly by patching the "Solr" portion of Lucene.  We can move all the code to it's appropriate place, add a contrib module for the WAR stuff and the response writers and voila, Solr is in Lucene, the dev mailing lists have merged by the fact that Solr dev would be defunct and all of the proposals in this vote are implemented simply by employing our commit privileges in a concerted way.  Yet, somehow, me thinks that isn't a good solution either, right?  Yet it is perfectly "legal" and is just as valid a solution as the "poaching" solution and in a lot of ways seems to be what Chris is proposing.

-Grant

Dropping private@.

On Mar 9, 2010, at 6:30 AM, mark harwood wrote:

> Another 2 cents late to the party.....
>
>> I believe this is a question of identity.  What is Lucene?
>
> Absolutely.
> I think one of the clearest differences in outlook between Lucene and Solr is in the support for distributed deployments. Solr clearly aims to support distributed deployments while Lucene is "just a library".
> Many index operations (faceting, search, top terms) that work in a distributed fashion must be written differently to a single-index counterpart.
> If we do aim to share any distribute-capable functionality will Lucene need a brand new set of abstractions to avoid binding directly to the Solr server platform? Is that at all realistic?
> I speak as someone else who needs to maintain a Lucene extension similar to Solr, but where using Solr is not the answer so am keen for Lucene to maintain independence.
>
> Another potential big difference is any functionality that is Solr-schema-aware. Again, would we need to introduce an abstraction for schemas?
>
> Maybe it's useful to consider what is fundamentally different between Solr and Lucene (I suggest schema vs no schema and distributed vs local) and use this to help put a limit on what functionality we consider sharing.
> If a function is untainted by a fundamental difference (e.g. Analyzers typically couldnt care less about schemas or distribution) then that is a candidate for sharing.
>
> At the end of this process we get a good idea about what really can be shared.

I agree.  I maintain both Lucene and Solr instances.  Sometimes I need things that are in Solr that are Lucene.  Sometimes I need things in Lucene that are in Solr.  In the Lucene instances I maintain/help with, I don't need the Solr server stuff.  So, to me, there will always need to be that distinction.  At the same time, it is very frustrating for me to write code that I know belongs in Lucene, but that I put into Solr for the sole fact that I need it for one of the Solr instances and simply can't afford to wait for Solr to be on the appropriate version of trunk.  Likewise, I may want something for Lucene from Solr but it is a fair amount of work to bring it up to the new Lucene APIs.

As for the sharing list, I started such a list on the other thread, but can duplicate here.

To me, there are at least the following:
1. Analyzers
2. Functions
3. Schema (although likely abstracted/reworked)
4. Warming/Reopen - this is hard code to get right and I've seen many people do it wrong.  It is also yet another area of duplication where something started in Solr b/c for years the Lucene community had no interest in donating code for it (incRef/decRef)
5. Faceting
6. Spatial

and on and on.  In fact, in my mind, it's pretty much everything other than stuff that is explicitly to do with Input/Output (Request Handlers, Response Writers)  and HTTP as the server mechanism.  Even with that list, though, I believe we can keep these separated enough that people can pick and choose.  In fact, your input, Mark, would be valuable in helping maintain that distinction.  As they say in the ASF, those who do, decide.

-Grant

Hi Mike,

>> As someone who works on both, I don't think it is fine.  Just look at the
>> function query mess.  Just look at the version mess.  It's very frustrating
>> as a developer and it makes me choose between two projects that I happen to
>> like equally, but for different reasons.  If I worked on Nutch, I would feel
>> the same way.
>
> But... Lucene should poach from external (eg non-Apache) projects, if
> the license works?
>
> Ie if some great analyzer is out there, and Robert spots it, and the
> license works, we should poach it?  (In fact he just did this w/
> Andrzej's Polish stemmer ;) ).

Yep. This is what I was talking about before when I was talking about
"insulation". Code duplication is a fact of software development, and
happens all the time in open source, ROTS, GOTS, OTS,
research/academia/whatever. It doesn't suffice to say it's bad in all cases,
nor is it always good either.

In this case, it maintains the separation between projects that are really
layered on top of one another (Lucene being the lower layer, and Solr being
the higher).

In addition, FWIW, I agree with Andrzej that to the best of my knowledge,
there is nothing wrong with doing so at the ASF, with proper attribution and
so long as the licenses are compatible.

>
> So we have something of a double standard...

Yep.

>
> And, ironically, I think it's the fact that there's so much committer
> overlap between Solr and Lucene that is causing this antagonism
> towards poaching.
>
> When in fact I think poaching, at a wider scale (across unrelated
> projects) is a very useful means for any healthy open source software
> to evolve.

Agreed. It allows sound innovative technology infusion and solutions to
develop over time, and then be integrated back into the operational fray
with reduced risk and cost.

>
> Why should Lucene be prevented from having a useful feature just
> because Solr happened to create it first?

IMO, it shouldn't.

Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I think the problem is political - and that leads to both technical
and political problems.
We came up with a largely political solution that should solve both.

We can't have a one way street of pulling everything interesting out
of Solr for Lucene, or poaching, or expanding Lucene's domain while
shrinking Solr's (just limit to "server stuff", etc).  Lucene and Solr
committers are headed down the road toward greater competition - but
with this proposal, we said we'd rather work together instead.

-Yonik

True.  There are features that aren't useful for every search.  But the
features in Lucene are meant for full text search, not for serving full
text search.  Maybe faceting was a bad example, it was the first that
came to mind and defines what many people use Solr for.

Lucene IMO is a full text search *library*.  When features are added to
it, that is the perspective that should be taken.  Does it work as a
general purpose indexing library?

I am all for adding in *very* useful features, especially when someone
else has done the work, as long as they fall into that boundary.  But
Solr isn't a search library, it is a search server.  Aren't those
separate responsibilities?  Should we take some of the things out of
Solr and put them into Lucene?  Absolutely.  Should we merge to do this.
  No, not IMO.

And Power is good in the right hands, but that is another discussion :)

Dennis

Ted Dunning wrote:
> There are scads of features of Lucene that are not useful for all
> applications (payloads, for one example, back compatibility for another).
>
> The point is that the option to use faceting or not would be *very* useful
> for all search applications.  Power is good, especially since somebody else
> has done the work already.
>
> On Mon, Mar 8, 2010 at 10:10 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote:
>
>> Faceting for example, great feature, but not useful in every full text
>> search.
>>
>

Hey Grant,

On 3/9/10 5:49 AM, "Grant Ingersoll" <[EMAIL PROTECTED]> wrote:

> For that matter, why do we even need to have this discussion at all?  Most of
> us Solr committers are Lucene committers.  We can simply start committing Solr
> code to Lucene such that in 6 months the whole discussion is moot and the
> three committers on Solr who aren't Lucene committers can earn their Lucene
> merit very quickly by patching the "Solr" portion of Lucene.

Sure, if folks agree on those patches and the community finds them useful,
and the patches follow the dev process of Lucene(-java), then so be it.
However, it seems like this could have been done already, no? Many of the
things you and others have discussed merging have been around for a while
besides spatial. Is it simply developers/resources that is lacking in
Lucene(-java) and time? Or are there other reasons? It sounds to me based on
the desire to sync up tests, to follow the same release schedule/etc., that
there are in fact, other reasons.

> We can move all
> the code to it's appropriate place, add a contrib module for the WAR stuff and
> the response writers and voila, Solr is in Lucene, the dev mailing lists have
> merged by the fact that Solr dev would be defunct and all of the proposals in
> this vote are implemented simply by employing our commit privileges in a
> concerted way.  Yet, somehow, me thinks that isn't a good solution either,
> right?  Yet it is perfectly "legal" and is just as valid a solution as the
> "poaching" solution and in a lot of ways seems to be what Chris is proposing.

Whether or not what you're saying is good or what I'm saying is good or not
will be decided by the community within Lucene(-java), as well as the one
within Solr. All I'm for is not circumventing that process, in any
direction. If what you suggest above happens in a concise, traceable,
beneficial way to both projects and communities, then I'm for that.

At the same time, I also favor insulation wherever possible and I personally
like the separation of the 2 projects. I have built 10s of projects that
have simply used Lucene as an API and had no need for Solr, and I've built
10s of projects where Solr made perfect sense. So, I appreciate their
separation. I also have a lot of experience in these types of situations as
I've been involved in 2-3 of them over the past few years at NASA in terms
of maintaining separation and merging projects/etc. There are quite a few
lessons learned that I have been trying to share but that have seemingly not
really been appreciated and that have been in my mind dismissed, rather than
discussed, through this process.

Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

It was late when I wrote that, maybe my analogy was not clear.  You are
echoing what I was trying to say that that Hadoop != Nutch and it
wouldn't have been as useful if it had only ever been viewed that way.
I think part of this discussion is looking at Lucene as needing things
that are beyond it.  That should be other projects.

Here is my logic FWIW:

Solr depends on Lucene.
Many other projects depend critically on Lucene
Not all of those projects depend on Solr
Solr and Lucene have different responsibilities
Therefore Solr != Lucene and should not merge dev.

Should Solr work more closely to move some of it pieces into Lucene if
they are applicable.  Yes.  To me that doesn't mean merge.

Dennis

Ted Dunning wrote:
> This logic escapes me.
>
> Nutch hatched Hadoop.  Hadoop was perceived to be of much broader utility
> than just for nutch so it was made more general and a separate project was
> formed.  Hadoop does not depend on Nutch.
>
> Lucene existed.  Solr was built to make it easier to use Lucene.  The
> developers of Solr built a bunch of stuff that was specific to server-ness
> and a bunch of stuff that would have general utility to many Lucene
> developers.  Solr depends critically on Lucene and can be seen as a Lucene
> wrapper.
>
> How does this analogy fit together?  Is it supposed to be Hadoop is to Nutch
> as Solr is to Lucene?  That seems so clearly wrong it can't be what you were
> saying.
>
> On Mon, Mar 8, 2010 at 10:10 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote:
>
>>> 3) For new Lucene features, there would be an effort to integrate it
>>> into Solr.
>> No.  Because by specializing towards Solr, or Nutch, or any of the hundred
>> other applications that use Lucene, it looses its general applicability.
>>  Where would Hadoop be if it never made it past Nutch?
>>
>

Hi Dennis,

> It was late when I wrote that, maybe my analogy was not clear.  You are
> echoing what I was trying to say that that Hadoop != Nutch and it
> wouldn't have been as useful if it had only ever been viewed that way.
> I think part of this discussion is looking at Lucene as needing things
> that are beyond it.  That should be other projects.
>
> Here is my logic FWIW:
>
> Solr depends on Lucene.
> Many other projects depend critically on Lucene
> Not all of those projects depend on Solr
> Solr and Lucene have different responsibilities
> Therefore Solr != Lucene and should not merge dev.
>
> Should Solr work more closely to move some of it pieces into Lucene if
> they are applicable.  Yes.  To me that doesn't mean merge.

+1, my sentiments exactly.

Cheers,
Chris

>
> Dennis
>
> Ted Dunning wrote:
>> This logic escapes me.
>>
>> Nutch hatched Hadoop.  Hadoop was perceived to be of much broader utility
>> than just for nutch so it was made more general and a separate project was
>> formed.  Hadoop does not depend on Nutch.
>>
>> Lucene existed.  Solr was built to make it easier to use Lucene.  The
>> developers of Solr built a bunch of stuff that was specific to server-ness
>> and a bunch of stuff that would have general utility to many Lucene
>> developers.  Solr depends critically on Lucene and can be seen as a Lucene
>> wrapper.
>>
>> How does this analogy fit together?  Is it supposed to be Hadoop is to Nutch
>> as Solr is to Lucene?  That seems so clearly wrong it can't be what you were
>> saying.
>>
>> On Mon, Mar 8, 2010 at 10:10 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote:
>>
>>>> 3) For new Lucene features, there would be an effort to integrate it
>>>> into Solr.
>>> No.  Because by specializing towards Solr, or Nutch, or any of the hundred
>>> other applications that use Lucene, it looses its general applicability.
>>>  Where would Hadoop be if it never made it past Nutch?
>>>
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

> besides spatial. Is it simply developers/resources that is lacking in
> Lucene(-java) and time? Or are there other reasons?

While its true my analysis patches to fix solr just sit there in JIRA
because Solr developers
are busy working on other tasks such as spatial, on the other hand,
Chris Male's spatial
patches to lucene just sit there in JIRA because Lucene developers are
busy working on
other tasks such as analysis.

So in my opinion, it would be nice if both projects tried to make the
best of our limited
 resources.

--
Robert Muir
[EMAIL PROTECTED]

Michael McCandless wrote:
> On Tue, Mar 9, 2010 at 5:10 AM, Andrzej Bialecki <[EMAIL PROTECTED]> wrote:
>
>> Re: Nutch components - those that are reusable in Lucene or Solr
>> contexts eventually find their way to respective projects, witness
>> e.g. CommonGrams.
>
> In fact I think this is a great example -- as far as I can tell,
> CommonGrams was poached from Nutch, into Solr, and then was
> nurtured/improved in both projects separately right?
>
> So.... can/should we freely poach across all our sub projects?

IMO yes.  Absolutely.  That is exactly what OSS is all about.  Find
something useful, improve upon it.

>
> It has obvious downsides (it's essentially a fork that will confuse
> those users that use both Solr & Lucene, in the short term, until
> things "stabilize" into a clean refactoring; it's double the dev; we
> must re-sync with time; etc.).

True.  OSS development is messy at times.  And it can take longer.

>
> But it has a massive upside: it means we don't rely only on "push"
> (Solr devs to push into Lucene or vice/versa).  We can also use "pull"
> (Lucene devs can pull pieces from Nutch/Solr into Lucene).  It becomes
> a 2-way street for "properly" factoring our shared code with time.
>
> If we had that freedom ("poaching is perfectly fine"), then,
> interested devs could freely "refactor" across sub projects.

There is nothing stopping any developer, committer or not from making
changes to any apache project including Nutch, Lucene, and Solr.
Merging doesn't change or improve that.  At best it makes it more
confusing where responsibilities lie.

Dennis

>
> Not having this freedom today, and not having merged dev, is stunting
> both Solr & Lucene's growth.
>
> Mike

On Tue, Mar 9, 2010 at 9:48 AM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:
> I have built 10s of projects that
> have simply used Lucene as an API and had no need for Solr, and I've built
> 10s of projects where Solr made perfect sense. So, I appreciate their
> separation.

As does everyone - which is why there will always be separate
downloads.  As a user, the only side affect you should see is an
improved Lucene and Solr.

Saying that Solr should move some stuff to Lucene for Lucene's
benefit, without regard to if it's actually benefitial to Solr, is a
non-starter.  The lucene/solr committers have been down that road
before.  The solution that most committers agreed would improve the
development of both projects is to merge development.

-Yonik

Hi Yonik,

>> I have built 10s of projects that
>> have simply used Lucene as an API and had no need for Solr, and I've built
>> 10s of projects where Solr made perfect sense. So, I appreciate their
>> separation.
>
> As does everyone - which is why there will always be separate
> downloads.  As a user, the only side affect you should see is an
> improved Lucene and Solr.

Developers make downloads. Software processes guide developers who are
producing those downloads. Policies guide the direction of a project. They
are intimately intertwined.

>
> Saying that Solr should move some stuff to Lucene for Lucene's
> benefit, without regard to if it's actually benefitial to Solr, is a
> non-starter.  

I'm not sure it's Solr's decision what the Lucene committers decide to move
to Lucene, neither is it Lucene's decision in the opposite direction. These
are all Apache projects, subprojects of the Lucene TLP. I'm not sure what
the debate is? If Solr wants elements from Lucene that aren't part of Solr
yet b/c Solr is relying on an old version of Lucene:

1. upgrade to Lucene trunk and address the issues it brings in Solr
2. duplicate the Lucene code in Solr, address any issues there, and then
contribute it back

I'd recommend the same to any project, regardless of what TLP it resides in,
and in many cases, where it's at the ASF, or Sourceforge, or wherever.

It seems kind of incestuous and an abuse of power to make the case that
"well because we're all committers on both projects, then this..." I keep
hearing a lot of talk about "hats", which in analogy means that though you
are one person you have different concerns/projects/etc. This is another
example of the need to maintain separate hats.

Cheers,
Chris
 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

> 2. duplicate the Lucene code in Solr, address any issues there, and then
> contribute it back
>

Not that I can stop anything, but -1 to any further analysis code
duplication. There has to be a better way.

Its stupid to duplicate the code. Its also stupid to just move the code.

In my opinion, if Solr has an analysis feature that belongs in lucene,
we want not just the code, but improvements, ideas, comments from solr
developers that have experience with it, we want to pay
attention to open Solr JIRA tickets against that feature, we want
things like the Solr example schema to have good "defaults" for any
potential improvements to it, and we want the wiki to reflect
additional improvements it supports.

--
Robert Muir
[EMAIL PROTECTED]

Hi Robert,

>> 2. duplicate the Lucene code in Solr, address any issues there, and then
>> contribute it back
>>
>
> Not that I can stop anything, but -1 to any further analysis code
> duplication. There has to be a better way.

There might be, but as a first start, duplication is a quick way to get
going and experiment. As solutions that evolve over time are matured, the
time can come for integration. Parallel tracks allows projects to move
forward operationally, and enforces insulation, loose coupling and other
properties.

>
> Its stupid to duplicate the code. Its also stupid to just move the code.

I'm not sure anything is "stupid" per-se. There are different approaches
which address different concerns.

>
> In my opinion, if Solr has an analysis feature that belongs in lucene,
> we want not just the code, but improvements, ideas, comments from solr
> developers that have experience with it, we want to pay
> attention to open Solr JIRA tickets against that feature, we want
> things like the Solr example schema to have good "defaults" for any
> potential improvements to it, and we want the wiki to reflect
> additional improvements it supports.

Then I think it's fair to say that those that want this can follow the
normal Apache process to do so:

* subscribe to the mailing lists
* post JIRA issues with patches
* get those patches committed
* become a committer, etc.

Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Mar 9, 2010, at 9:48 AM, Mattmann, Chris A (388J) wrote:

> Hey Grant,
>
> On 3/9/10 5:49 AM, "Grant Ingersoll" <[EMAIL PROTECTED]> wrote:
>
>> For that matter, why do we even need to have this discussion at all?  Most of
>> us Solr committers are Lucene committers.  We can simply start committing Solr
>> code to Lucene such that in 6 months the whole discussion is moot and the
>> three committers on Solr who aren't Lucene committers can earn their Lucene
>> merit very quickly by patching the "Solr" portion of Lucene.
>
> Sure, if folks agree on those patches and the community finds them useful,
> and the patches follow the dev process of Lucene(-java), then so be it.
> However, it seems like this could have been done already, no? Many of the
> things you and others have discussed merging have been around for a while
> besides spatial. Is it simply developers/resources that is lacking in
> Lucene(-java) and time? Or are there other reasons? It sounds to me based on
> the desire to sync up tests, to follow the same release schedule/etc., that
> there are in fact, other reasons.

Um, I'm a committer.  I've earned the right to apply patches that fit with the project and I've earned the merit to make that decision.  So have all the other committers.   Besides the fact, all I would be committing are the things people have already expressed an interest in anyway.

>
>> We can move all
>> the code to it's appropriate place, add a contrib module for the WAR stuff and
>> the response writers and voila, Solr is in Lucene, the dev mailing lists have
>> merged by the fact that Solr dev would be defunct and all of the proposals in
>> this vote are implemented simply by employing our commit privileges in a
>> concerted way.  Yet, somehow, me thinks that isn't a good solution either,
>> right?  Yet it is perfectly "legal" and is just as valid a solution as the
>> "poaching" solution and in a lot of ways seems to be what Chris is proposing.
>
> Whether or not what you're saying is good or what I'm saying is good or not
> will be decided by the community within Lucene(-java), as well as the one
> within Solr. All I'm for is not circumventing that process, in any
> direction. If what you suggest above happens in a concise, traceable,
> beneficial way to both projects and communities, then I'm for that.

No one is circumventing any process and the implication is just wrong.  We are having the discussion.  But even so, as a committer, my job is to work within community to fix/improve the code.  Right now, I see lots of room for improvement in Lucene by integrating some of those things from Solr (and Nutch) while keeping Lucene, Solr and Nutch whole from an end user perspective.  At the same time, I want to see Solr and Nutch whole.   Any other implication is simply wrong.

>
> At the same time, I also favor insulation wherever possible and I personally
> like the separation of the 2 projects. I have built 10s of projects that
> have simply used Lucene as an API and had no need for Solr, and I've built
> 10s of projects where Solr made perfect sense.

And how at all would those 10 projects be affected at all?  Please read the proposal again.  It's not like there is going to be some uber JAR.  I won't let it happen as I have more than 10 projects that are pure Lucene.  Part of my day job is supporting Lucene.  I've spent the past 5 years of my life donating to Lucene, and so have many others.  The argument is simply invalid and has been refuted so many times now by ALL those who actually do the work that I don't understand why you insist on bringing it up over and over again.
> So, I appreciate their
> separation. I also have a lot of experience in these types of situations as
> I've been involved in 2-3 of them over the past few years at NASA in terms
> of maintaining separation and merging projects/etc. There are quite a few
> lessons learned that I have been trying to share but that have seemingly not
> really been appreciated and that have been in my mind dismissed, rather than

I'd hardly say they've been dismissed.  This isn't about you, it's about what is best for the project.  You have one opinion, that, by the face of the votes, is in the minority.  It doesn't make the majority right and you wrong.  In fact, those in the majority are trying to answer your concerns and come up with a better suggestion.  This is in fact the process and it is how the ASF works.  This is one of the great things about the Lucene community.  We have real discussions about the issues.

-Grant

I think many of the objections I've seen so far come from the fact that people don't really know what Solr is.  Solr is much more than simply a "server" around Lucene.

Look at the other thread.  Here's a minimal list of things that a very large chunk of people who writes a Lucene app for production has to do:

1. Analyzers
2. Functions
3. Schema (although likely abstracted/reworked)
4. Warming/Reopen - this is hard code to get right and I've seen many people do it wrong.  It is also yet another area of duplication where something started in Solr b/c for years the Lucene community had no interest in donating code for it (incRef/decRef)
5. Faceting

If someone came in and contributed all of those things to Lucene, there would be no objection.  Simply the fact that Solr has other things around it doesn't mean people have to use them and no one is proposing some Uber JAR.
On Mar 9, 2010, at 10:13 AM, Mattmann, Chris A (388J) wrote:

> Hi Yonik,
>
>>> I have built 10s of projects that
>>> have simply used Lucene as an API and had no need for Solr, and I've built
>>> 10s of projects where Solr made perfect sense. So, I appreciate their
>>> separation.
>>
>> As does everyone - which is why there will always be separate
>> downloads.  As a user, the only side affect you should see is an
>> improved Lucene and Solr.
>
> Developers make downloads. Software processes guide developers who are
> producing those downloads. Policies guide the direction of a project. They
> are intimately intertwined.
>
>>
>> Saying that Solr should move some stuff to Lucene for Lucene's
>> benefit, without regard to if it's actually benefitial to Solr, is a
>> non-starter.  
>
> I'm not sure it's Solr's decision what the Lucene committers decide to move
> to Lucene, neither is it Lucene's decision in the opposite direction. These
> are all Apache projects, subprojects of the Lucene TLP. I'm not sure what
> the debate is? If Solr wants elements from Lucene that aren't part of Solr
> yet b/c Solr is relying on an old version of Lucene:
>
> 1. upgrade to Lucene trunk and address the issues it brings in Solr
> 2. duplicate the Lucene code in Solr, address any issues there, and then
> contribute it back
>
> I'd recommend the same to any project, regardless of what TLP it resides in,
> and in many cases, where it's at the ASF, or Sourceforge, or wherever.
>
> It seems kind of incestuous and an abuse of power to make the case that
> "well because we're all committers on both projects, then this..." I keep
> hearing a lot of talk about "hats", which in analogy means that though you
> are one person you have different concerns/projects/etc. This is another
> example of the need to maintain separate hats.
>
> Cheers,
> Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: [EMAIL PROTECTED]
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search

I agree.  Most of those things can/should be moved into Lucene.  That
doesn't necessitate merging.  Separate responsibilities.

 > For that matter, why do we even need to have this discussion at all?
  > Most of us Solr committers are Lucene committers.  We can simply start
 > committing Solr code to Lucene such that in 6 months the whole
 > discussion is moot and the three committers on Solr who aren't Lucene
 > committers can earn their Lucene merit very quickly by patching the
 > "Solr" portion of Lucene.  We can move all the code to it's
 > appropriate place, add a contrib module for the WAR stuff and the
 > response writers and voila, Solr is in Lucene, the dev mailing lists
 > have merged by the fact that Solr dev would be defunct and all of the
 > proposals in this vote are implemented simply by employing our commit
 > privileges in a concerted way.

Am I reading you right.  Are you are proposing a hostile takeover of the
Lucene project?  Even being committers there needs to be discussion with
the community about the best path.  Or are you suggesting we bypass
discussion?  I am now even more concerned that merging is not the right
way to go.

Dennis

Grant Ingersoll wrote:
> I think many of the objections I've seen so far come from the fact that people don't really know what Solr is.  Solr is much more than simply a "server" around Lucene.
>
> Look at the other thread.  Here's a minimal list of things that a very large chunk of people who writes a Lucene app for production has to do:
>
> 1. Analyzers
> 2. Functions
> 3. Schema (although likely abstracted/reworked)
> 4. Warming/Reopen - this is hard code to get right and I've seen many people do it wrong.  It is also yet another area of duplication where something started in Solr b/c for years the Lucene community had no interest in donating code for it (incRef/decRef)
> 5. Faceting
>
> If someone came in and contributed all of those things to Lucene, there would be no objection.  Simply the fact that Solr has other things around it doesn't mean people have to use them and no one is proposing some Uber JAR.
>
>
> On Mar 9, 2010, at 10:13 AM, Mattmann, Chris A (388J) wrote:
>
>> Hi Yonik,
>>
>>>> I have built 10s of projects that
>>>> have simply used Lucene as an API and had no need for Solr, and I've built
>>>> 10s of projects where Solr made perfect sense. So, I appreciate their
>>>> separation.
>>> As does everyone - which is why there will always be separate
>>> downloads.  As a user, the only side affect you should see is an
>>> improved Lucene and Solr.
>> Developers make downloads. Software processes guide developers who are
>> producing those downloads. Policies guide the direction of a project. They
>> are intimately intertwined.
>>
>>> Saying that Solr should move some stuff to Lucene for Lucene's
>>> benefit, without regard to if it's actually benefitial to Solr, is a
>>> non-starter.  
>> I'm not sure it's Solr's decision what the Lucene committers decide to move
>> to Lucene, neither is it Lucene's decision in the opposite direction. These
>> are all Apache projects, subprojects of the Lucene TLP. I'm not sure what
>> the debate is? If Solr wants elements from Lucene that aren't part of Solr
>> yet b/c Solr is relying on an old version of Lucene:
>>
>> 1. upgrade to Lucene trunk and address the issues it brings in Solr
>> 2. duplicate the Lucene code in Solr, address any issues there, and then
>> contribute it back
>>
>> I'd recommend the same to any project, regardless of what TLP it resides in,
>> and in many cases, where it's at the ASF, or Sourceforge, or wherever.
>>
>> It seems kind of incestuous and an abuse of power to make the case that
>> "well because we're all committers on both projects, then this..." I keep
>> hearing a lot of talk about "hats", which in analogy means that though you
>> are one person you have different concerns/projects/etc. This is another
>> example of the need to maintain separate hats.
>>
>> Cheers,
>> Chris
>>

> On Mar 9, 2010, at 9:48 AM, Mattmann, Chris A (388J) wrote:
>
>> Hey Grant,
>>
>> On 3/9/10 5:49 AM, "Grant Ingersoll" <[EMAIL PROTECTED]> wrote:
>>
>>> For that matter, why do we even need to have this discussion at all?  Most
>>> of
>>> us Solr committers are Lucene committers.  We can simply start committing
>>> Solr
>>> code to Lucene such that in 6 months the whole discussion is moot and the
>>> three committers on Solr who aren't Lucene committers can earn their Lucene
>>> merit very quickly by patching the "Solr" portion of Lucene.
>>
>> Sure, if folks agree on those patches and the community finds them useful,
>> and the patches follow the dev process of Lucene(-java), then so be it.
>> However, it seems like this could have been done already, no? Many of the
>> things you and others have discussed merging have been around for a while
>> besides spatial. Is it simply developers/resources that is lacking in
>> Lucene(-java) and time? Or are there other reasons? It sounds to me based on
>> the desire to sync up tests, to follow the same release schedule/etc., that
>> there are in fact, other reasons.
>
> Um, I'm a committer.  I've earned the right to apply patches that fit with the
> project and I've earned the merit to make that decision.  So have all the
> other committers.   Besides the fact, all I would be committing are the things
> people have already expressed an interest in anyway.

Then it's not an issue, right? Additionally, you didn't really answer my
question to what the cause of it not happening is (resources/time/process?)

>
>>
>>> We can move all
>>> the code to it's appropriate place, add a contrib module for the WAR stuff
>>> and
>>> the response writers and voila, Solr is in Lucene, the dev mailing lists
>>> have
>>> merged by the fact that Solr dev would be defunct and all of the proposals
>>> in
>>> this vote are implemented simply by employing our commit privileges in a
>>> concerted way.  Yet, somehow, me thinks that isn't a good solution either,
>>> right?  Yet it is perfectly "legal" and is just as valid a solution as the
>>> "poaching" solution and in a lot of ways seems to be what Chris is
>>> proposing.
>>
>> Whether or not what you're saying is good or what I'm saying is good or not
>> will be decided by the community within Lucene(-java), as well as the one
>> within Solr. All I'm for is not circumventing that process, in any
>> direction. If what you suggest above happens in a concise, traceable,
>> beneficial way to both projects and communities, then I'm for that.
>
> No one is circumventing any process and the implication is just wrong.  We are
> having the discussion.  But even so, as a committer, my job is to work within
> community to fix/improve the code.  Right now, I see lots of room for
> improvement in Lucene by integrating some of those things from Solr (and
> Nutch) while keeping Lucene, Solr and Nutch whole from an end user
> perspective.  At the same time, I want to see Solr and Nutch whole.   Any
> other implication is simply wrong.

Sweeping proposals with TBDs leave room for implications. Smaller, concrete
steps do not.

>
>>
>> At the same time, I also favor insulation wherever possible and I personally
>> like the separation of the 2 projects. I have built 10s of projects that
>> have simply used Lucene as an API and had no need for Solr, and I've built
>> 10s of projects where Solr made perfect sense.
>
> And how at all would those 10 projects be affected at all?  Please read the
> proposal again.  It's not like there is going to be some uber JAR.

I think the point that you and some others are missing is that JARs are not
the only artifact of a system. Just as you develop in Lucene "officially" as
an ASF committer for Lucene(-java), and just as you and others develop in
Solr "officially" as Solr ASF committers, it doesn't mean others don't also
develop using the same code on their own projects. JARs are not the only
artifact that is being reused.

> I won't

It's extremely unclear to me about how you can be so sure about how
something will work when there have been many questions, the proposal itself
includes TBDs or things as Yonik put it that "will be figured out later."
Well I'm sorry you feel that way. However, like I said it seems to be like
the discussion of the real issues is only happening recently over the past
few days. There has been a lot of effort to push this through before then.
Appreciate it, in fact I'm not looking to assign right and wrong.
The recent discussions have been picked up and I am feeling like we're
starting to discuss the issues so that's a good thing.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

> There might be, but as a first start, duplication is a quick way to get
> going and experiment. As solutions that evolve over time are matured, the
> time can come for integration. Parallel tracks allows projects to move
> forward operationally, and enforces insulation, loose coupling and other
> properties.

Unfortunately, this experiment has already happened and has failed.

Instead it just creates more work, especially when it comes time for
code maintenance. This is one reason why Solr is still using
deprecated analysis APIs and one reason why they cannot use Lucene
trunk.

If there weren't so much duplication, then doing efforts such as this
would be easier, instead of having to convert two SynonymFilters we
would only have to convert one.
--
Robert Muir
[EMAIL PROTECTED]

Here my vote:

+1 for the latest proposal to merge the development.

I am still against the requirement that all changes in Lucene need all tests to pass in solr, but that can be discussed later. I would like to simply open an issue then, if a test does not pass and let the Solr people fix it (applies to all bugs in solr's tests). Also releases at the same time for both projects should not be coupled. Each project should be able to release when they think it's time.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]

> -----Original Message-----
> From: Yonik Seeley [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, March 09, 2010 3:12 AM
> To: [EMAIL PROTECTED]
> Subject: [VOTE] merge lucene/solr development (take 3)
>
> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr
> development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vot
> e_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
> > Subject: Merge the development of Solr/Lucene (take 2)
> > A new vote, that slightly changes proposal from last vote (adding
> only
> > that Lucene can cut a release even if Solr doesn't):
> >
> >  * Merging the dev lists into a single list.
> >
> >  * Merging committers.
> >
> >  * When any change is committed (to a module that "belongs to" Solr
> or
> >    to Lucene), all tests must pass.
> >
> >  * Release details will be decided by dev community, but, Lucene may
> >    release without Solr.
> >
> >  * Modulariize the sources: pull things out of Lucene's core (break
> >    out query parser, move all core queries & analyzers under their
> >    contrib counterparts), pull things out of Solr's core (analyzers,
> >    queries).
> >
> > These things would not change:
> >
> >  * Besides modularizing (above), the source code would remain
> factored
> >    into separate dirs/modules the way it is now.
> >
> >  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
> >    issues).
> >
> >  * User's lists remain separate.
> >
> >  * Web sites remain separate.
> >
> >  * Release artifacts/jars remain separate.

On Tue, Mar 9, 2010 at 11:00 AM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:
> However, like I said it seems to be like
> the discussion of the real issues is only happening recently over the past
> few days.

This certainly isn't new territory for lucene/solr devs though - the
issue of what belongs in Solr and what belongs in Lucene, and problems
around pulling out schema and faceting and putting it in Lucene have
come up before (also in lengthy threads).

-Yonik

Hi Robert,

>> There might be, but as a first start, duplication is a quick way to get
>> going and experiment. As solutions that evolve over time are matured, the
>> time can come for integration. Parallel tracks allows projects to move
>> forward operationally, and enforces insulation, loose coupling and other
>> properties.
>
> Unfortunately, this experiment has already happened and has failed.
>
> Instead it just creates more work, especially when it comes time for
> code maintenance. This is one reason why Solr is still using
> deprecated analysis APIs and one reason why they cannot use Lucene
> trunk.

Can you provide more detail on how it's failed? Did it fail because Solr
wasn't able to upgrade to a newer Lucene that would fix the deprecations? If
so, what were the reasons?

>
> If there weren't so much duplication, then doing efforts such as this
> would be easier, instead of having to convert two SynonymFilters we
> would only have to convert one.

With which "hat" on? Where are you converting them? I'm guessing you mean
one for Solr and one for Lucene(-java), right?

I hear you on the duplication of patches/etc. Sadly, it's a trade in SE to
maintain good separation of concerns, which provides other benefits
(identification of load bearing walls; insulation of code changes so that
upstream or downstream providers aren't well affected, etc. etc.)

One potential solution to this was what was originally proposed by Mike: a
shared analyzers, that then Lucene(-java), Solr, and Nutch can choose to
depend on. That might help.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Mar 9, 2010, at 10:59 AM, Dennis Kubes wrote:

> I agree.  Most of those things can/should be moved into Lucene.  That doesn't necessitate merging.  Separate responsibilities.
>
> > For that matter, why do we even need to have this discussion at all?  > Most of us Solr committers are Lucene committers.  We can simply start
> > committing Solr code to Lucene such that in 6 months the whole
> > discussion is moot and the three committers on Solr who aren't Lucene
> > committers can earn their Lucene merit very quickly by patching the
> > "Solr" portion of Lucene.  We can move all the code to it's
> > appropriate place, add a contrib module for the WAR stuff and the
> > response writers and voila, Solr is in Lucene, the dev mailing lists
> > have merged by the fact that Solr dev would be defunct and all of the
> > proposals in this vote are implemented simply by employing our commit
> > privileges in a concerted way.
>
> Am I reading you right.  Are you are proposing a hostile takeover of the Lucene project?  Even being committers there needs to be discussion with the community about the best path.  Or are you suggesting we bypass discussion?  I am now even more concerned that merging is not the right way to go.
>

No.  Would you please re-read it and not quote me out of context.  You left the next sentence off, which is of course the vital one.

Hey Yonik,

>> However, like I said it seems to be like
>> the discussion of the real issues is only happening recently over the past
>> few days.
>
> This certainly isn't new territory for lucene/solr devs though - the
> issue of what belongs in Solr and what belongs in Lucene, and problems
> around pulling out schema and faceting and putting it in Lucene have
> come up before (also in lengthy threads).

I hear ya. I've seen some on solr-{user|dev}@ within the past months.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Mar 9, 2010, at 11:28 AM, Grant Ingersoll wrote:

>
> On Mar 9, 2010, at 10:59 AM, Dennis Kubes wrote:
>
>> I agree.  Most of those things can/should be moved into Lucene.  That doesn't necessitate merging.  Separate responsibilities.
>>
>>> For that matter, why do we even need to have this discussion at all?  > Most of us Solr committers are Lucene committers.  We can simply start
>>> committing Solr code to Lucene such that in 6 months the whole
>>> discussion is moot and the three committers on Solr who aren't Lucene
>>> committers can earn their Lucene merit very quickly by patching the
>>> "Solr" portion of Lucene.  We can move all the code to it's
>>> appropriate place, add a contrib module for the WAR stuff and the
>>> response writers and voila, Solr is in Lucene, the dev mailing lists
>>> have merged by the fact that Solr dev would be defunct and all of the
>>> proposals in this vote are implemented simply by employing our commit
>>> privileges in a concerted way.
>>
>> Am I reading you right.  Are you are proposing a hostile takeover of the Lucene project?  Even being committers there needs to be discussion with the community about the best path.  Or are you suggesting we bypass discussion?  I am now even more concerned that merging is not the right way to go.
>>
>
> No.  Would you please re-read it and not quote me out of context.  You left the next sentence off, which is of course the vital one.

Namely:
"Yet, somehow, me thinks that isn't a good solution either, right?  Yet it is perfectly "legal" and is just as valid a solution as the "poaching" solution and in a lot of ways seems to be what Chris is proposing."

My point is, if people would just step back for a minute and remove the labels of Solr and Lucene and look at the code, the discussion would be a whole lot different.  Furthermore, if they stepped back and looked at the actual people who do the actual work on Lucene and Solr, you would see that these separations are slowing down both Lucene and Solr and hurting them more than helping.   This has been expressed time and time again by committers in both Lucene and in Solr and even in Nutch, including many committers who do not even work on Solr.  

-Grant

On Mar 9, 2010, at 11:29 AM, Mattmann, Chris A (388J) wrote:

> Hey Yonik,
>
>>> However, like I said it seems to be like
>>> the discussion of the real issues is only happening recently over the past
>>> few days.
>>
>> This certainly isn't new territory for lucene/solr devs though - the
>> issue of what belongs in Solr and what belongs in Lucene, and problems
>> around pulling out schema and faceting and putting it in Lucene have
>> come up before (also in lengthy threads).
>
> I hear ya. I've seen some on solr-{user|dev}@ within the past months.

I know it dates me, but:  s/months/years/  :-)

-Grant

On 3/9/10 7:35 AM, Robert Muir wrote:
>> 2. duplicate the Lucene code in Solr, address any issues there, and then
>> contribute it back
>>
>>      
> Not that I can stop anything, but -1 to any further analysis code
> duplication. There has to be a better way.
>
> Its stupid to duplicate the code. Its also stupid to just move the code.
>
>    

I agree with that.  No matter if this dev-merge vote passes or not, we
still want a separate analysis module, right?  Gathering all the
analyzers from the different projects and putting them into one place
was Mike's first proposal and I believe everyone was excited about it.

So how about we start with that, considering that we want to do it
anyway?  It seems like this solves the immediate problem Robert and
others are having with working on the analyzer patches and will help us
learn about what such a re-structuring entails.  And then, maybe in a
few weeks or months, we can discuss again if more modules should be
factored out or if we still see the need for merging the Solr and Lucene
devs.
It seems like the analysis house is currently the only one on fire.

  Michael

On Tue, Mar 9, 2010 at 10:01 AM, Mattmann, Chris A (388J)
<[EMAIL PROTECTED]> wrote:

> I think the point that you and some others are
> missing is that JARs are not the only artifact of a system.
This is the critical point.  The Solr and Lucene sources must be kept seprate for those of us that reference the Lucene source but have no use whatsoever for the JARs or the Solr source.
- Neal

Haha it dates me too but that's OK I know I'm a young-un! :)

Cheers,
Chris
On 3/9/10 8:35 AM, "Grant Ingersoll" <[EMAIL PROTECTED]> wrote:

On Mar 9, 2010, at 11:29 AM, Mattmann, Chris A (388J) wrote:

> Hey Yonik,
>
>>> However, like I said it seems to be like
>>> the discussion of the real issues is only happening recently over the past
>>> few days.
>>
>> This certainly isn't new territory for lucene/solr devs though - the
>> issue of what belongs in Solr and what belongs in Lucene, and problems
>> around pulling out schema and faceting and putting it in Lucene have
>> come up before (also in lengthy threads).
>
> I hear ya. I've seen some on solr-{user|dev}@ within the past months.

I know it dates me, but:  s/months/years/  :-)

-Grant

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

On Tue, Mar 9, 2010 at 11:35 AM, Michael Busch <[EMAIL PROTECTED]> wrote:
> No matter if this dev-merge vote passes or not, we still
> want a separate analysis module, right?

No.  That's the point of the dev merge - to allow free movement of
source code that benefits both projects.

-Yonik

On 3/9/10 8:41 AM, Yonik Seeley wrote:
> On Tue, Mar 9, 2010 at 11:35 AM, Michael Busch<[EMAIL PROTECTED]>  wrote:
>    
>> No matter if this dev-merge vote passes or not, we still
>> want a separate analysis module, right?
>>      
> No.  That's the point of the dev merge - to allow free movement of
> source code that benefits both projects.
>
> -Yonik
>
>    

Now I'm confused. This is part of the proposal:

>  Modulariize the sources: pull things out of Lucene's core (break
>      out query parser, move all core queries&  analyzers under their
>      contrib counterparts), pull things out of Solr's core (analyzers,
>      queries).
  Michael

From what I see I don't think we are going to agree on this.  Some of
us  believe there should be clear separations, some of us don't.  Let's
move on.

My vote is still a -1 on this.  If that means I get overruled, so be it.
  I have brought up what I believe to be important points in this
discussion.  I think we are just looping now.

Dennis

Yonik Seeley wrote:
> On Tue, Mar 9, 2010 at 11:35 AM, Michael Busch <[EMAIL PROTECTED]> wrote:
>> No matter if this dev-merge vote passes or not, we still
>> want a separate analysis module, right?
>
> No.  That's the point of the dev merge - to allow free movement of
> source code that benefits both projects.
>
> -Yonik

I'm still +1 for merging Solr/Lucene dev.

I think poaching, when we have so much that needs to be shared, is
going to cause far more problems than it'll solve.  It's not the right
tool for [this] job.

I do think poaching is good & legitimate tool when it's less code (eg
the CommonGrams case), so, we should do both ;)

Mike

On Tue, Mar 9, 2010 at 8:49 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote:
>
> On Mar 9, 2010, at 8:21 AM, Michael McCandless wrote:
>
>> On Tue, Mar 9, 2010 at 7:21 AM, Grant Ingersoll <[EMAIL PROTECTED]> wrote:
>>
>>>> If we had that freedom ("poaching is perfectly fine"), then,
>>>> interested devs could freely "refactor" across sub projects.
>>>
>>> As someone who works on both, I don't think it is fine.  Just look at the function query mess.  Just look at the version mess.  It's very frustrating as a developer and it makes me choose between two projects that I happen to like equally, but for different reasons.  If I worked on Nutch, I would feel the same way.
>>
>> But... Lucene should poach from external (eg non-Apache) projects, if
>> the license works?
>>
>> Ie if some great analyzer is out there, and Robert spots it, and the
>> license works, we should poach it?  (In fact he just did this w/
>> Andrzej's Polish stemmer ;) ).
>
> I'd prefer "donate" to poach, but, realize that isn't always the case.
>
>
>>
>> So we have something of a double standard...
>>
>> And, ironically, I think it's the fact that there's so much committer
>> overlap between Solr and Lucene that is causing this antagonism
>> towards poaching.
>>
>> When in fact I think poaching, at a wider scale (across unrelated
>> projects) is a very useful means for any healthy open source software
>> to evolve.
>>
>> Why should Lucene be prevented from having a useful feature just
>> because Solr happened to create it first?
>
> But why should I be forced to maintain two versions due to some arbitrary code separation?  And why should you force a good chunk of us to do a whole lot of extra work simply because of some arbitrary code separation?  Here, it is the Lucene PMC that releases code and it is just silly that with all of this overlap at the committer level we still have this duplication.   I can't speak for the external projects (I don't believe any of them have even responded here other than Jackrabbit), but if they don't like it, they should get more involved in the community and work to be committers.
>
> At any rate, this is exactly why merging makes sense.  You would no longer have this issue of "first".  I would no longer have to choose where to add my spatial work based on some arbitrary line that someone drew in the sand that isn't all that pertinent anymore given the desires of most in the community to blur that line.  It would be available to everyone.
>
> For that matter, why do we even need to have this discussion at all?  Most of us Solr committers are Lucene committers.  We can simply start committing Solr code to Lucene such that in 6 months the whole discussion is moot and the three committers on Solr who aren't Lucene committers can earn their Lucene merit very quickly by patching the "Solr" portion of Lucene.  We can move all the code to it's appropriate place, add a contrib module for the WAR stuff and the response writers and voila, Solr is in Lucene, the dev mailing lists have merged by the fact that Solr dev would be defunct and all of the proposals in this vote are implemented simply by employing our commit privileges in a concerted way.  Yet, somehow, me thinks that isn't a good solution either, right?  Yet it is perfectly "legal" and is just as valid a solution as the "poaching" solution and in a lot of ways seems to be what Chris is proposing.
>
> -Grant
>
>
>
>
>
>
>

>
> Can you provide more detail on how it's failed? Did it fail because Solr
> wasn't able to upgrade to a newer Lucene that would fix the deprecations? If
> so, what were the reasons?
>

Its just more work. Besides the duplication functionality itself, in
Lucene we (mostly Uwe) developed code to assist with tasks like this.
For example, we have utility code that makes it easier to maintain
backwards compatibility, test harnesses, etc.

The job still isn't done, but to do it right, it only makes sense to
re-use these resources so its done safely and more effectively. But
this would require even more duplication!!!

>
> With which "hat" on? Where are you converting them? I'm guessing you mean
> one for Solr and one for Lucene(-java), right?

Both Solr and Lucene. I'm not wearing a hat in this case: its just
something that needs to be done and we are in it together.
--
Robert Muir
[EMAIL PROTECTED]

Hello,

(just using Yonik's email to reply, but my comments are more general)
----- Original Message ----
> From: Yonik Seeley <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Tue, March 9, 2010 10:04:20 AM
> Subject: Re: [VOTE] merge lucene/solr development (take 3)
>
> On Tue, Mar 9, 2010 at 9:48 AM, Mattmann, Chris A (388J)
> wrote:
> > I have built 10s of projects that
> > have simply used Lucene as an API and had no need for Solr, and I've built
> > 10s of projects where Solr made perfect sense. So, I appreciate their
> > separation.
>
> As does everyone - which is why there will always be separate
> downloads.  As a user, the only side affect you should see is an
> improved Lucene and Solr.
>
> Saying that Solr should move some stuff to Lucene for Lucene's
> benefit, without regard to if it's actually benefitial to Solr, is a
> non-starter.  The lucene/solr committers have been down that road
> before.  The solution that most committers agreed would improve the
> development of both projects is to merge development.

* I'd completely understand the "non-starter" part if Lucene and Solr had disjoint sets of committers.  But that's not the case.

* Which is why I (like a few others) don't see why this whole thing cannot be solved by "better discussion of what to develop where from the get-go"

* Whenever people listed features built in Solr that really should have been in Lucene, I wondered "so why were not they developed in Lucene in the first place?"  Again, this should be possible because the same person can commit to both projects.

* I hear Grant's explanation on wanting something in Solr ASAP and not wanting to commit that something to Lucene (even though it logically belongs there) because Solr is not on Lucene trunk, but isn't this just a matter of getting "Lucene trunk nightly -> Solr trunk lib in svn" process going?

* Ian is 100% right.  This stuff clearly requires more discussion and a proper VOTE should wait a week or so.

Otis

* Re poaching (aka cross-project refactoring) - I think this is the way to go.  I think this is normal evolution of OSS projects.  I think this should be done if the functionality was not committed to the best (lowest common denominator?) project from the beginning, as in all the Solr/Lucene examples brought up

* I think Grant may be right.  We don't need this discussion.  Because the Solr/Lucene developer overlap is excellent, why not just start moving selected Solr code to new Lucene modules, just like Mike proposed we move Analysis from Lucene core to a new Lucene module?

* What do people think about doing what I wrote above as step 1 in this whole process?  When that is done in N months, we can see if we can improve on it?  This would also fit "progress, not perfection" mantra.

Otis
----- Original Message ----
> From: Otis Gospodnetic <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Tue, March 9, 2010 12:23:59 PM
> Subject: Re: [VOTE] merge lucene/solr development (take 3)
>
> Hello,
>
> (just using Yonik's email to reply, but my comments are more general)
>
>
> ----- Original Message ----
> > From: Yonik Seeley
> > To: [EMAIL PROTECTED]
> > Sent: Tue, March 9, 2010 10:04:20 AM
> > Subject: Re: [VOTE] merge lucene/solr development (take 3)
> >
> > On Tue, Mar 9, 2010 at 9:48 AM, Mattmann, Chris A (388J)
> > wrote:
> > > I have built 10s of projects that
> > > have simply used Lucene as an API and had no need for Solr, and I've built
> > > 10s of projects where Solr made perfect sense. So, I appreciate their
> > > separation.
> >
> > As does everyone - which is why there will always be separate
> > downloads.  As a user, the only side affect you should see is an
> > improved Lucene and Solr.
> >
> > Saying that Solr should move some stuff to Lucene for Lucene's
> > benefit, without regard to if it's actually benefitial to Solr, is a
> > non-starter.  The lucene/solr committers have been down that road
> > before.  The solution that most committers agreed would improve the
> > development of both projects is to merge development.
>
> * I'd completely understand the "non-starter" part if Lucene and Solr had
> disjoint sets of committers.  But that's not the case.
>
> * Which is why I (like a few others) don't see why this whole thing cannot be
> solved by "better discussion of what to develop where from the get-go"
>
> * Whenever people listed features built in Solr that really should have been in
> Lucene, I wondered "so why were not they developed in Lucene in the first
> place?"  Again, this should be possible because the same person can commit to
> both projects.
>
> * I hear Grant's explanation on wanting something in Solr ASAP and not wanting
> to commit that something to Lucene (even though it logically belongs there)
> because Solr is not on Lucene trunk, but isn't this just a matter of getting
> "Lucene trunk nightly -> Solr trunk lib in svn" process going?
>
> * Ian is 100% right.  This stuff clearly requires more discussion and a proper
> VOTE should wait a week or so.
>
> Otis

> * I think Grant may be right.  We don't need this discussion.  Because the Solr/Lucene developer overlap is excellent, why not just start moving selected Solr code to new Lucene modules, just like Mike proposed we move Analysis from Lucene core to a new Lucene module?

The underlying hesitation could be that the initial proposal sounds
like a press release.  With all the discussing, a few modules could
already have been moved (given the number of keys pressed to enter the
emails).

On Tue, Mar 9, 2010 at 9:38 AM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
> * Re poaching (aka cross-project refactoring) - I think this is the way to go.  I think this is normal evolution of OSS projects.  I think this should be done if the functionality was not committed to the best (lowest common denominator?) project from the beginning, as in all the Solr/Lucene examples brought up
>
> * I think Grant may be right.  We don't need this discussion.  Because the Solr/Lucene developer overlap is excellent, why not just start moving selected Solr code to new Lucene modules, just like Mike proposed we move Analysis from Lucene core to a new Lucene module?
>
> * What do people think about doing what I wrote above as step 1 in this whole process?  When that is done in N months, we can see if we can improve on it?  This would also fit "progress, not perfection" mantra.
>
> Otis
>
>
>
>
> ----- Original Message ----
>> From: Otis Gospodnetic <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Tue, March 9, 2010 12:23:59 PM
>> Subject: Re: [VOTE] merge lucene/solr development (take 3)
>>
>> Hello,
>>
>> (just using Yonik's email to reply, but my comments are more general)
>>
>>
>> ----- Original Message ----
>> > From: Yonik Seeley
>> > To: [EMAIL PROTECTED]
>> > Sent: Tue, March 9, 2010 10:04:20 AM
>> > Subject: Re: [VOTE] merge lucene/solr development (take 3)
>> >
>> > On Tue, Mar 9, 2010 at 9:48 AM, Mattmann, Chris A (388J)
>> > wrote:
>> > > I have built 10s of projects that
>> > > have simply used Lucene as an API and had no need for Solr, and I've built
>> > > 10s of projects where Solr made perfect sense. So, I appreciate their
>> > > separation.
>> >
>> > As does everyone - which is why there will always be separate
>> > downloads.  As a user, the only side affect you should see is an
>> > improved Lucene and Solr.
>> >
>> > Saying that Solr should move some stuff to Lucene for Lucene's
>> > benefit, without regard to if it's actually benefitial to Solr, is a
>> > non-starter.  The lucene/solr committers have been down that road
>> > before.  The solution that most committers agreed would improve the
>> > development of both projects is to merge development.
>>
>> * I'd completely understand the "non-starter" part if Lucene and Solr had
>> disjoint sets of committers.  But that's not the case.
>>
>> * Which is why I (like a few others) don't see why this whole thing cannot be
>> solved by "better discussion of what to develop where from the get-go"
>>
>> * Whenever people listed features built in Solr that really should have been in
>> Lucene, I wondered "so why were not they developed in Lucene in the first
>> place?"  Again, this should be possible because the same person can commit to
>> both projects.
>>
>> * I hear Grant's explanation on wanting something in Solr ASAP and not wanting
>> to commit that something to Lucene (even though it logically belongs there)
>> because Solr is not on Lucene trunk, but isn't this just a matter of getting
>> "Lucene trunk nightly -> Solr trunk lib in svn" process going?
>>
>> * Ian is 100% right.  This stuff clearly requires more discussion and a proper
>> VOTE should wait a week or so.
>>
>> Otis
>
>

On Mar 9, 2010, at 12:38 PM, Otis Gospodnetic wrote:

>
>
> * I think Grant may be right.  We don't need this discussion.  Because the Solr/Lucene developer overlap is excellent, why not just start moving selected Solr code to new Lucene modules, just like Mike proposed we move Analysis from Lucene core to a new Lucene module?

Note, if you read what I said again you will realize I wasn't actually proposing this.  I was saying actually, that I think it would not be something that people really wanted, even though it is perfectly "legal", just like poaching is perfectly "legal", but isn't, in my mind a good solution.  Sigh.  The problem with email, I guess, especially on long threads.

Sigh...
http://www.surveymonkey.com/

On Tue, Mar 9, 2010 at 2:00 PM, Grant Ingersoll <[EMAIL PROTECTED]> wrote:

>
> On Mar 9, 2010, at 12:38 PM, Otis Gospodnetic wrote:
>
> >
> >
> > * I think Grant may be right.  We don't need this discussion.  Because
> the Solr/Lucene developer overlap is excellent, why not just start moving
> selected Solr code to new Lucene modules, just like Mike proposed we move
> Analysis from Lucene core to a new Lucene module?
>
> Note, if you read what I said again you will realize I wasn't actually
> proposing this.  I was saying actually, that I think it would not be
> something that people really wanted, even though it is perfectly "legal",
> just like poaching is perfectly "legal", but isn't, in my mind a good
> solution.  Sigh.  The problem with email, I guess, especially on long
> threads.

: from Solr into Lucene don't need to be in there. All of a sudden I'm agreeing
: with Hoss about goals rather than actual steps ;) Because those points are not
: important to this vote at all - they are more examples of what we will be able
...
: This is about merging dev so we can put code where it belongs and do things
: that can make sense - its not a vote where specific code refactorings matter

This is the point where i say...

AHHHHHHHHHHHHH!!!!!!! HA HA HA HA HAAAAAAA!!!!!

HA!

I've had relationships (with live people that i cared deeply about) that
recieved less of my attention this all of this, and i just don't give a
fuck anymore.

I'm -1 to "the proposal" on the grounds that 150+ messages into the
discusison, it still seems like no one has any fucking clue what it is
we're voting on.

If anyone has any patches, releases, or a potential committers they'd like
me to vote on please let me know -- otherwise i'm wiping my hands of the
whole thing.  I'm done with this thread, and i don't plan on reading any
more messages posted to it.  (one of the bueatiful things about Lucene:
the community (and the PMC) are large enough that the wheels will keep
turning and smart people will make smart choices even if close my eyes and
stick my fingers in my ears)

if you'll excuse me: i desperately need to go review some patches, and
answer some questions on the user lists so i can ultimately finish this
day with some sense of accomplishment.
-Hoss

On Mon, Mar 8, 2010 at 8:24 PM, Grant Ingersoll <[EMAIL PROTECTED]> wrote:

> It should also be noted that a good chunk of the Solr committers are already Lucene committers, and of the remaining there are: Bill Au, Mike Klaas, Ryan McKinley, Shalin and Noble.  Mike has been inactive for quite some time (and has elected to go emeritus even though it's not marked on the page) and and Ryan, Shalin and Noble already contribute to Lucene in various parts (AFAICT), so to me it's not a big stretch to say bring them into the fold.  I haven't tracked Bill's involvement, but I also know Bill and trust he knows what it means to be a committer, i.e. he knows as much what not to touch as what to touch.  Of course, we can do a separate vote on that if that helps satisfy Chris' issue on the committers.

I don't expect that it makes much of a difference either way, but feel
free to leave me out of the Lucene auto-committership should that be
an issue.  I don't expect to become an active committer in the near
future.

> In the end, for me anyway, the current separation hurts Lucene a good deal as much as it hurts Solr, if not more.

Agreed.

The central issue is that Solr committers often work on features which
are core to "full-text search" rather than "an HTTP full-text search
server".  The parts of the features related to full-text search would
improve Lucene (the fact that Solr is used by people as a library in
an embedded context provides glaring testament to that).  But they
don't end up there, due to the friction of simultaneously developing a
feature in two projects that aren't synchronized.

Does this happen often?  It does.  I'd say over 50% of my non-trivial
changes to Solr could have been useful in Lucene.  (This is probably
not representative of the entire Solr committerdom, of course.)  In
fact, the *very first patch* I developed for Solr was adding in hit
highlighting, and I ended up copy&pasting a class from contrib
Highlighter to extend it.  Of course, I was a committer for neither
project at the time; I don't want to think about the logistics of
trying to submit patches to both projects which are so inter-dependent
(and would pretty much rely on review by someone who was a committer
on both projects anyway).

I think someone could make an argument that I should have been more
conscientious about submitting patches to Lucene, and they are
probably right.  But, I ultimately wasn't, and many other committers
weren't as well (even those who were committers for both projects),
and so we've ended up in this situation where we really have *three*
projects:  Lucene, the java search library, Lucene+, the set of
improvements and extensions to Lucene which could be in Lucene itself
and developed by the same people as Lucene, and Solr, the HTTP server
around Lucene.

The Lucene Project as a whole would benefit if this situation were
improved.  Auto-syncing Lucene-trunk with Solr would bring an
improvement, but it isn't a fundamental solution and has its own set
of problems.  The current proposal seems reasonable, but I worry about
the level of contention it is receiving.

+0

-Mike

On Tue, Mar 9, 2010 at 11:54 PM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:
>
> : from Solr into Lucene don't need to be in there. All of a sudden I'm agreeing
> : with Hoss about goals rather than actual steps ;) Because those points are not
> : important to this vote at all - they are more examples of what we will be able
>        ...
> : This is about merging dev so we can put code where it belongs and do things
> : that can make sense - its not a vote where specific code refactorings matter
>
> This is the point where i say...
>
> AHHHHHHHHHHHHH!!!!!!! HA HA HA HA HAAAAAAA!!!!!
>
>        HA!
>
> I've had relationships (with live people that i cared deeply about) that
> recieved less of my attention this all of this, and i just don't give a
> fuck anymore.
>
> I'm -1 to "the proposal" on the grounds that 150+ messages into the
> discusison, it still seems like no one has any fucking clue what it is
> we're voting on.

Thanks Chris for pointing this out! I'm somewhat lost and I can not
keep up with the amount of email regarding this "vote".
IMO if we vote on something a reply should be +1, -1 or +-0. If you
want to discuss, vote -1.

simon
>
> If anyone has any patches, releases, or a potential committers they'd like
> me to vote on please let me know -- otherwise i'm wiping my hands of the
> whole thing.  I'm done with this thread, and i don't plan on reading any
> more messages posted to it.  (one of the bueatiful things about Lucene:
> the community (and the PMC) are large enough that the wheels will keep
> turning and smart people will make smart choices even if close my eyes and
> stick my fingers in my ears)
>
> if you'll excuse me: i desperately need to go review some patches, and
> answer some questions on the user lists so i can ultimately finish this
> day with some sense of accomplishment.
>
>
> -Hoss
>
>

+1

Andi..

On Mar 9, 2010, at 3:11, Yonik Seeley <[EMAIL PROTECTED]> wrote:

> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr  
> development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>> Subject: Merge the development of Solr/Lucene (take 2)
>> A new vote, that slightly changes proposal from last vote (adding  
>> only
>> that Lucene can cut a release even if Solr doesn't):
>>
>> * Merging the dev lists into a single list.
>>
>> * Merging committers.
>>
>> * When any change is committed (to a module that "belongs to" Solr or
>>   to Lucene), all tests must pass.
>>
>> * Release details will be decided by dev community, but, Lucene may
>>   release without Solr.
>>
>> * Modulariize the sources: pull things out of Lucene's core (break
>>   out query parser, move all core queries & analyzers under their
>>   contrib counterparts), pull things out of Solr's core (analyzers,
>>   queries).
>>
>> These things would not change:
>>
>> * Besides modularizing (above), the source code would remain factored
>>   into separate dirs/modules the way it is now.
>>
>> * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>   issues).
>>
>> * User's lists remain separate.
>>
>> * Web sites remain separate.
>>
>> * Release artifacts/jars remain separate.

+1
On Mon, Mar 8, 2010 at 9:11 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>> Subject: Merge the development of Solr/Lucene (take 2)
>> A new vote, that slightly changes proposal from last vote (adding only
>> that Lucene can cut a release even if Solr doesn't):
>>
>>  * Merging the dev lists into a single list.
>>
>>  * Merging committers.
>>
>>  * When any change is committed (to a module that "belongs to" Solr or
>>    to Lucene), all tests must pass.
>>
>>  * Release details will be decided by dev community, but, Lucene may
>>    release without Solr.
>>
>>  * Modulariize the sources: pull things out of Lucene's core (break
>>    out query parser, move all core queries & analyzers under their
>>    contrib counterparts), pull things out of Solr's core (analyzers,
>>    queries).
>>
>> These things would not change:
>>
>>  * Besides modularizing (above), the source code would remain factored
>>    into separate dirs/modules the way it is now.
>>
>>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>    issues).
>>
>>  * User's lists remain separate.
>>
>>  * Web sites remain separate.
>>
>>  * Release artifacts/jars remain separate.
>

Thanks everyone, this vote has passed.
A bit more contentious of a PMC vote than usual, but the committer
vote was clear.

-Yonik
On Mon, Mar 8, 2010 at 9:11 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> Apoligies in advance for calling yet another vote, but I just wanted
> to make sure this was official.
> Mike's second VOTE thread could probably technically stand on it's own
> (since it included PMC votes), but given that I said in my previous
> VOTE thread that I was just polling Lucene/Solr committers and would
> call a second PMC vote, that may have acted to suppress PMC votes on
> Mike's thread also.
>
> Please vote for the proposal quoted below to merge lucene/solr development.
> Here's my +1
>
> -Yonik
>
> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merge_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>> Subject: Merge the development of Solr/Lucene (take 2)
>> A new vote, that slightly changes proposal from last vote (adding only
>> that Lucene can cut a release even if Solr doesn't):
>>
>>  * Merging the dev lists into a single list.
>>
>>  * Merging committers.
>>
>>  * When any change is committed (to a module that "belongs to" Solr or
>>    to Lucene), all tests must pass.
>>
>>  * Release details will be decided by dev community, but, Lucene may
>>    release without Solr.
>>
>>  * Modulariize the sources: pull things out of Lucene's core (break
>>    out query parser, move all core queries & analyzers under their
>>    contrib counterparts), pull things out of Solr's core (analyzers,
>>    queries).
>>
>> These things would not change:
>>
>>  * Besides modularizing (above), the source code would remain factored
>>    into separate dirs/modules the way it is now.
>>
>>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>    issues).
>>
>>  * User's lists remain separate.
>>
>>  * Web sites remain separate.
>>
>>  * Release artifacts/jars remain separate.
>

Hi Yonik,

IMO, this vote has not passed. A bullet of this proposal proposes code
modifications and this is subject to VETO per Apache guidelines:

http://www.apache.org/foundation/voting.html#Veto

Since that point is up for debate, I think we can get clarification on this
from the board at their next meeting, but I dispute calling the VOTE
"passed" until that time.

In the meanwhile there has been much community discussion and points made in
favor of each point of view over the past week. My recommendation is to sit
on this for at least a week, then revisit the issue with clear and concise
goals, and incremental pieces to vote on.

Cheers,
Chris

On 3/11/10 6:29 PM, "Yonik Seeley" <[EMAIL PROTECTED]> wrote:

> Thanks everyone, this vote has passed.
> A bit more contentious of a PMC vote than usual, but the committer
> vote was clear.
>
> -Yonik
>
>
> On Mon, Mar 8, 2010 at 9:11 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>> Apoligies in advance for calling yet another vote, but I just wanted
>> to make sure this was official.
>> Mike's second VOTE thread could probably technically stand on it's own
>> (since it included PMC votes), but given that I said in my previous
>> VOTE thread that I was just polling Lucene/Solr committers and would
>> call a second PMC vote, that may have acted to suppress PMC votes on
>> Mike's thread also.
>>
>> Please vote for the proposal quoted below to merge lucene/solr development.
>> Here's my +1
>>
>> -Yonik
>>
>> Mike's call for a VOTE (amongst lucene/solr committers +11 to -1):
>> http://search.lucidimagination.com/search/document/a400ffe62ae21aca/vote_merg
>> e_the_development_of_solr_lucene_take_2#22d7cd086d9c5cf0
>>> Subject: Merge the development of Solr/Lucene (take 2)
>>> A new vote, that slightly changes proposal from last vote (adding only
>>> that Lucene can cut a release even if Solr doesn't):
>>>
>>>  * Merging the dev lists into a single list.
>>>
>>>  * Merging committers.
>>>
>>>  * When any change is committed (to a module that "belongs to" Solr or
>>>    to Lucene), all tests must pass.
>>>
>>>  * Release details will be decided by dev community, but, Lucene may
>>>    release without Solr.
>>>
>>>  * Modulariize the sources: pull things out of Lucene's core (break
>>>    out query parser, move all core queries & analyzers under their
>>>    contrib counterparts), pull things out of Solr's core (analyzers,
>>>    queries).
>>>
>>> These things would not change:
>>>
>>>  * Besides modularizing (above), the source code would remain factored
>>>    into separate dirs/modules the way it is now.
>>>
>>>  * Issue tracking remains separate (SOLR-XXX and LUCENE-XXX
>>>    issues).
>>>
>>>  * User's lists remain separate.
>>>
>>>  * Web sites remain separate.
>>>
>>>  * Release artifacts/jars remain separate.
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: [EMAIL PROTECTED]
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++