hey folks,

now that we have everything under codec control and norms are cut over
to docvalues I wonder what keeps us from moving towards a 4.0 beta
release. IMO its more than time to get it out there and from a
technical perspective we are close enough to take that step. We need
to figure out how we are doing this release, ie. are we releasing a
beta (we have never done this since I have been around though)
I recall that robert had some good ideas about this but can't find
them right now. I think we should get this going so I wonder if people
have any blockers or strong opinions?

simon

---------------------------------------------------------------------

Can I first split composite from atomic readers?

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]

> -----Original Message-----
> From: Simon Willnauer [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, January 08, 2012 3:31 PM
> To: [EMAIL PROTECTED]
> Subject: Lucene 4.0 Beta
>
> hey folks,
>
> now that we have everything under codec control and norms are cut over to
> docvalues I wonder what keeps us from moving towards a 4.0 beta release.
> IMO its more than time to get it out there and from a technical perspective we
> are close enough to take that step. We need to figure out how we are doing this
> release, ie. are we releasing a beta (we have never done this since I have been
> around though) I recall that robert had some good ideas about this but can't
> find them right now. I think we should get this going so I wonder if people have
> any blockers or strong opinions?
>
> simon
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
> commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

On Jan 8, 2012, at 9:30 AM, Simon Willnauer wrote:

> hey folks,
>
> now that we have everything under codec control and norms are cut over
> to docvalues I wonder what keeps us from moving towards a 4.0 beta
> release. IMO its more than time to get it out there and from a
> technical perspective we are close enough to take that step. We need
> to figure out how we are doing this release, ie. are we releasing a
> beta (we have never done this since I have been around though)

I think a beta makes a lot of sense.  4.0 is a big set of changes.

As an aside, I would love to get our new website up to coincide with 4.0.  It really is close and just needs some graphics love, as all the underpinnings, etc. are in place.  Unfortunately, I have the graphic design skills of a 1st grader.  I can code it up once someone provides the images, etc. and I know what I want on which pages, just need a volunteer with some modicum of talent to lend a hand.  Volunteers?

> I recall that robert had some good ideas about this but can't find
> them right now. I think we should get this going so I wonder if people
> have any blockers or strong opinions?

From the Solr side, I think Solr cloud has a lot of nice updates I'd like to see in 4.0.  Mark or Yonik can jump in on when they think they will have something for merging.

-Grant
---------------------------------------------------------------------

On Sun, Jan 8, 2012 at 9:30 AM, Simon Willnauer
<[EMAIL PROTECTED]> wrote:
> hey folks,
>
> now that we have everything under codec control and norms are cut over
> to docvalues I wonder what keeps us from moving towards a 4.0 beta
> release.

here are my thoughts:

But just taking that issue (not to try to pick on it): this is a
little incomplete for a few reasons:
1. Shouldn't we remove the limitation that norms are single byte? I
started looking at this and it reveals some hair I think and creates
some interesting questions.
2. Shouldn't docvalues have a a simpletext implementation? I know the
api is a little simpler now, but previously there was no way I could
have written a simpletext implementation if my life depended on it.
3. There were some API todos listed on the issue: I think those TODOs
are important (like all the extra add methods and lucene-40 private
bulk shit in the abstract api). These are new APIs so I think we
should try to tackle these.

Otherwise, as before we still have to figure out how modules are being
packaged, and we still have to review other apis and in general do a
lot of testing and refactoring and bugfixing.

>From the tests/bugfixing perspective, on trunk its not hard to dig
into stuff here and there and find lots of really horrible bugs that
exist in trunk-only. This is a sign that our tests are not good enough
and that the code is not stable enough.

For apis perspective: all the apis really need some serious reviewing:
if anything is abstract it needs multiple implementations (even ones
in src/test). Lots of the javadoc is outdated, even in trunk itself
and needs to be cleaned up. The MIGRATE.txt and CHANGES.txt needs to
be reviewed and up to date. We have to deal with things like
fileformats.html. I'm just going to quit typing now because this email
will just keep getting longer and longer.

--
lucidimagination.com

---------------------------------------------------------------------

On Sun, Jan 8, 2012 at 11:18 AM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Sun, Jan 8, 2012 at 9:30 AM, Simon Willnauer
> <[EMAIL PROTECTED]> wrote:
>> hey folks,
>>
>> now that we have everything under codec control

By the way: this is a little misleading: Deletes and compound-file
format are still not done.

--
lucidimagination.com

---------------------------------------------------------------------

On Sun, Jan 8, 2012 at 5:18 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Sun, Jan 8, 2012 at 9:30 AM, Simon Willnauer
> <[EMAIL PROTECTED]> wrote:
>> hey folks,
>>
>> now that we have everything under codec control and norms are cut over
>> to docvalues I wonder what keeps us from moving towards a 4.0 beta
>> release.
>
> here are my thoughts:
>
> But just taking that issue (not to try to pick on it): this is a
> little incomplete for a few reasons:
> 1. Shouldn't we remove the limitation that norms are single byte? I
> started looking at this and it reveals some hair I think and creates
> some interesting questions.

+1
> 2. Shouldn't docvalues have a a simpletext implementation? I know the
> api is a little simpler now, but previously there was no way I could
> have written a simpletext implementation if my life depended on it.
+1
> 3. There were some API todos listed on the issue: I think those TODOs
> are important (like all the extra add methods and lucene-40 private
> bulk shit in the abstract api). These are new APIs so I think we
> should try to tackle these.

FYI - those extra add methods are unrelated to the bulk stuff - I
pulled them up to implement the default merging.
>
> Otherwise, as before we still have to figure out how modules are being
> packaged, and we still have to review other apis and in general do a
> lot of testing and refactoring and bugfixing.

Agreed.
>
> From the tests/bugfixing perspective, on trunk its not hard to dig
> into stuff here and there and find lots of really horrible bugs that
> exist in trunk-only. This is a sign that our tests are not good enough
> and that the code is not stable enough.

I don't understand this entirely, can you elaborate? if there are
horrible bugs there should be issues marked as blocker.

>
> For apis perspective: all the apis really need some serious reviewing:
> if anything is abstract it needs multiple implementations (even ones
> in src/test). Lots of the javadoc is outdated, even in trunk itself
> and needs to be cleaned up. The MIGRATE.txt and CHANGES.txt needs to
> be reviewed and up to date. We have to deal with things like
> fileformats.html. I'm just going to quit typing now because this email
> will just keep getting longer and longer.

I did ask to push the release tomorrow. I personally feel some
frustration that lucene 4 is still not released and I think we are
really close. From my perspective we should build some kind of roadmap
to get lucene 4 or a beta out lets say within the next 2 month and
concentrate on getting the remaining issues resolved. We should
collect issues and problems and mark them as blockers so we can pick
them up and get it out of the door. Right now all the issues you
listed are somewhat known but there is no way to access them, I
suggest we open JIRA ticket and decide issue per issue if they are
blockers or not. A SimpleText impl for docvalues are certainly nice to
have but not a blocker from my perspective. I already worked on it and
it will make it into 4.0 but we need to mark those issues that are
real blockers.

thoughts?

simon
>
> --
> lucidimagination.com

---------------------------------------------------------------------

On Sun, Jan 8, 2012 at 1:36 PM, Simon Willnauer
<[EMAIL PROTECTED]> wrote:
>>
>> From the tests/bugfixing perspective, on trunk its not hard to dig
>> into stuff here and there and find lots of really horrible bugs that
>> exist in trunk-only. This is a sign that our tests are not good enough
>> and that the code is not stable enough.
>
> I don't understand this entirely, can you elaborate? if there are
> horrible bugs there should be issues marked as blocker.
>

Well maybe I delivered this idea the wrong way: i didn't mean i
secretly know of any bugs or anything like that:
I just mean as a trend, recently over the last few months I have
noticed lots of bugs that only affect 4.x but don't affect 3.x

I think this is natural, a lot has changed, but to mitigate that we
need to beef up tests and spend more time reviewing and trying to
clean up.
I'm just as guilty as anyone else when it comes to these changes
without having adequate tests.

>
> I did ask to push the release tomorrow. I personally feel some
> frustration that lucene 4 is still not released and I think we are
> really close.

I'm pretty frustrated working on some code for years that is unreleased.

> From my perspective we should build some kind of roadmap
> to get lucene 4 or a beta out lets say within the next 2 month and
> concentrate on getting the remaining issues resolved. We should
> collect issues and problems and mark them as blockers so we can pick
> them up and get it out of the door. Right now all the issues you
> listed are somewhat known but there is no way to access them, I
> suggest we open JIRA ticket and decide issue per issue if they are
> blockers or not.

I do think we need some kind of 'vague plan' as much as we can have
one in open source. Here is what I propose:

1. we decide that 3.6 (no reason to call it 3.9?) should be the last
minor release on 3.6, lets try to make a nice 3.6,... we can then
issue bugfixes like 3.6.1, 3.6.2, ...
2. copy trunk to branch_4x and call it our 'stable branch' (even
though it isn't, not yet).
3. trunk is now 5.x and open for scary changes like trunk is today.
4. eventually, when its right, we issue alpha or beta or real release
off branch_4x for 4.0

I think this has some good practical implications:
1. people can still add new scary cool fun stuff to trunk as always.
2. new scary stuff isnt being added to branch_4x constantly
destabilizing it (but certainly features that are safe, just like
branch_3x today!)
3. since initially branch_4x and trunk are close, we essentially
double our jenkins coverage (currently we spend half of it on 3.x, but
this isn't helping stabilize 4.x code)

> A SimpleText impl for docvalues are certainly nice to
> have but not a blocker from my perspective. I already worked on it and
> it will make it into 4.0 but we need to mark those issues that are
> real blockers.
>

I actually think we should have a SimpleText impl for everything, here's why:
1. we need multiple implementations to flush out any assumptions in
the abstract APIs.
2. we need to ensure multiple impls actually work for backwards
compatibility to actually work in the future.
3. we need to fix tests to not have things like hard-coded assumptions
about filenames so the tests are actually useful.

Sorry for the example by the way, i didnt mean to pick on docvalues.
Probably even worse is the fact I left a lot of codec impls really
bundled together inside 4.x impls without splitting out the 3.x stuff.
Until we do this, how do we know back compat will really work with
codecs?

--
lucidimagination.com

---------------------------------------------------------------------

I haven't tried the trunk/4.0 stuff, so maybe I'm out of line, but how about a public alpha first. It would have appropriate disclaimers regarding quality and that the API, while reasonably stable, is subject to change w/ little notice.

I tink beta implies that there is a high degree of confidence in the sufficiency, quality and stability (API) of the code. Alpha usually notes the insufficiency (what remains to be done), the potential for bugs (test cases are not as robust as they should be) and that the API is unstable (subject to change).

If I knew that it was close to release (i.e. alpha or beta), I'd give it a try.

-- DM

P.S. I think a road map for a 4.0 release is a wonderful idea.

On Jan 8, 2012, at 2:05 PM, Robert Muir wrote:

> On Sun, Jan 8, 2012 at 1:36 PM, Simon Willnauer
> <[EMAIL PROTECTED]> wrote:
>>>
>>> From the tests/bugfixing perspective, on trunk its not hard to dig
>>> into stuff here and there and find lots of really horrible bugs that
>>> exist in trunk-only. This is a sign that our tests are not good enough
>>> and that the code is not stable enough.
>>
>> I don't understand this entirely, can you elaborate? if there are
>> horrible bugs there should be issues marked as blocker.
>>
>
> Well maybe I delivered this idea the wrong way: i didn't mean i
> secretly know of any bugs or anything like that:
> I just mean as a trend, recently over the last few months I have
> noticed lots of bugs that only affect 4.x but don't affect 3.x
>
> I think this is natural, a lot has changed, but to mitigate that we
> need to beef up tests and spend more time reviewing and trying to
> clean up.
> I'm just as guilty as anyone else when it comes to these changes
> without having adequate tests.
>
>>
>> I did ask to push the release tomorrow. I personally feel some
>> frustration that lucene 4 is still not released and I think we are
>> really close.
>
> I'm pretty frustrated working on some code for years that is unreleased.
>
>> From my perspective we should build some kind of roadmap
>> to get lucene 4 or a beta out lets say within the next 2 month and
>> concentrate on getting the remaining issues resolved. We should
>> collect issues and problems and mark them as blockers so we can pick
>> them up and get it out of the door. Right now all the issues you
>> listed are somewhat known but there is no way to access them, I
>> suggest we open JIRA ticket and decide issue per issue if they are
>> blockers or not.
>
> I do think we need some kind of 'vague plan' as much as we can have
> one in open source. Here is what I propose:
>
> 1. we decide that 3.6 (no reason to call it 3.9?) should be the last
> minor release on 3.6, lets try to make a nice 3.6,... we can then
> issue bugfixes like 3.6.1, 3.6.2, ...
> 2. copy trunk to branch_4x and call it our 'stable branch' (even
> though it isn't, not yet).
> 3. trunk is now 5.x and open for scary changes like trunk is today.
> 4. eventually, when its right, we issue alpha or beta or real release
> off branch_4x for 4.0
>
> I think this has some good practical implications:
> 1. people can still add new scary cool fun stuff to trunk as always.
> 2. new scary stuff isnt being added to branch_4x constantly
> destabilizing it (but certainly features that are safe, just like
> branch_3x today!)
> 3. since initially branch_4x and trunk are close, we essentially
> double our jenkins coverage (currently we spend half of it on 3.x, but
> this isn't helping stabilize 4.x code)
>
>> A SimpleText impl for docvalues are certainly nice to
>> have but not a blocker from my perspective. I already worked on it and
>> it will make it into 4.0 but we need to mark those issues that are
>> real blockers.
>>
>
> I actually think we should have a SimpleText impl for everything, here's why:
> 1. we need multiple implementations to flush out any assumptions in
> the abstract APIs.
> 2. we need to ensure multiple impls actually work for backwards
> compatibility to actually work in the future.

I'll defer to the collective wisdon of folks like rmuir & simon & uwe &
mccandless about when the major new APIs & features like codecs &
docvalues are stable and vetted enough to consider releasing, and i'll
defer to miller & yonik about when the solr cloud update stuff is ready;
but i think we'd be letting down a lot of users if we tried to release
something called "4.0" that didn't have those things in it.

i'm also +1 to everything quoted below...

: I haven't tried the trunk/4.0 stuff, so maybe I'm out of line, but how
: about a public alpha first. It would have appropriate disclaimers

...an alpha off of trunk even before doing beta(s) off of a 4x_branch
would be a good wake up call to folks that Lucene 4.0 is not a myth and
that people need to start paying attention and speak up if they see any
major api/doc problems or weird bugs they haven't brought up because they
just assumed it would get fixed eventually (be nice to fix as much as we
can on trunk before branching so we don't have to do a bunch of anoying
merges)

: > 2. copy trunk to branch_4x and call it our 'stable branch' (even
: > though it isn't, not yet).
: > 3. trunk is now 5.x and open for scary changes like trunk is today.
: > 4. eventually, when its right, we issue alpha or beta or real release
: > off branch_4x for 4.0
...
: > 1. we need multiple implementations to flush out any assumptions in
: > the abstract APIs.
: > 2. we need to ensure multiple impls actually work for backwards
: > compatibility to actually work in the future.

... +1 to all that
As for this comment...

: > 1. we decide that 3.6 (no reason to call it 3.9?) should be the last
: > minor release on 3.6, lets try to make a nice 3.6,... we can then
: > issue bugfixes like 3.6.1, 3.6.2, ...

..i'm not sure that we need to paint ourselves into that particular
corner.  

once we make a 4x branch, the only hard line we *have* to stand behind is
not to make file format changes to 3x -- that doesn't mean there can't be
minor feature releases like 3.7, etc... (it's unlikely that there should
ever be a need for them, but there's no reason to forbid it -- just like
there's nothing stoping us from putting out a 2.10 release right now if
some previously silent but large user base of the 2.x APIs comes a long
clammoring for a backport of some small feature)
-Hoss

---------------------------------------------------------------------

On Jan 9, 2012, at 1:57 PM, Chris Hostetter wrote:

+1

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------

On Mon, Jan 9, 2012 at 1:57 PM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:

> once we make a 4x branch, the only hard line we *have* to stand behind is
> not to make file format changes to 3x -- that doesn't mean there can't be
> minor feature releases like 3.7, etc... (it's unlikely that there should
> ever be a need for them, but there's no reason to forbid it -- just like
> there's nothing stoping us from putting out a 2.10 release right now if
> some previously silent but large user base of the 2.x APIs comes a long
> clammoring for a backport of some small feature)
>

Well its not just file format changes, for example its also APIs. If
3.6 had an API that worked in a backwards compatible way with 4.0,
but then we broke it in 3.7 and 4.1 or something, that would be way
too confusing.  Then a user upgrades to 3.6 and finds its hard
to go from 3.6 to 4.0 (which they think is an upgrade, but for that
API its in fact a downgrade).

Setting 3.x to "bugfix-only" prevents these kinds things too... I
still think its the only reasonable approach.

--
lucidimagination.com

---------------------------------------------------------------------

On Sun, Jan 8, 2012 at 8:05 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Sun, Jan 8, 2012 at 1:36 PM, Simon Willnauer
> <[EMAIL PROTECTED]> wrote:
>>>
>>> From the tests/bugfixing perspective, on trunk its not hard to dig
>>> into stuff here and there and find lots of really horrible bugs that
>>> exist in trunk-only. This is a sign that our tests are not good enough
>>> and that the code is not stable enough.
>>
>> I don't understand this entirely, can you elaborate? if there are
>> horrible bugs there should be issues marked as blocker.
>>
>
> Well maybe I delivered this idea the wrong way: i didn't mean i
> secretly know of any bugs or anything like that:
> I just mean as a trend, recently over the last few months I have
> noticed lots of bugs that only affect 4.x but don't affect 3.x

ah ok phew :)
>
> I think this is natural, a lot has changed, but to mitigate that we
> need to beef up tests and spend more time reviewing and trying to
> clean up.
> I'm just as guilty as anyone else when it comes to these changes
> without having adequate tests.
>
>>
>> I did ask to push the release tomorrow. I personally feel some
>> frustration that lucene 4 is still not released and I think we are
>> really close.
>
> I'm pretty frustrated working on some code for years that is unreleased.
>
>> From my perspective we should build some kind of roadmap
>> to get lucene 4 or a beta out lets say within the next 2 month and
>> concentrate on getting the remaining issues resolved. We should
>> collect issues and problems and mark them as blockers so we can pick
>> them up and get it out of the door. Right now all the issues you
>> listed are somewhat known but there is no way to access them, I
>> suggest we open JIRA ticket and decide issue per issue if they are
>> blockers or not.
>
> I do think we need some kind of 'vague plan' as much as we can have
> one in open source. Here is what I propose:
>
> 1. we decide that 3.6 (no reason to call it 3.9?) should be the last
> minor release on 3.6, lets try to make a nice 3.6,... we can then
> issue bugfixes like 3.6.1, 3.6.2, ...

+1 to this. lets get 4.0 rolling... I don't think we need to jump to 3.9

> 2. copy trunk to branch_4x and call it our 'stable branch' (even
> though it isn't, not yet).

+1 I think we should actually do that soonish, is there anything which
keeps us from branching IMO today is as good as tomorrow. What is the
status of solr-cloud? I mean is this going to be ready within a
reasonable amount of time? That feature by itself could justify a 4.1
release immediately I'd say.

> 3. trunk is now 5.x and open for scary changes like trunk is today.

+!
> 4. eventually, when its right, we issue alpha or beta or real release
> off branch_4x for 4.0

agreed, I think we should never again release even a alpha off trunk.
On trunk folks should be allowed to go crazy and do scary changes...
on the stable / stabelizing branches we should be more conservative.
>
> I think this has some good practical implications:
> 1. people can still add new scary cool fun stuff to trunk as always.
> 2. new scary stuff isnt being added to branch_4x constantly
> destabilizing it (but certainly features that are safe, just like
> branch_3x today!)
> 3. since initially branch_4x and trunk are close, we essentially
> double our jenkins coverage (currently we spend half of it on 3.x, but
> this isn't helping stabilize 4.x code)
>
>> A SimpleText impl for docvalues are certainly nice to
>> have but not a blocker from my perspective. I already worked on it and
>> it will make it into 4.0 but we need to mark those issues that are
>> real blockers.
>>
>
> I actually think we should have a SimpleText impl for everything, here's why:
> 1. we need multiple implementations to flush out any assumptions in
> the abstract APIs.
> 2. we need to ensure multiple impls actually work for backwards
> compatibility to actually work in the future.
> 3. we need to fix tests to not have things like hard-coded assumptions

+1 to getting 4.0 alpha out asap for user feedback.  We have
incredible (but largely unused) new features in 4.0.

I like the plan to cut branch for 4.0, and trunk becomes future 5.0,
and 3.6.x becomes bugfix only, as an atomic event :)

Then we should try to focus on prepping/iterating on the 4.0 branch
for real release: going through Jira and marking (and fixing!) blocker
issues, testing, scrutinizing the new APIs, etc.

Mike McCandless

http://blog.mikemccandless.com

On Wed, Jan 11, 2012 at 2:31 PM, Simon Willnauer
<[EMAIL PROTECTED]> wrote:
> On Sun, Jan 8, 2012 at 8:05 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>> On Sun, Jan 8, 2012 at 1:36 PM, Simon Willnauer
>> <[EMAIL PROTECTED]> wrote:
>>>>
>>>> From the tests/bugfixing perspective, on trunk its not hard to dig
>>>> into stuff here and there and find lots of really horrible bugs that
>>>> exist in trunk-only. This is a sign that our tests are not good enough
>>>> and that the code is not stable enough.
>>>
>>> I don't understand this entirely, can you elaborate? if there are
>>> horrible bugs there should be issues marked as blocker.
>>>
>>
>> Well maybe I delivered this idea the wrong way: i didn't mean i
>> secretly know of any bugs or anything like that:
>> I just mean as a trend, recently over the last few months I have
>> noticed lots of bugs that only affect 4.x but don't affect 3.x
>
> ah ok phew :)
>>
>> I think this is natural, a lot has changed, but to mitigate that we
>> need to beef up tests and spend more time reviewing and trying to
>> clean up.
>> I'm just as guilty as anyone else when it comes to these changes
>> without having adequate tests.
>>
>>>
>>> I did ask to push the release tomorrow. I personally feel some
>>> frustration that lucene 4 is still not released and I think we are
>>> really close.
>>
>> I'm pretty frustrated working on some code for years that is unreleased.
>>
>>> From my perspective we should build some kind of roadmap
>>> to get lucene 4 or a beta out lets say within the next 2 month and
>>> concentrate on getting the remaining issues resolved. We should
>>> collect issues and problems and mark them as blockers so we can pick
>>> them up and get it out of the door. Right now all the issues you
>>> listed are somewhat known but there is no way to access them, I
>>> suggest we open JIRA ticket and decide issue per issue if they are
>>> blockers or not.
>>
>> I do think we need some kind of 'vague plan' as much as we can have
>> one in open source. Here is what I propose:
>>
>> 1. we decide that 3.6 (no reason to call it 3.9?) should be the last
>> minor release on 3.6, lets try to make a nice 3.6,... we can then
>> issue bugfixes like 3.6.1, 3.6.2, ...
>
> +1 to this. lets get 4.0 rolling... I don't think we need to jump to 3.9
>
>> 2. copy trunk to branch_4x and call it our 'stable branch' (even
>> though it isn't, not yet).
>
> +1 I think we should actually do that soonish, is there anything which
> keeps us from branching IMO today is as good as tomorrow. What is the
> status of solr-cloud? I mean is this going to be ready within a
> reasonable amount of time? That feature by itself could justify a 4.1
> release immediately I'd say.
>
>> 3. trunk is now 5.x and open for scary changes like trunk is today.
>
> +!
>> 4. eventually, when its right, we issue alpha or beta or real release
>> off branch_4x for 4.0
>
> agreed, I think we should never again release even a alpha off trunk.
> On trunk folks should be allowed to go crazy and do scary changes...
> on the stable / stabelizing branches we should be more conservative.
>
>
>>
>> I think this has some good practical implications:
>> 1. people can still add new scary cool fun stuff to trunk as always.
>> 2. new scary stuff isnt being added to branch_4x constantly
>> destabilizing it (but certainly features that are safe, just like
>> branch_3x today!)
>> 3. since initially branch_4x and trunk are close, we essentially
>> double our jenkins coverage (currently we spend half of it on 3.x, but

On Wed, Jan 11, 2012 at 2:31 PM, Simon Willnauer
<[EMAIL PROTECTED]> wrote:
> On Sun, Jan 8, 2012 at 8:05 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>> 2. copy trunk to branch_4x and call it our 'stable branch' (even
>> though it isn't, not yet).
>
> +1 I think we should actually do that soonish, is there anything which
> keeps us from branching IMO today is as good as tomorrow.

>From the other comments in this thread, it feels like an actual 4.0
would be months off.
Branching so soon would just cause that much more merging work, which
would quickly get difficult if people made meaningful changes to 5x.
Are there really any large 5x changes that people want to start
working on *now* rather than working on 4.0?

> What is the
> status of solr-cloud?

Coming up quickly - I think we're down to one no-commit, and we should
be ready to merge back to trunk hopefully next week.

-Yonik
http://www.lucidimagination.com

---------------------------------------------------------------------

On Wed, Jan 11, 2012 at 8:41 PM, Yonik Seeley
<[EMAIL PROTECTED]> wrote:
> On Wed, Jan 11, 2012 at 2:31 PM, Simon Willnauer
> <[EMAIL PROTECTED]> wrote:
>> On Sun, Jan 8, 2012 at 8:05 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>>> 2. copy trunk to branch_4x and call it our 'stable branch' (even
>>> though it isn't, not yet).
>>
>> +1 I think we should actually do that soonish, is there anything which
>> keeps us from branching IMO today is as good as tomorrow.
>
> From the other comments in this thread, it feels like an actual 4.0
> would be months off.

I disagree. the issues mentioned (the feature parts) are in the
pipeline and patches available. mike started a branch for fixing some
of the Field API issues, Norms are cut over and Similarity support is
close. The remaining part is smoothing APIs, more tests and fixing
documentation. the main issue here is letting the features and the API
stabilize itself which needs time.

> Branching so soon would just cause that much more merging work, which
> would quickly get difficult if people made meaningful changes to 5x.
> Are there really any large 5x changes that people want to start
> working on *now* rather than working on 4.0?

thats not the issue here IMO. we should not merge much from 5.0
feature wise and since there are no huge features planned its a good
time I think. this gives us time until 5.x and 4.0 diverge too much.

>
>> What is the
>> status of solr-cloud?
>
> Coming up quickly - I think we're down to one no-commit, and we should
> be ready to merge back to trunk hopefully next week.

great! so we can then consider branching no?
>
> -Yonik
> http://www.lucidimagination.com

---------------------------------------------------------------------

On Wed, Jan 11, 2012 at 2:51 PM, Simon Willnauer
<[EMAIL PROTECTED]> wrote:
> On Wed, Jan 11, 2012 at 8:41 PM, Yonik Seeley
>> Branching so soon would just cause that much more merging work, which
>> would quickly get difficult if people made meaningful changes to 5x.
>> Are there really any large 5x changes that people want to start
>> working on *now* rather than working on 4.0?
>
> thats not the issue here IMO. we should not merge much from 5.0
> feature wise and since there are no huge features planned its a good
> time I think.

I was talking about merging all of the changes to 4x to get it ready
for release back into 5.0.
A branch at this point seems unnecessary and just causes a lot of
extra work up until 4.0 is released.
If someone does want to work on a large 5.0 feature *right now* as
opposed to working on 4.0, then they can create a branch and merge
that later.

-Yonik
http://www.lucidimagination.com

---------------------------------------------------------------------

On Wed, Jan 11, 2012 at 2:57 PM, Yonik Seeley
<[EMAIL PROTECTED]> wrote:

> If someone does want to work on a large 5.0 feature *right now* as
> opposed to working on 4.0, then they can create a branch and merge
> that later.
>

No, they can commit it to trunk.

If we want 4.x to be stabilized, then we need to branch it, not try to
'stop all future development'.

(Personally i will be committing to trunk, feel free to do your own
thing, if we decide to release 4.x off of trunk, don't be angry when i
commit something super-unstable the day before release).

--
lucidimagination.com

---------------------------------------------------------------------

On Jan 11, 2012, at 12:03, Robert Muir <[EMAIL PROTECTED]> wrote:

> On Wed, Jan 11, 2012 at 2:57 PM, Yonik Seeley
> <[EMAIL PROTECTED]> wrote:
>
>> If someone does want to work on a large 5.0 feature *right now* as
>> opposed to working on 4.0, then they can create a branch and merge
>> that later.
>>
>
> No, they can commit it to trunk.
>
> If we want 4.x to be stabilized, then we need to branch it, not try to
> 'stop all future development'.

Agreed. The presence of this branch_4x and the api stability it implies is what I've been waiting for to port all the python samples and tests again in PyLucene. With the api a moving target as it's been in trunk, this is much harder.

Andi..

>
> (Personally i will be committing to trunk, feel free to do your own
> thing, if we decide to release 4.x off of trunk, don't be angry when i
> commit something super-unstable the day before release).
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Wed, Jan 11, 2012 at 2:41 PM, Yonik Seeley
<[EMAIL PROTECTED]> wrote:

> From the other comments in this thread, it feels like an actual 4.0
> would be months off.

I'd think we could cut a 4.0 alpha today, after branching.

Sure it'd have caveats (certain issues are in progress, APIs may
change, etc.), but that's all par-for-the-course for an alpha release.

> Branching so soon would just cause that much more merging work, which
> would quickly get difficult if people made meaningful changes to 5x.

Merging is low-cost when the branches haven't diverged much, as will
be the case during 4.0's release process.

It does take discipline though, on the committers' parts, to
remember/choose/decide on each commit whether/where it should be back
ported.

> Are there really any large 5x changes that people want to start
> working on *now* rather than working on 4.0?

That question is by definition unanswerable in open-source.

You can never know who suddenly wants to work on something crazy...
and trunk should never be frozen to such changes.  Trunk should always
be open for the next major release, no restrictions.

By having a bug-fix only 3.6 branch, a stable 4.x branch, and an
"anything goes" trunk, we don't have to answer this question.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------

On Wed, Jan 11, 2012 at 3:07 PM, Michael McCandless
<[EMAIL PROTECTED]> wrote:
>
> I'd think we could cut a 4.0 alpha today, after branching.
>

How? the packaging for modules is totally hosed for example: how would
we even package such a thing?

Also when we push out alpha/beta/etc shouldn't we at the minimum say
'apis might change but this is the index format' ? Otherwise how is it
different than a nightly build to end-users?

These are reasons why we must branch: because at any time we should be
able to add new packages (like LUCENE-3305) that will 'destabilize'
packaging (which is important to get right here), or we might change
the index format (like LUCENE-3687) which I think we might not
necessarily want to do.

(I listed two issues that I feel are very close to being committed
right now as examples)

--
lucidimagination.com

---------------------------------------------------------------------

Having some interest in this issue, may I suggest setting a branch date?  On the agreed-upon date, a branch is made.  After that date, commits go to trunk and (maybe) are pulled up into the 4.0 branch.  If the date is oh, say, 1 week away, people can plan accordingly to yield a relatively stable branch right out of the gate.

Karl

-----Original Message-----
From: ext Robert Muir [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, January 11, 2012 3:15 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Lucene 4.0 Beta

On Wed, Jan 11, 2012 at 3:07 PM, Michael McCandless <[EMAIL PROTECTED]> wrote:
>
> I'd think we could cut a 4.0 alpha today, after branching.
>

How? the packaging for modules is totally hosed for example: how would we even package such a thing?

Also when we push out alpha/beta/etc shouldn't we at the minimum say 'apis might change but this is the index format' ? Otherwise how is it different than a nightly build to end-users?

These are reasons why we must branch: because at any time we should be able to add new packages (like LUCENE-3305) that will 'destabilize'
packaging (which is important to get right here), or we might change the index format (like LUCENE-3687) which I think we might not necessarily want to do.

(I listed two issues that I feel are very close to being committed right now as examples)

--
lucidimagination.com

---------------------------------------------------------------------

On Jan 11, 2012, at 3:07 PM, Michael McCandless wrote:

> I'd think we could cut a 4.0 alpha today, after branching.

I'm hoping we can commit the rest of the SolrCloud stuff by next week. We have been marching strong towards that goal for a month
now when I first mentioned those intentions in the issue. It would be great to get this into a 4.0 Alpha release.
For feedback, but also because the layout in ZooKeeper is not back compat with what we currently have in trunk, and we
def don't want to have to try supporting a migration path. It would be a shame to release what we have now with the worse than experimental
caveat that you *will* have to start over shortly when SolrCloud is ready to go into a release.

P.S. I've mentioned this before, but I'm very pro doing an alpha/beta/whatever of 4.0. We still have plenty of other ducks to get in a row, but we
don't have to be too terribly concerned about that if we call it an Alpha. At the same time, lets not short change this 4.0 release process - it's been a long time in the
making.
- Mark Miller
lucidimagination.com


---------------------------------------------------------------------