I've had a lingering frustration with the recent blowup due to the
solr-commons-csv Maven artifact (SOLR-3204).  We've worked around the
particular issue (we now hide the dependency)...

But in thinking it over, and avoiding rehashing all the technical
details, what bothers me most about what happened is that the Lucene
PMC is/was held accountable for "having released code in another
project's namespace", yet, none of us realized we had done so.  We are
(or at least I am) generally ignorant of the consequences of deploying
artifacts and dependencies into Maven.

I think that's bad: I'm not comfortable being held accountable for
something I don't understand.

I am very sorry (to the Apache Commons project) that we "released"
their sources like that; I had no idea we were doing so.  Ignorance is
not an excuse and really the Lucene PMC is/was negligent...

I think, to fix this, we (the Apache Lucene PMC) should stop
officially posting artifacts to Maven ourselves.  I think it's great
if Steve (and/or others) continue to do so, just outside of the Apache
Lucene project and outside of the Lucene PMC.  This would mean the
Maven build/deploy is fully downstream from our official releases,
just like how the numerous other package managers/repositories (yum,
apt, pkg, etc.) distribute our release artifacts.

This can be beneficial for users who consume our artifacts via Maven
as well: for the 3.4.0 release, the Maven artifacts were broken, but
we couldn't change the already-released bits (SOLR-2770).  Had Maven
been downstream, this should have been easily resolved since it'd just
be a re-spin of Maven's artifacts, not Lucene/Solr's.

All of this being said, I'm very very grateful to Steve for working so
hard to understand Maven, and build/deploy the Lucene/Solr Maven
artifacts, for our releases.  I know it's a huge amount of work, and
in general rather thankless around here, being stuck between people
who love Maven and people who don't, yet Steve has done an amazing job
at it.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------

Hi,

You might all know me as "die, maven, die" person, but this special case is
definitely not related to the issue mentioned at all:

The problem we had with commons csv was *not* related to maven at all, it
also affected our standard artifacts. The problem was that we were releasing
code in the namespace of another projects in their namespace. This also
affects standard binary distribution (and at that time also source
distributions, because we were shipping a jar file using code with an
unreleased namespace of another project). I just repeat, that has nothing to
do with maven!

The second thing I wanted to say: Lucene was always shipping "maven
artifacts in the maven repository", and this is a really good thing! We were
always doing this and it helps users to use Lucene more easy. E.g., I never
download a Lucene release using the official binary JAR file, I always
download the JAR files using maven/ivy! By publishing maven artifacts, we
are not "centralized to maven", we simply offer another and very convenient
way to include Lucene into your App. On the other hand, I think, we should
*not* publish Solr using maven, because it is not a library, but a complete
application, so users must download the release to work with it.

So we must publish maven artifacts to be successful on the open source
market, sorry! (that's my opinion!) Publishing artifacts on behalf of other
parties is wrong (but that's unrelated). By publishing Maven artifacts, not
only people using Maven can have a better experience, also people using IVY
(and we are also using IVY) have a better experience and usability. As we
use Ivy, we should give back that to the community.

We also have a chicken and egg problem by using IVY. Lucene 3.6 downloads
the JAR file for backwards compatibility tests (Version 3.5) via IVY
fromMaven using the maven artifact ID. This version is only available via
IVY as Steven published them after releasing 3.5.

Finally, publishing maven artifacts has nothing to do with maven at all. We
don't need a maven build to do that! I love what Steven did, but in my
opinion, the old way of creating JAR files (which ANT does for us) and
copying them to the maven servers can be done completely without maven (at
least I did it in the past, maybe that has changed). It was simply a copy
operation of some JAR files and a metadata file (XML). As we are now using
IVY, we should maybe do the automated deploy using IVY instead of a parallel
maven build.

So my strong -1 to remove maven artifacts!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Michael McCandless [mailto:[EMAIL PROTECTED]]
> Sent: Monday, April 23, 2012 4:15 PM
> To: Lucene/Solr dev
> Subject: [DISCUSS] Build/deploy Maven artifacts outside of Lucene/Solr
>
> I've had a lingering frustration with the recent blowup due to the solr-
> commons-csv Maven artifact (SOLR-3204).  We've worked around the
> particular issue (we now hide the dependency)...
>
> But in thinking it over, and avoiding rehashing all the technical details,
what
> bothers me most about what happened is that the Lucene PMC is/was held
> accountable for "having released code in another project's namespace",
yet,
> none of us realized we had done so.  We are (or at least I am) generally
> ignorant of the consequences of deploying artifacts and dependencies into
> Maven.
>
> I think that's bad: I'm not comfortable being held accountable for
something I
> don't understand.
>
> I am very sorry (to the Apache Commons project) that we "released"
> their sources like that; I had no idea we were doing so.  Ignorance is not
an
> excuse and really the Lucene PMC is/was negligent...
>
> I think, to fix this, we (the Apache Lucene PMC) should stop officially
posting
> artifacts to Maven ourselves.  I think it's great if Steve (and/or others)
continue
> to do so, just outside of the Apache Lucene project and outside of the
Lucene
artifacts.
well:
change
hard to
our
thankless

On Apr 23, 2012, at 5:09 PM, Uwe Schindler wrote:

> So we must publish maven artifacts to be successful on the open source
> market, sorry! (that's my opinion!)

I disagree that Lucene/Solr would not be popular if the committers and PMC did not support Maven. In fact, I expect if there is real demand, a downstream contributor would easily pick this up.

I don't think anyone doubts Maven's value in terms of distribution. I also find amazing value in the apt-get system. I think if you want to be a popular open source project, you really want your stuff in a linux packaging world!

But I don't think we should make that part of the project. I think it belongs downstream where it happily happens now :)

Unless we get a wave of anti maven sentiment, it looks like things are staying as they are right now anyway though. I just don't think any of us arguing that Maven should not be our responsibility thinks that that would mean Maven support for Lucene/Solr would go away.

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------

> Unless we get a wave of anti maven sentiment, it looks like things are
staying
> as they are right now anyway though. I just don't think any of us arguing
that
> Maven should not be our responsibility thinks that that would mean Maven
> support for Lucene/Solr would go away.

+1

No Anti-Maven statement from the Maven-Hater. I only don't like Maven as
software tool, but the infrastructure behind is great and important!
---------------------------------------------------------------------

On 4/23/2012 at 5:09 PM, Uwe Schindler wrote:
> Finally, publishing maven artifacts has nothing to do with
> maven at all. We don't need a maven build to do that!

The Maven build is there to enable using Maven to check the correctness of the POMs.  I think this is essential.  Prior to this, the Lucene/Solr Maven POMs were *always* out of date, and nobody knew about it.

> in my opinion, the old way of creating JAR files (which ANT
> does for us) and copying them to the maven servers can be done
> completely without maven

That's how it's done now.  The "maven artifacts" are just the Ant-produced .jar files, along with the Maven POM files.  Copying to the maven servers is performed using Maven Ant Tasks, which runs under Ant, not Maven.

> As we are now using IVY, we should maybe do the automated
> deploy using IVY instead of a parallel maven build.

I don't know IVY - maybe it could serve the same copy-to-maven-servers purpose as Maven Ant Tasks?

And again, the parallel maven build is not involved in preparation of maven artifacts or in pushing them to Maven Central.

Steve
---------------------------------------------------------------------

Hi,

Thanks for clarification! This was one of the things that I never
understood, for me the additional maven build was just magic :-) This powers
my opinion, that the so called "maven builds" are simply another way of
publishing artifacts and that's important in my opinion, as we are using the
infrastructure of IVY, which relies on Maven servers to work (Robert: I know
that IVY also supports downloading from arbitrary URLs, but that is missing
stability and metadata).

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Steven A Rowe [mailto:[EMAIL PROTECTED]]
> Sent: Monday, April 23, 2012 11:21 PM
> To: [EMAIL PROTECTED]
> Subject: RE: [DISCUSS] Build/deploy Maven artifacts outside of Lucene/Solr
>
> On 4/23/2012 at 5:09 PM, Uwe Schindler wrote:
> > Finally, publishing maven artifacts has nothing to do with maven at
> > all. We don't need a maven build to do that!
>
> The Maven build is there to enable using Maven to check the correctness of
the
> POMs.  I think this is essential.  Prior to this, the Lucene/Solr Maven
POMs
> were *always* out of date, and nobody knew about it.
>
> > in my opinion, the old way of creating JAR files (which ANT does for
> > us) and copying them to the maven servers can be done completely
> > without maven
>
> That's how it's done now.  The "maven artifacts" are just the Ant-produced
.jar
> files, along with the Maven POM files.  Copying to the maven servers is
> performed using Maven Ant Tasks, which runs under Ant, not Maven.
>
> > As we are now using IVY, we should maybe do the automated deploy using
> > IVY instead of a parallel maven build.
>
> I don't know IVY - maybe it could serve the same copy-to-maven-servers
> purpose as Maven Ant Tasks?
>
> And again, the parallel maven build is not involved in preparation of
maven
> artifacts or in pushing them to Maven Central.
>
> Steve
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
> commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 5:29 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> Robert: I know
> that IVY also supports downloading from arbitrary URLs, but that is missing
> stability and metadata).
>

My questions have nothing to do with whether or not we release maven
artifacts (if you read them, you will see that).

Instead I'm asking how we handle bugfixes.

What if we get everything in shape, working very hard and through say
many RCs, finally we produce 4.0 RC #23434343
with fantastic documentation, easy APIs, perfect, etc, etc, .... but
during release testing someone finds a ZooKeeper bug.

Do we just wait (possibly for a long time) until they release, so that
we can? How do we handle patches to third party dependencies.

Nobody right now seems to have any kind of clue how we should handle
this situation with maven and its a serious problem, I don't care how
friendly it is, I don't care how how stable it is, or any of that if
we can't deal with this.

--
lucidimagination.com

---------------------------------------------------------------------

Rob,

I personally think that the Apache policy issues here are not entirely
worked out, but let me try to summarize the situation.

Starting condition: a marooned bugfix or feature in some other Apache TLP.

1. Communicate with the other TLP, possibly siccing Uwe on their chair.

2. If they are unresponsive or unhelpful, I think that further
discussion with the board would be apropos. However,

3. Make a choice between three alternatives:

a) Fork their source under your svn, rename packages, and go for it.
Include that source in your source release.

b) Create a patching process that grabs their release package, fixes
it, and builds it, renaming packages. Include the patching process in
your source release.

c) A volunteer goes off to github and does the necessary, with this
being that volunteer's individual effort, not an 'action of the PMC'.
In this case, you can hypothetically leave the package names alone,
but, in this case you are walking on a somewhat fine line. In this
case, you use it as a binary dependency.

Does this help?

I think that more discussion with the board would be helpful on all of
this, and I'd encourage your chair to do so.

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 5:47 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:

> b) Create a patching process that grabs their release package, fixes
> it, and builds it, renaming packages. Include the patching process in
> your source release.
>

I would *love* to be able to have this option available, but how will
the maven artifacts work? Can maven artifacts say "i depend on x.y.z
but i need to download its source and patch it first" ?

See with ant/ivy, i feel this is still an option, but when we support
maven artifacts, I feel that it shackles us, and removes options like
this.

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 5:49 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 5:47 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>
>> b) Create a patching process that grabs their release package, fixes
>> it, and builds it, renaming packages. Include the patching process in
>> your source release.
>>
>
> I would *love* to be able to have this option available, but how will
> the maven artifacts work? Can maven artifacts say "i depend on x.y.z
> but i need to download its source and patch it first" ?

You publish it. No one objects to the process of changing the packages
and publishing it under a suitable descriptive name. Or, as Uwe
pointed out, it entirely possible to use a couple of maven plugins to
pull this off dynamically.
>
> See with ant/ivy, i feel this is still an option, but when we support
> maven artifacts, I feel that it shackles us, and removes options like
> this.
>
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 6:13 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 5:49 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 23, 2012 at 5:47 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>
>>> b) Create a patching process that grabs their release package, fixes
>>> it, and builds it, renaming packages. Include the patching process in
>>> your source release.
>>>
>> I would *love* to be able to have this option available, but how will
>> the maven artifacts work? Can maven artifacts say "i depend on x.y.z
>> but i need to download its source and patch it first" ?
>
> You publish it. No one objects to the process of changing the packages
> and publishing it under a suitable descriptive name. Or, as Uwe
> pointed out, it entirely possible to use a couple of maven plugins to
> pull this off dynamically.
>

And now you hit the nail on the head. Because your 'option 2'
(download+patch) with ivy doesnt involve 'publishing'. we just
download and patch.

But with maven you MUST publish. so this is really forcing us to your
option 3 (its a published fork).

Give me an option 2 for maven that doesnt have the word 'publish' and
then its an option 2.

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 6:15 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 6:13 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 23, 2012 at 5:49 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>>> On Mon, Apr 23, 2012 at 5:47 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>>
>>>> b) Create a patching process that grabs their release package, fixes
>>>> it, and builds it, renaming packages. Include the patching process in
>>>> your source release.
>>>>
>>> I would *love* to be able to have this option available, but how will
>>> the maven artifacts work? Can maven artifacts say "i depend on x.y.z
>>> but i need to download its source and patch it first" ?
>>
>> You publish it. No one objects to the process of changing the packages
>> and publishing it under a suitable descriptive name. Or, as Uwe
>> pointed out, it entirely possible to use a couple of maven plugins to
>> pull this off dynamically.
>>
>
> And now you hit the nail on the head. Because your 'option 2'
> (download+patch) with ivy doesnt involve 'publishing'. we just
> download and patch.

I think the answer to this is Uwe's suggestion. Set up Maven to do the
download+patch.

I can't promise 100% automation of this in a single top-level maven
build for every possible complex maven-built thing you might want to
patch. I can promise no worse than two steps: (1) run a script that
downloads, patches, and builds the patched items, and (2) run the
regular maven (or ant/ivy) build.  Step (1) puts its results in your
'local repository' (~/.m2/repository) and step 2 uses them there.

For many maven things, I could see how to do this with a maven plugin
folded inside your build, but since I think that would fail in some
cases, I don't think it is wise.
>
> But with maven you MUST publish. so this is really forcing us to your
> option 3 (its a published fork).
>
> Give me an option 2 for maven that doesnt have the word 'publish' and
> then its an option 2.
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

>> And now you hit the nail on the head. Because your 'option 2'
>> (download+patch) with ivy doesnt involve 'publishing'. we just
>> download and patch.

You can take a look at Google Caliper -- they used maven for building
but required
JARs that were inside the project (not on maven central). The build
followed what Benson suggested -- deployment to a local maven
repository, then maven pulled the dependency from there and didn't
look any further.

I think they moved on to officially available JARs since it was like I
described though.

The issue with such poms is that once you want to publish your stuff
to maven central it won't work (because all dependencies must be
available and there is no "local" repository).

Dawid

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 6:28 PM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>
> The issue with such poms is that once you want to publish your stuff
> to maven central it won't work (because all dependencies must be
> available and there is no "local" repository).
>

This is the problem I am asking about.

I don't care about local maven builds or any of that. I care about
release artifacts.

If we have a buggy dependency, i can easily tweak ant+ivy (even ant by
itself) to download its source, patch it, and use it.
I can generate lucene-4.0-src.tgz, solr-4.0-src.tgz, lucene-4.0.zip,
etc etc with this.

How will the stuff in the maven/ folder work? will it work at all?

(for reference, here is a release candidate to see what i mean:
http://people.apache.org/~rmuir/staging_area/lucene-solr-3.6RC1-rev1310449/)

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 6:31 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 6:28 PM, Dawid Weiss
> <[EMAIL PROTECTED]> wrote:
>>
>> The issue with such poms is that once you want to publish your stuff
>> to maven central it won't work (because all dependencies must be
>> available and there is no "local" repository).
>>
>
> This is the problem I am asking about.
>
> I don't care about local maven builds or any of that. I care about
> release artifacts.
>
> If we have a buggy dependency, i can easily tweak ant+ivy (even ant by
> itself) to download its source, patch it, and use it.
> I can generate lucene-4.0-src.tgz, solr-4.0-src.tgz, lucene-4.0.zip,
> etc etc with this.

If your build and release process creates a modified version of
xerces.jar with unmodified package names, and that jar file ends up on
www.apache.org/dist, there will be unhappiness. You can't have such a
thing in svn, you can't have it in your binary package, you can't have
it in a 'convenience binary bundle.' You can auto-download it from
non-Apache infrastructure. You could download it from Maven central,
but you can't officially, as a committee, put it there. If have a way
to get what you want with ant(+ivy) without breaking those rules,
please explain it to me again, and I'll endeavor to provide the maven
analog.

Ultimately, what makes people unhappy is pulling modified versions,
however improved, into users' classpaths under unmodified package
names. Doing this internally for your own development is no big deal,
but people find it rude for releases. It doesn't matter what tool is
being used to produce, release, or consume. If someone outside Apache
facilitates it, the unhappiness will merely come from users who get
all confused trying to deal with multiple versions of the item.

So, in my view, you really should rename the packages if you want
these things to be used in releases. Once you rename the packages, you
can incorporate the results in your release. You can either publish
jar files under clear names, or you can incorporate them in your own
jar files. You can do these things with or without maven. You can
publish them to maven central -- as an official act of the pmc.

Thus, I claim that this debate is fundamentally about package
renaming, not publication.

Shall I be maximally annoying and mention OSGi again?

>
> How will the stuff in the maven/ folder work? will it work at all?
>
> (for reference, here is a release candidate to see what i mean:
> http://people.apache.org/~rmuir/staging_area/lucene-solr-3.6RC1-rev1310449/)
>
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 6:49 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 6:31 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 23, 2012 at 6:28 PM, Dawid Weiss
>> <[EMAIL PROTECTED]> wrote:
>>>
>>> The issue with such poms is that once you want to publish your stuff
>>> to maven central it won't work (because all dependencies must be
>>> available and there is no "local" repository).
>>>
>>
>> This is the problem I am asking about.
>>
>> I don't care about local maven builds or any of that. I care about
>> release artifacts.
>>
>> If we have a buggy dependency, i can easily tweak ant+ivy (even ant by
>> itself) to download its source, patch it, and use it.
>> I can generate lucene-4.0-src.tgz, solr-4.0-src.tgz, lucene-4.0.zip,
>> etc etc with this.
>
> If your build and release process creates a modified version of
> xerces.jar with unmodified package names, and that jar file ends up on
> www.apache.org/dist, there will be unhappiness. You can't have such a
> thing in svn, you can't have it in your binary package, you can't have
> it in a 'convenience binary bundle.' You can auto-download it from
> non-Apache infrastructure. You could download it from Maven central,
> but you can't officially, as a committee, put it there. If have a way
> to get what you want with ant(+ivy) without breaking those rules,
> please explain it to me again, and I'll endeavor to provide the maven
> analog.
>

Its pretty simple (for the record): for the lucene case the source
release would build this stuff, but we simply wouldnt provide any
third party jars in the binary release at all.

Since the lucene core itself has no dependencies, it just means if you
want to use module-xyz that uses icu you need to go and get that jar
yourself: really not that bad. In fact lucene binary releases worked
this way for a very long time.

The complications all involve solr: because its an application that
people expect to work out of box. So its binary releases are a totally
different animal. I hope this makes sense.

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 7:30 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 6:49 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 23, 2012 at 6:31 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>>> On Mon, Apr 23, 2012 at 6:28 PM, Dawid Weiss
>>> <[EMAIL PROTECTED]> wrote:
>>>>
>>>> The issue with such poms is that once you want to publish your stuff
>>>> to maven central it won't work (because all dependencies must be
>>>> available and there is no "local" repository).
>>>>
>>>
>>> This is the problem I am asking about.
>>>
>>> I don't care about local maven builds or any of that. I care about
>>> release artifacts.
>>>
>>> If we have a buggy dependency, i can easily tweak ant+ivy (even ant by
>>> itself) to download its source, patch it, and use it.
>>> I can generate lucene-4.0-src.tgz, solr-4.0-src.tgz, lucene-4.0.zip,
>>> etc etc with this.
>>
>> If your build and release process creates a modified version of
>> xerces.jar with unmodified package names, and that jar file ends up on
>> www.apache.org/dist, there will be unhappiness. You can't have such a
>> thing in svn, you can't have it in your binary package, you can't have
>> it in a 'convenience binary bundle.' You can auto-download it from
>> non-Apache infrastructure. You could download it from Maven central,
>> but you can't officially, as a committee, put it there. If have a way
>> to get what you want with ant(+ivy) without breaking those rules,
>> please explain it to me again, and I'll endeavor to provide the maven
>> analog.
>>
>
> Its pretty simple (for the record): for the lucene case the source
> release would build this stuff, but we simply wouldnt provide any
> third party jars in the binary release at all.
>
> Since the lucene core itself has no dependencies, it just means if you
> want to use module-xyz that uses icu you need to go and get that jar
> yourself: really not that bad. In fact lucene binary releases worked
> this way for a very long time.

One way to read the above in the light of your previous message would
be: 'just go ahead and publish Lucene stuff to Maven Central pointing
at the unpatched dependencies, and leave it to the competent
programmer-type users to pick up the source of the patches, build
them, and use them if they care.' Works for me and simple as mud, so
long as you aren't dealing with a dependency so badly broken that
module-xxx doesn't work *at all* without it. In which case, I'd be
trying to convince you that, whatever the build tool, you want to
package-rename and incorporate the patched-up version, so as to
deliver a working module.

>
> The complications all involve solr: because its an application that
> people expect to work out of box. So its binary releases are a totally
> different animal. I hope this makes sense.

It makes perfect sense. Some of us do, however, embed Solr (for tests
or other unrecommended reasons), or build our own Solr plugins with
dependencies on the existing Solr plugins, and we would thus mourn, or
have to go back to doing extra work, if you stopped publication of
Solr artifacts altogether.

However, you could stop publishing the full release package out there
in Maven central but continued to push the bits and pieces. And those
don't need to work perfectly out of the box in the sense of your
comments above about Lucene. I confess that I've scripted downloading
the 'real release' from Maven, but I could script downloading it from
dist without any horrible loss.

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 7:46 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
> It makes perfect sense. Some of us do, however, embed Solr (for tests
> or other unrecommended reasons), or build our own Solr plugins with
> dependencies on the existing Solr plugins, and we would thus mourn, or
> have to go back to doing extra work, if you stopped publication of
> Solr artifacts altogether.
>

Yeah I actually think the monolithic approach is not ideal, i would
prefer something more modular.

But i'm balancing this against the costs of over 100 third party
dependencies, which is currently overwhelming the project (or at
least, release managers).

We might be able to simplify things with other approaches, too: just
as an example, over half of solr's third-party jars are in contrib
modules, some things that might be a better fit as client-side
integrations or just separate plugins, (maybe as separate projects
under our TLP with their own release cycles, or on apache-extras, or
anything like that).

Or maybe we can just do a full scrub and try to see which dependencies
we can remove. I bet there are a few we could nuke.

Its definitely overwhelming right now, the size of the codebase and
the number of dependencies, given what we have. I think we should be
considering all options. Maven packaging just adds to the complexity
due to the fact of how it manages dependencies (they must themselves
be in maven somehow).
--
lucidimagination.com

---------------------------------------------------------------------

> Its definitely overwhelming right now, the size of the codebase and
> the number of dependencies, given what we have. I think we should be
> considering all options. Maven packaging just adds to the complexity
> due to the fact of how it manages dependencies (they must themselves
> be in maven somehow).

If you like (or can live with) everything about the situation except
for the additional Maven burden, then it's worthwhile for us
Maven-heads to offer schemes that purport to do Maven without making
it (much) worse. But if the situation has grown beyond what the RMs
should have to deal with even without Maven, then we're (or at least
I'm) just adding noise.

If the Maven straw that breaks the camel's back is 'how do we get the
fixed binaries into the Maven universe," I want to underline my
previous failure to be clear when talking about 'the github trick'. If
a group of folks wants to make forks on github of whatever, Apache or
not, and apply bugfixes and push, they can do that. If they organize
their work on this list or in this list's JIRA, they might take some
flak. They might win the resulting argument. Especially if in each
case they can document that they diligently tried to work with the
upstream.

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 8:28 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>
> If you like (or can live with) everything about the situation except
> for the additional Maven burden, then it's worthwhile for us
> Maven-heads to offer schemes that purport to do Maven without making
> it (much) worse. But if the situation has grown beyond what the RMs
> should have to deal with even without Maven, then we're (or at least
> I'm) just adding noise.

Well I think Steve is doing a fantastic job. I worry if he gets by a
bus, what would we do?

>
> If the Maven straw that breaks the camel's back is 'how do we get the
> fixed binaries into the Maven universe," I want to underline my
> previous failure to be clear when talking about 'the github trick'. If
> a group of folks wants to make forks on github of whatever, Apache or
> not, and apply bugfixes and push, they can do that. If they organize
> their work on this list or in this list's JIRA, they might take some
> flak. They might win the resulting argument. Especially if in each
> case they can document that they diligently tried to work with the
> upstream.

I don't think its the straw the broke the camel's back but it
definitely has a cost. For 3.6, i know i spent a ton of time (along
with all the maven supporters, who stayed up all night and all that
kind of stuff too helping out), trying to get everything to be as
'clean' as possible. I don't know maven at all and I was busy trying
all kinds of possibilities (like making fake maven repositories under
google code projects and all kinds of nasty stuff, fortunately we
didnt have to resort to most of that as other avenues worked out).

I found this was really difficult, like herding 100 cats. I think its
not unrealistic or asking too much for our releases to be legit. :)
But its also not asking too much for them to have good quality, if a
bug is serious in a dependency, we should have a way to fix it without
doing something dirty.

Like i said before, in my perfect world without maven, lucene releases
just wouldnt have any jars ever, and if we need to patch one we just
have scripts that do that.

But as long as maven is part of the actual release artifacts the way
it is now, we can't even consider approaches like that, and need a
strategy in place for 'if/when we need to have a patched dependency,
this is what we will do'.

--
lucidimagination.com

---------------------------------------------------------------------

I'm with Steve here, -1.  I think the arguments have already been well
articulated and the issue seems to be about dependency management in
general not just about Maven.  However I also absolutely agree with Robert
that we need to make this easier somehow.

One thing I thought, is there any way we can get our own repo? In
organisations that use Maven for their projects, that's how they tend to
handle this situation.  They have their own repo for dependencies they need
to manage and they use repo1 for the rest.   As an ASF project, do we get
any help in this regards?

On Tue, Apr 24, 2012 at 12:28 PM, Benson Margulies <[EMAIL PROTECTED]>wrote:

> > Its definitely overwhelming right now, the size of the codebase and
> > the number of dependencies, given what we have. I think we should be
> > considering all options. Maven packaging just adds to the complexity
> > due to the fact of how it manages dependencies (they must themselves
> > be in maven somehow).
>
> If you like (or can live with) everything about the situation except
> for the additional Maven burden, then it's worthwhile for us
> Maven-heads to offer schemes that purport to do Maven without making
> it (much) worse. But if the situation has grown beyond what the RMs
> should have to deal with even without Maven, then we're (or at least
> I'm) just adding noise.
>
> If the Maven straw that breaks the camel's back is 'how do we get the
> fixed binaries into the Maven universe," I want to underline my
> previous failure to be clear when talking about 'the github trick'. If
> a group of folks wants to make forks on github of whatever, Apache or
> not, and apply bugfixes and push, they can do that. If they organize
> their work on this list or in this list's JIRA, they might take some
> flak. They might win the resulting argument. Especially if in each
> case they can document that they diligently tried to work with the
> upstream.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Chris Male | Software Developer | DutchWorks | www.dutchworks.nl

On Mon, Apr 23, 2012 at 6:49 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:

> Thus, I claim that this debate is fundamentally about package
> renaming, not publication.
>
> Shall I be maximally annoying and mention OSGi again?
>

Its not really a debate, its 'how do we do this'. (I can argue your
claim is wrong, but i won't, ive said it before and I stick with what
I've said)...

The current problem is for releases, how to handle this stuff is
currently confusing. There are a few ideas here, and some of them
maybe are possible, but we need to come to some sort of agreement on
how to handle these kind of situations. Currently we have no patched
jars, magically, but since we depend on over 100 of them, the
probability of this situation staying the same for any length of time
is very low.

I don't really care what we do, but i just want:
* a process for dealing with patched dependencies (that doesn't
involve illegal publication of other people's stuff in maven under our
name)
* a process for dealing with dependencies that aren't in maven (that
doesn't involve illegal publication of other people's stuff in maven
under our name)
* the two processes to not be the release manager's job to deal with.
* the two processes to be easy enough to understand to pmc members so
they actually understand WTF is going on with our releases without it
being a huge mystery.

One implication of what I want is that 'for development' (whats
committed to trunk) is the same as 'for releases'. Because anything
else is just a big 'RM, you go deal with fixing this later'. So I
don't think we should let trunk get all kinds of screwy dependencies
and leave it to release managers to fix up the mess.

Another is that even if we work and figure all this out, its still so
crazy complicated that a bunch of people here just say 'this isnt
worth it, as a PMC member i dont understand all this, this is supposed
to be a search engine project' (I'm not trying to call out Mike on
this, but his email maybe alludes that some people could possibly feel
this way about maven).

And as a side note: Uwe's ideas bring up a nice potential compromise
for simplification: maybe just lucene (as an API) should be in maven
and not solr (as an application?). Its worth thinking about: solr has
many many more third-party dependencies.  Dependencies are really
really expensive.

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 7:21 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 6:49 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>
>> Thus, I claim that this debate is fundamentally about package
>> renaming, not publication.
>>
>> Shall I be maximally annoying and mention OSGi again?
>>
>
> Its not really a debate, its 'how do we do this'. (I can argue your
> claim is wrong, but i won't, ive said it before and I stick with what
> I've said)...
>
> The current problem is for releases, how to handle this stuff is
> currently confusing. There are a few ideas here, and some of them
> maybe are possible, but we need to come to some sort of agreement on
> how to handle these kind of situations. Currently we have no patched
> jars, magically, but since we depend on over 100 of them, the
> probability of this situation staying the same for any length of time
> is very low.
>
> I don't really care what we do, but i just want:
> * a process for dealing with patched dependencies (that doesn't
> involve illegal publication of other people's stuff in maven under our
> name)
> * a process for dealing with dependencies that aren't in maven (that
> doesn't involve illegal publication of other people's stuff in maven
> under our name)
> * the two processes to not be the release manager's job to deal with.
> * the two processes to be easy enough to understand to pmc members so
> they actually understand WTF is going on with our releases without it
> being a huge mystery.
>
> One implication of what I want is that 'for development' (whats
> committed to trunk) is the same as 'for releases'. Because anything
> else is just a big 'RM, you go deal with fixing this later'. So I
> don't think we should let trunk get all kinds of screwy dependencies
> and leave it to release managers to fix up the mess.
>
> Another is that even if we work and figure all this out, its still so
> crazy complicated that a bunch of people here just say 'this isnt
> worth it, as a PMC member i dont understand all this, this is supposed
> to be a search engine project' (I'm not trying to call out Mike on
> this, but his email maybe alludes that some people could possibly feel
> this way about maven).
>
> And as a side note: Uwe's ideas bring up a nice potential compromise
> for simplification: maybe just lucene (as an API) should be in maven
> and not solr (as an application?). Its worth thinking about: solr has
> many many more third-party dependencies.  Dependencies are really
> really expensive.
>

I have a suggestion for how to get from here to the exit.

Forget about maven for the moment, and please explain to me exactly
how you'd like to handle bugfixes to other Apache items in releases if
all you had to worry about was ant(+ivy).

If I or others can hand you a simple enough prescription for how to
extend that to Maven builds and Maven releases, we're done. If we
can't, we're also done, where 'enough' is of course defined by the
consensus of the community, not just you and me.

For things that are not Apache, there's an very simple way to deal
with all this using Apache Extras, and no need to spill email over it
so far.

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 7:31 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
> I have a suggestion for how to get from here to the exit.
>
> Forget about maven for the moment, and please explain to me exactly
> how you'd like to handle bugfixes to other Apache items in releases if
> all you had to worry about was ant(+ivy).
>

OK (but this is just my opinion, and I'm sure a ton of people disagree).

we have to separate out how to handle each product, in my opinion
lucene/solrj/and solr are all different here.

for Lucene, as i mentioned, I dont think we ever have to really
include any jars in anything (not even the binary release).
You can look at 2.9's binary release, it has no third party jars
(except servlet-api.jar, which was unnecessary). People managed and
got anything they needed on their own.
Most lucene modules don't depend on third party stuff... its just a
fact, and for a long time nobody ever complained about this.
Instead we could just generate (maybe with XSLT) a .txt file listing
the dependencies and their versions and people get them on their own.
That frees us totally from jars completely for lucene. Source release
can download with ivy, patch with ant, who cares.

for Solr, as an application, maybe we should just be making a massive
solr.jar anyway for the binary release that you java -jar (leaving the
webapp as an implementation detail or something): really I have no
idea. We could jarjar all 'jars that are implementation details' just
as a matter of course and there would never be any issues at all.

for Solrj, this is really a client library, so i think its different.
but its dependencies (similar, but not the same as lucene's) are very
minimal... perhaps the solrj binary release could include these jars
since there just isnt that many of them to make it easy for people to
integrate solrj, or not, doesn't matter.

So thats my idea of something that would really simplify all of this:
* simplifying and making lucene releases smaller, instead in binary
releases just documenting what we depend on (even if thats -PATCHED).
* hiding third-party dependencies in solr releases as an implementation detail.
* some kind of hybrid (maybe just what we do today) for solrj.

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 7:45 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 7:31 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>> I have a suggestion for how to get from here to the exit.
>>
>> Forget about maven for the moment, and please explain to me exactly
>> how you'd like to handle bugfixes to other Apache items in releases if
>> all you had to worry about was ant(+ivy).
>>
>
> OK (but this is just my opinion, and I'm sure a ton of people disagree).
>
> we have to separate out how to handle each product, in my opinion
> lucene/solrj/and solr are all different here.
>
> for Lucene, as i mentioned, I dont think we ever have to really
> include any jars in anything (not even the binary release).
> You can look at 2.9's binary release, it has no third party jars
> (except servlet-api.jar, which was unnecessary). People managed and
> got anything they needed on their own.
> Most lucene modules don't depend on third party stuff... its just a
> fact, and for a long time nobody ever complained about this.
> Instead we could just generate (maybe with XSLT) a .txt file listing
> the dependencies and their versions and people get them on their own.
> That frees us totally from jars completely for lucene. Source release
> can download with ivy, patch with ant, who cares.
>
> for Solr, as an application, maybe we should just be making a massive
> solr.jar anyway for the binary release that you java -jar (leaving the
> webapp as an implementation detail or something): really I have no
> idea. We could jarjar all 'jars that are implementation details' just
> as a matter of course and there would never be any issues at all.
>
> for Solrj, this is really a client library, so i think its different.
> but its dependencies (similar, but not the same as lucene's) are very
> minimal... perhaps the solrj binary release could include these jars
> since there just isnt that many of them to make it easy for people to
> integrate solrj, or not, doesn't matter.
>
> So thats my idea of something that would really simplify all of this:
> * simplifying and making lucene releases smaller, instead in binary
> releases just documenting what we depend on (even if thats -PATCHED).
> * hiding third-party dependencies in solr releases as an implementation detail.
> * some kind of hybrid (maybe just what we do today) for solrj.
>

If you go to this point, yes, indeed, all this bother about patched
dependencies goes away. I completely agree. And you get a cigar from
Roy Fielding, who reminds us all periodically that 'Apache Releases
are Source Releases', and all these binaries are irrelevant noise. And
perhaps some of us would indeed club together to do maven publication
on the side, though we couldn't call the results 'org.apache.'
anything in naming the artifacts.

However, you(all)  accepted all of Steve's commits because a bunch of
noisy users like me begged and pleaded for this project to act like a
whole raft of other projects that publish binaries out to Maven
Central.

I think that you are spot-on in diagnosing the difference between the
full release package of Solr and everything else. If there's one thing
you want to deliver in binary, with working dependencies, that's it.
And, I believe, you could defend a decision to deliver patched
binaries with unrenamed packages in such a package, since you would't
be publishing them into application classpaths. I can't personally
guarantee smooth sailing there, but I'd be willing to help push for a
rational view at the Foundation level.

However, that leaves us annoying whiners who wanted Lucene, and the
Solr modules we need as dependencies when building our own Solr
plugins, and maybe even embeddable Solr, as Maven dependencies.
Personally, I could live with a policy in which you publish these with
unpatched dependencies and document the situation and the
dependencies, or a number of other schemes in which you don't thread
the needle of publishing bugfix binaries without offending people, up
to and including a downloadable source package that automates local
creation of patched binaries that the user can then choose to use as
dependencies.

It's also really quite wonderful that I can download the trunk, run a
script, type mvn, and incorporate your latest and greatest into my
stuff.

At the bottom line here, it seems to me, you're looking hard at real
flaws in the combination of Java and the repository/dependency systems
we have, and bridling at participating. All I can tell you is that
there's an awful lot of successful participation out there in spite of
the flaws.

> for Solr, as an application, maybe we should just be making a massive
> solr.jar anyway for the binary release that you java -jar (leaving the
> webapp as an implementation detail or something):

The solr application is solr.war (not .jar) -- we can safely dump
whatever hacked/patched jar files into the .war and that is fine.

The solr.jar is often used by other applications as a library.

FWIW, I see no reason to publish the .war file in the maven repo:
http://people.apache.org/~rmuir/staging_area/lucene-solr-3.6RC1-rev1310449/solr/maven/org/apache/solr/solr/3.6.0/
but I don't think it really matters one way or the other

ryan

---------------------------------------------------------------------

That works, you can download the source release using maven ("source" config) and patch it (not automatically, but it's possible!).

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Robert Muir [mailto:[EMAIL PROTECTED]]
> Sent: Monday, April 23, 2012 11:50 PM
> To: [EMAIL PROTECTED]
> Subject: Re: [DISCUSS] Build/deploy Maven artifacts outside of Lucene/Solr
>
> On Mon, Apr 23, 2012 at 5:47 PM, Benson Margulies
> <[EMAIL PROTECTED]> wrote:
>
> > b) Create a patching process that grabs their release package, fixes
> > it, and builds it, renaming packages. Include the patching process in
> > your source release.
> >
>
> I would *love* to be able to have this option available, but how will the maven
> artifacts work? Can maven artifacts say "i depend on x.y.z but i need to
> download its source and patch it first" ?
>
> See with ant/ivy, i feel this is still an option, but when we support maven
> artifacts, I feel that it shackles us, and removes options like this.
>
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
> commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

> Instead I'm asking how we handle bugfixes.

This is a real problem. I've gone through this with recent updates to
randomizedtesting package. There is
no "clean" way of integrating a snapshot build or an interim package.
With maven the integration can be done via snapshot repositories
(there are drawbacks like moving targets for builds, but I'm not going
to go there). With ivy the same can be done, but you need to
physically publish a JAR somewhere (and what, keep it forever? until
the next update?).

I don't have any good ideas how to solve this but I see the advantage
of just storing a versioned binary in the project's repo...

D.

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 5:40 PM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>> Instead I'm asking how we handle bugfixes.
>
> This is a real problem. I've gone through this with recent updates to
> randomizedtesting package. There is
> no "clean" way of integrating a snapshot build or an interim package.

Thank you! This is all i was bringing up all along.

Its not pro-maven, its not anti-maven. Its "we have to be able to do this".

We depend upon 144 jars (as of 'ant resolve' ; find -name "*.jar" | wc
-l), we are bound to find bugs, and some might really completely make
solr unusable or something like that. I just want us to have an
agreed-upon process: "here is how we do this, that isnt dirty"
--
lucidimagination.com

---------------------------------------------------------------------

On 4/23/2012 at 5:33 PM, Robert Muir wrote:
> On Mon, Apr 23, 2012 at 5:29 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> > Robert: I know that IVY also supports downloading from arbitrary
> > URLs, but that is missing stability and metadata).
>
> My questions have nothing to do with whether or not we release maven
> artifacts (if you read them, you will see that).
>
> Instead I'm asking how we handle bugfixes.
>
> What if we get everything in shape, working very hard and through
> say many RCs, finally we produce 4.0 RC #23434343 with fantastic
> documentation, easy APIs, perfect, etc, etc, .... but during
> release testing someone finds a ZooKeeper bug.
>
> Do we just wait (possibly for a long time) until they release,
> so that we can? How do we handle patches to third party dependencies.
>
> Nobody right now seems to have any kind of clue how we should handle
> this situation with maven and its a serious problem, I don't care how
> friendly it is, I don't care how how stable it is, or any of that if
> we can't deal with this.

Maybe I'm missing something, but why can't maven artifacts be pushed from the github fork of a third-party dependency?
---------------------------------------------------------------------

Hi,

The answer is quite clear: Not our problem! Note that bug in our release notes that there is a not-yet fixed zookeeper bug. We cannot make the world perfect. If there is a bug outside our responsibility, we can document it, but not necessarily fix it.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Robert Muir [mailto:[EMAIL PROTECTED]]
> Sent: Monday, April 23, 2012 11:33 PM
> To: [EMAIL PROTECTED]
> Subject: Re: [DISCUSS] Build/deploy Maven artifacts outside of Lucene/Solr
>
> On Mon, Apr 23, 2012 at 5:29 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> > Robert: I know
> > that IVY also supports downloading from arbitrary URLs, but that is
> > missing stability and metadata).
> >
>
> My questions have nothing to do with whether or not we release maven
> artifacts (if you read them, you will see that).
>
> Instead I'm asking how we handle bugfixes.
>
> What if we get everything in shape, working very hard and through say many
> RCs, finally we produce 4.0 RC #23434343 with fantastic documentation, easy
> APIs, perfect, etc, etc, .... but during release testing someone finds a ZooKeeper
> bug.
>
> Do we just wait (possibly for a long time) until they release, so that we can?
> How do we handle patches to third party dependencies.
>
> Nobody right now seems to have any kind of clue how we should handle this
> situation with maven and its a serious problem, I don't care how friendly it is, I
> don't care how how stable it is, or any of that if we can't deal with this.
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
> commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 5:37 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> Hi,
>
> The answer is quite clear: Not our problem! Note that bug in our release notes that there is a not-yet fixed zookeeper bug. We cannot make the world perfect. If there is a bug outside our responsibility, we can document it, but not necessarily fix it.
>

Just bookmarking this response to link to in the next release vote, so
I can reference it for the guy that i know likes to run tests with
-Dmultiplier=10000 -Diters=10000 and find horrible JVM bugs, etc
before release :)

--
lucidimagination.com

---------------------------------------------------------------------

Hi,

Don't be disappointed - I just say this: I would never have denied a release with the Jetty UTF-8 bug! This bug was clearly out of our scope; so I don’t care, I helped to investigate it, but fixing was not in Lucene's responsibility, sorry. A release with the Jetty UTF-8 bug or the XALAN XSLTC-Uppercase-Bug would always get +1 from me - especially as Apache releases source code not binary artifacts. And OUR source code was correct - PERIOD. If it does not work with Jetty version XYZ its not our problem.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: [EMAIL PROTECTED]
> -----Original Message-----
> From: Robert Muir [mailto:[EMAIL PROTECTED]]
> Sent: Monday, April 23, 2012 11:39 PM
> To: [EMAIL PROTECTED]
> Subject: Re: [DISCUSS] Build/deploy Maven artifacts outside of Lucene/Solr
>
> On Mon, Apr 23, 2012 at 5:37 PM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > The answer is quite clear: Not our problem! Note that bug in our release
> notes that there is a not-yet fixed zookeeper bug. We cannot make the world
> perfect. If there is a bug outside our responsibility, we can document it, but not
> necessarily fix it.
> >
>
> Just bookmarking this response to link to in the next release vote, so I can
> reference it for the guy that i know likes to run tests with
> -Dmultiplier=10000 -Diters=10000 and find horrible JVM bugs, etc before
> release :)
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED] For additional
> commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------

On Apr 23, 2012, at 5:46 PM, Uwe Schindler wrote:

>  And OUR source code was correct - PERIOD.

I think more important than our code being correct is that our distributions work correctly. Being correct for the sake of it has no practical value. We should strive for the best user experience. The value comes from what we deliver - and if we can deliver something that works correctly over something that does not, we should IMO.

It's the same as working around JVM bugs - should we need to? No. But we live in the real world where people use real JVMs.

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 5:54 PM, Mark Miller <[EMAIL PROTECTED]> wrote:
>
> On Apr 23, 2012, at 5:46 PM, Uwe Schindler wrote:
>
>>  And OUR source code was correct - PERIOD.
>
> I think more important than our code being correct is that our distributions work correctly. Being correct for the sake of it has no practical value. We should strive for the best user experience. The value comes from what we deliver - and if we can deliver something that works correctly over something that does not, we should IMO.
>

+1, thats how i feel too. Maybe others disagree: thats ok. Whats
important I think is that as a community, at least we have the option?

At the least we should be able to discuss on a case by case basis and
have the opportunity to say 'this shoudl really be fixed, otherwise
the release is FUBAR and won't actually work, who cares where the
blame lies'

I feel that currently, there is no option (maven limits us here). I'm
not sure if thats due to actual technical limitations in maven, or
just because its complicated or unconventional or whatever, but I
don't think we should let maven hold us back here.

I'd rather have distributions that work correctly without maven than
buggy distributions with maven.

--
lucidimagination.com

---------------------------------------------------------------------

> I feel that currently, there is no option (maven limits us here). I'm
> not sure if thats due to actual technical limitations in maven, or

What Benson mentioned will work for maven, I don't see how it holds
you back. If you publish an artefact for Ivy somewhere you can package
this JAR and publish it to maven central as a third party?

There is also a hackier alternative -- set up your own maven repo, add
it to master Lucene pom.xml and voila, you can publish anything you
like. Will work with maven and ivy at the same time...

Dawid

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 6:11 PM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>> I feel that currently, there is no option (maven limits us here). I'm
>> not sure if thats due to actual technical limitations in maven, or
>
> What Benson mentioned will work for maven, I don't see how it holds
> you back. If you publish an artefact for Ivy somewhere you can package
> this JAR and publish it to maven central as a third party?
>
> There is also a hackier alternative -- set up your own maven repo, add
> it to master Lucene pom.xml and voila, you can publish anything you
> like. Will work with maven and ivy at the same time...

Just for emphasis: no one objected to 'publish bugfixed binaries.'
They objected to 'publish bugfixed binaries with unchanged package
names.' I personally think that there is room for a more nuanced
discussion at the Foundation level about this, but, for now, if you
change the package names, you can publish -- whether to maven central,
or some other repo, or in a 'convenience binary' package on
www.apache.org/dist.

I fully appreciate that changing the package names is a pita, and so I
think that Uwe's emphasis on scripted patching, supported in both
Maven and ant/ivy, is good. And I'd help build it.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On 4/23/2012 at 5:58 PM, Robert Muir wrote:
> I feel that currently, there is no option (maven limits us here).
> I'm not sure if thats due to actual technical limitations in maven,
> or just because its complicated or unconventional or whatever, but
> I don't think we should let maven hold us back here.

The workflow for publishing third-party bugfix Maven artifacts:

1. Either change the source package names, or not.

2. Patch and build.

3. Publish to Sonatype's OSS repo using a different groupId.

How exactly does this "hold us back"?

AFAICT, the *only* difference Maven brings is wide use of its repository.  (That is, there is a greater likelihood that other people will notice what we have done and possibly take exception to it.)
 
Steve

> copying them to the maven servers can be done completely without maven (at

This is true but doing it via maven (that is: compiling and testing
with actual dependencies) helps you to ensure it really is a usable
artefact. I've been doing tricks like building a JAR in ANT and then
simply attaching it to a POM but this sucks in the long run as you
need to hand-check POMs every time. Think of a parallel maven build as
an automated test suite :)

As always, there is a drawback -- maven-compiled JARs will be
different from those compiled using other tools (timestamps, possibly
manifest entries if I remember correctly).

Dawid

---------------------------------------------------------------------

> But in thinking it over, and avoiding rehashing all the technical
> details, what bothers me most about what happened is that the Lucene
> PMC is/was held accountable for "having released code in another
> project's namespace", yet, none of us realized we had done so.  We are

I don't know how maven is of relevance here. Yes, maven repositories
will last forever but it's no different than the release of binary
packages that contained modified binaries of another Apache project
(they shouldn't be re-released and once released, they'll have the
life of their own). To me it's the same thing from technical and moral
point of view -- nothing connected to maven.

> I think that's bad: I'm not comfortable being held accountable for
> something I don't understand.

That's taking things too seriously I think. Nobody in the maven
community will blink an eye at this...

> their sources like that; I had no idea we were doing so.  Ignorance is
> not an excuse and really the Lucene PMC is/was negligent...

...and I don't see negligence here. We were not aware this can raise
legal problems from the board, were we? Steven may disagree but I
still fail to find something offensive in publishing an artefact in
the same Java package namespace but _with a different group/artefact
ID_. Yes, ok -- there is a possibility of a package class clashes with
official releases (and it's bad) but this isn't like we just sneakily
published commons-csv under their own group/artefact ID to confuse
people.

> I think, to fix this, we (the Apache Lucene PMC) should stop
> officially posting artifacts to Maven ourselves.  I think it's great

I'm -1 to this. Many people use maven, we have maven experts as
committers... I don't see an issue here.

Dawid

---------------------------------------------------------------------

I agree with David.  The maven error was not earth shattering --
working within the PMC was a good thing for the community at large.
We made the requested changes within a few days now things are better.
 Imagining a parallel world where the PMC was not involved turns out
much worse for many users.

-1 to remove maven support from the pseudo-non-official build

On Mon, Apr 23, 2012 at 10:01 AM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>> But in thinking it over, and avoiding rehashing all the technical
>> details, what bothers me most about what happened is that the Lucene
>> PMC is/was held accountable for "having released code in another
>> project's namespace", yet, none of us realized we had done so.  We are
>
> I don't know how maven is of relevance here. Yes, maven repositories
> will last forever but it's no different than the release of binary
> packages that contained modified binaries of another Apache project
> (they shouldn't be re-released and once released, they'll have the
> life of their own). To me it's the same thing from technical and moral
> point of view -- nothing connected to maven.
>
>> I think that's bad: I'm not comfortable being held accountable for
>> something I don't understand.
>
> That's taking things too seriously I think. Nobody in the maven
> community will blink an eye at this...
>
>> their sources like that; I had no idea we were doing so.  Ignorance is
>> not an excuse and really the Lucene PMC is/was negligent...
>
> ...and I don't see negligence here. We were not aware this can raise
> legal problems from the board, were we? Steven may disagree but I
> still fail to find something offensive in publishing an artefact in
> the same Java package namespace but _with a different group/artefact
> ID_. Yes, ok -- there is a possibility of a package class clashes with
> official releases (and it's bad) but this isn't like we just sneakily
> published commons-csv under their own group/artefact ID to confuse
> people.
>
>> I think, to fix this, we (the Apache Lucene PMC) should stop
>> officially posting artifacts to Maven ourselves.  I think it's great
>
> I'm -1 to this. Many people use maven, we have maven experts as
> committers... I don't see an issue here.
>
> Dawid
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

> I think, to fix this, we (the Apache Lucene PMC) should stop
> officially posting artifacts to Maven ourselves.

-1

I'm not happy that we're having this conversation again.

I personally don't have the resources to set up continuous integration for this, and I think that's a requirement that isn't served by other hosting providers.  Am I wrong?

> I'm not comfortable being held accountable for something I don't understand.

I don't think full understanding is a requirement.  Do you, Mike, fully understand all of Lucene/Solr's code?  If the answer is no, then how can you be comfortable voting for a release?

Steve

---------------------------------------------------------------------

when I see ' viral' in one of these debates, I get frustrated. please
state an actual practical problem, not a scare word.

publishing to central should be a central concern of this pmc, as an
ever-growing multitude uses gradle or ivy or maven or some other
consumer.

if you want a second volunteer to care and feed you have me.
On Apr 23, 2012, at 12:39 PM, Steven A Rowe <[EMAIL PROTECTED]> wrote:

>> I think, to fix this, we (the Apache Lucene PMC) should stop
>> officially posting artifacts to Maven ourselves.
>
> -1
>
> I'm not happy that we're having this conversation again.
>
> I personally don't have the resources to set up continuous integration for this, and I think that's a requirement that isn't served by other hosting providers.  Am I wrong?
>
>> I'm not comfortable being held accountable for something I don't understand.
>
> I don't think full understanding is a requirement.  Do you, Mike, fully understand all of Lucene/Solr's code?  If the answer is no, then how can you be comfortable voting for a release?
>
> Steve
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Apr 23, 2012, at 1:00 PM, Benson Margulies wrote:

> if you want a second volunteer to care and feed you have me.

I do feel better the more people that are contributing - at the same time that scares me too ;) The world moves closer and closer to where I *must* eventually contribute to these maven things.

I just would rather not. If the dev community (given their own thoughts and others) wants to support Maven though, then it just it was it is. We can't all be happy about everything ;)

I'd rather not have to worry about Maven as a PMC member or as a committer though. I'd much prefer that it was downstream and my vote will reflect that. But if the majority of us would vote to keep around Maven, I respect that.

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------

Mark,

As penance for my initial snappish response, here is a bit of
additional (I hope) constructive commentary, which may or may not make
you feel any better. As a mere occasional patch contributor I accept
that the decision should reflect the PMC as a whole, not especially my
view.

Publishing artifacts benefits users; I acknowledge that you accept
that, and are weighing it against costs. Taking on this publication
process is like any adding support for any other additional
infrastructure in incurring costs. On the other hand, the ivy
integration improves (in my opinion) the maintainability of the
lucene/solr build's usage of dependencies.

The kerfuffle with the board was, as someone else mentioned, really
not about Maven repositories at all. It was about a strangely
under-documented Apache fundamental policy about binaries, combined
with a completely undocumented Apache 'policy' about Other People's
Package names, not written down anywhere, not thought through, and
not, in my opinion, viable in the simplified form that it was
presented as a 'rule.' Taken to it's logical conclusion, you could
have suffered just as much hostile email for shipping a jar file in a
release package names 'mutant-version-of-commons-csv.jar' as you took
for publishing one under unique Maven coordinates.

There's a board election coming up, and my sadness at how some current
board members handled the situation will be reflected in my votes.

So, if you are in the market for any advice from me, try not to let
the pain of that incident add to whatever xeno-discomfort you have
with the Maven work on its own merits or lack thereof.

--benson

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 1:45 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:

> Publishing artifacts benefits users; I acknowledge that you accept
> that, and are weighing it against costs. Taking on this publication
> process is like any adding support for any other additional
> infrastructure in incurring costs.
>

This is a good point. Lately I worry that I'm acting like an old fart
too much on these kinds of issues: e.g. that it would be bad for us to
not support maven, bad for us to not support some serialization
support that makes lucene easier, or numerous other examples :)

for 3.6 we spent a lot of time cleaning up the 'hacks' in both 3.6 and
trunk so that hopefully nothing is questionable here about the way we
were doing things. Thanks to everyone that helped with this
(especially the maven parts, i wanted the maven artifacts to be
'clean').

On the other hand I have serious concerns about the tradeoffs here,
which no maven supporters have as of yet been able to answer:

How do we handle situations like
https://issues.apache.org/jira/browse/XERCESJ-1257 (open for 5 years,
still not incorporated in a release), without doing hackish things
(releasing someone elses code, sucking their entire codebase into
ours), or relying upon Uwe Schindler to come up with some clever
workaround?

I'm really worried about this, that in order to keep our releases
"legally" clean from a maven perspective (ivy can download a jar from
anywhere), that we have to sacrifice quality to get it.

I don't want this to get lost in the controversy: I'm willing to spin
off a new thread about this if we need. I'm really concerned about it.

--
lucidimagination.com

---------------------------------------------------------------------

>
> How do we handle situations like
> https://issues.apache.org/jira/browse/XERCESJ-1257 (open for 5 years,
> still not incorporated in a release), without doing hackish things
> (releasing someone elses code, sucking their entire codebase into
> ours), or relying upon Uwe Schindler to come up with some clever
> workaround?

As far as I can tell, you have exactly the same issues solving this
issue with and without Maven.

First thought: the board should be just as willing to apply pressure
to a project to deal with a patch sitting around for five years as to
deal with a project which has 'inappropriately' published another
project's stuff.

However, we're all volunteers around here, and everybody understands
that this kind of situation may ensue.

So; second thought: the problem here was never (really) that it was
published *at maven central*. The problem was that it *was published*.
For purposes of discussion, I'll accept the stricture that there are
*no* circumstances under which it's legitimate to publish another
TLP's material under the usual package names.

OK, so the package names gotta change. For Xerces, that's
comparatively easy, given the way JAX-P works. In a maven-less
universe, I believe that the screwdriver of choice is jarjar. in a
maven-full universe it's shade.

Am I addressing the right set of issues? I don't want to go on to the
'and what do you do about the source' issues here if I'm on the wrong
subject altogether.

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:07 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>
>> How do we handle situations like
>> https://issues.apache.org/jira/browse/XERCESJ-1257 (open for 5 years,
>> still not incorporated in a release), without doing hackish things
>> (releasing someone elses code, sucking their entire codebase into
>> ours), or relying upon Uwe Schindler to come up with some clever
>> workaround?
>
> As far as I can tell, you have exactly the same issues solving this
> issue with and without Maven.
>

I don't think thats really the case at all. If i want to release a
patched jar with ivy its pretty trivial.

I simply make a project in github or google code with a build.xml that
downloads their code, patches it, and produces the patched result. its
still an open source project: its basically a fork. I put the binaries
i generate under Downloads and point ivy directly at that jar.

Done.

How to do this with maven?
--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:10 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 2:07 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>>
>>> How do we handle situations like
>>> https://issues.apache.org/jira/browse/XERCESJ-1257 (open for 5 years,
>>> still not incorporated in a release), without doing hackish things
>>> (releasing someone elses code, sucking their entire codebase into
>>> ours), or relying upon Uwe Schindler to come up with some clever
>>> workaround?
>>
>> As far as I can tell, you have exactly the same issues solving this
>> issue with and without Maven.
>>
>
> I don't think thats really the case at all. If i want to release a
> patched jar with ivy its pretty trivial.
>
> I simply make a project in github or google code with a build.xml that
> downloads their code, patches it, and produces the patched result. its
> still an open source project: its basically a fork. I put the binaries
> i generate under Downloads and point ivy directly at that jar.

Typoed. I'll try again.

Make a build.xml that patches their code, and also patches their
pom.xml to change the groupId. Then publish to central via OSSRH under
your group ID. If you're not doing something 'officially Apache', no
one has any right to complain.

If you want to be friendly to people with opinions about package
names, you use shade to change this package names, just as you would
use jarjar under ant.
>
> Done.
>
> How to do this with maven?
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:10 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 2:07 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>>
>>> How do we handle situations like
>>> https://issues.apache.org/jira/browse/XERCESJ-1257 (open for 5 years,
>>> still not incorporated in a release), without doing hackish things
>>> (releasing someone elses code, sucking their entire codebase into
>>> ours), or relying upon Uwe Schindler to come up with some clever
>>> workaround?
>>
>> As far as I can tell, you have exactly the same issues solving this
>> issue with and without Maven.
>>
>
> I don't think thats really the case at all. If i want to release a
> patched jar with ivy its pretty trivial.
>
> I simply make a project in github or google code with a build.xml that
> downloads their code, patches it, and produces the patched result. its
> still an open source project: its basically a fork. I put the binaries
> i generate under Downloads and point ivy directly at that jar.
>
> Done.
>
> How to do this with maven?

If it's a maven project, you do the same thing, patching the pom to
shade to change the packages if you want, or just changing the
coordinates.

You publish the results to central via OSSRH under your coordinates.
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:17 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:

> If it's a maven project, you do the same thing, patching the pom to
> shade to change the packages if you want, or just changing the
> coordinates.
>
> You publish the results to central via OSSRH under your coordinates.
>

I think this illustrates Mike's point exactly though? As PMC members
how do we know this was done appropriately?

I have no idea about shade, coordinates, any of that, I understand
search engines pretty well, and the lucene/solr codebase ok. How do
PMC members know that 'fake maven releases' as you describe are any
better than 'fake maven releases' like we had before that caused all
the controversy?
--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:20 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 2:17 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>
>> If it's a maven project, you do the same thing, patching the pom to
>> shade to change the packages if you want, or just changing the
>> coordinates.
>>
>> You publish the results to central via OSSRH under your coordinates.
>>
>
> I think this illustrates Mike's point exactly though? As PMC members
> how do we know this was done appropriately?

How do you know it's done 'appropriately' in your scenario? You could
have had just as many board members up your nose by using your
ant/github procedure and depositing the resulting bundle on Apache
'dist' as an 'auxiliary binary bundle'.
>
> I have no idea about shade, coordinates, any of that, I understand
> search engines pretty well, and the lucene/solr codebase ok. How do
> PMC members know that 'fake maven releases' as you describe are any
> better than 'fake maven releases' like we had before that caused all
> the controversy?

By knowing a few simple facts. The policy issues here derive from the
following questions:

1) Did the PMC publish something (either to repository.apache.org, or
www.apache.org)?
2) Was the something a formal Apache release (consisting of source) or
was it a 'convenience binary'?
3) If it was a convenience binary, does it annoy someone by including
code under some other project's package names?

If you don't want to risk being hassled, don't end up answering:

   1: yes 2: yes, and there's a binary in it anywhere.
   1: yes, 2: doesn't matter, 3: yes
>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:27 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
> How do you know it's done 'appropriately' in your scenario? You could
> have had just as many board members up your nose by using your
> ant/github procedure and depositing the resulting bundle on Apache
> 'dist' as an 'auxiliary binary bundle'.
>

How so? what would the board members do me?

whats wrong with what i did legally with
https://github.com/rmuir/jetty6-unicode (3.6 solr depends on that).

I forked another project (Welcome to github!) and put up downloads.
ivy downloads from there.

The licensing terms are legit, its not doing any thing sheisty with
apache infrastructure, there are no jars in our source tree.

its just an open source fork of an abandoned project (jetty 6) that i
picked up and patched for unicode bugs. Anyone can use the build.xml
there to recreate it totally from source code.

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:32 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 2:27 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>> How do you know it's done 'appropriately' in your scenario? You could
>> have had just as many board members up your nose by using your
>> ant/github procedure and depositing the resulting bundle on Apache
>> 'dist' as an 'auxiliary binary bundle'.
>>
>
> How so? what would the board members do me?
>
> whats wrong with what i did legally with
> https://github.com/rmuir/jetty6-unicode (3.6 solr depends on that).
>
> I forked another project (Welcome to github!) and put up downloads.
> ivy downloads from there.
>
> The licensing terms are legit, its not doing any thing sheisty with
> apache infrastructure, there are no jars in our source tree.
>
> its just an open source fork of an abandoned project (jetty 6) that i
> picked up and patched for unicode bugs. Anyone can use the build.xml
> there to recreate it totally from source code.

You didn't do this as an official act of an Apache PMC. However, if
you had done this to Xerces, you might have gotten some (I think
ill-informed) trademark hassles if you didn't change the package
names. However, I think that's a digression.

You posed the question, I think, "How do I, as a PMC member, evaluate
the appropriateness of something done with Maven as a technology or
delivered to Maven central as a maven package."

My answer is that you ask these questions of the committer who
committed (or the patcher who proposes (or the release manager who
makes a candidate)) the changes that you are evaluating:

1) Are you introducing any binary materials into our source releases?
2) Are you introducing any forked code from another Apache TLP into
our source releases of binary packages?
3) If the answer to (2) is "yes", you ask, "and have you changed the
package names".

So far, no mention of Maven.

And if you are uncomfortable with the answers, you recruit a
knowledgable fellow-PMC member (or wandering Foundation member) to
help out. This seems to me to be parallel with how you would evaluate
some obscure point of the impossibly complex rules for NOTICE files.

>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:40 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 2:32 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 23, 2012 at 2:27 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>> How do you know it's done 'appropriately' in your scenario? You could
>>> have had just as many board members up your nose by using your
>>> ant/github procedure and depositing the resulting bundle on Apache
>>> 'dist' as an 'auxiliary binary bundle'.
>>>
>>
>> How so? what would the board members do me?
>>
>> whats wrong with what i did legally with
>> https://github.com/rmuir/jetty6-unicode (3.6 solr depends on that).
>>
>> I forked another project (Welcome to github!) and put up downloads.
>> ivy downloads from there.
>>
>> The licensing terms are legit, its not doing any thing sheisty with
>> apache infrastructure, there are no jars in our source tree.
>>
>> its just an open source fork of an abandoned project (jetty 6) that i
>> picked up and patched for unicode bugs. Anyone can use the build.xml
>> there to recreate it totally from source code.
>
> You didn't do this as an official act of an Apache PMC. However, if
> you had done this to Xerces, you might have gotten some (I think
> ill-informed) trademark hassles if you didn't change the package
> names. However, I think that's a digression.

Maybe, thats ok. I could have also ignored these. Seeing how Apache
enforces its trademark with Lucene (e.g. Zend Lucene), I'm not worried
in the slightest. Bring the lawyers on: I'm not afraid.

The best part about it: I forked this project independently, just like
any old Joe Schmoe can fork any old github project and improve it. And
if the licensing is ok, we are free to depend on it, and our PMC
doesnt need to worry about trademarks for projects we arent
responsible for.
--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 2:43 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 2:40 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 23, 2012 at 2:32 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>>> On Mon, Apr 23, 2012 at 2:27 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>>> How do you know it's done 'appropriately' in your scenario? You could
>>>> have had just as many board members up your nose by using your
>>>> ant/github procedure and depositing the resulting bundle on Apache
>>>> 'dist' as an 'auxiliary binary bundle'.
>>>>
>>>
>>> How so? what would the board members do me?
>>>
>>> whats wrong with what i did legally with
>>> https://github.com/rmuir/jetty6-unicode (3.6 solr depends on that).
>>>
>>> I forked another project (Welcome to github!) and put up downloads.
>>> ivy downloads from there.
>>>
>>> The licensing terms are legit, its not doing any thing sheisty with
>>> apache infrastructure, there are no jars in our source tree.
>>>
>>> its just an open source fork of an abandoned project (jetty 6) that i
>>> picked up and patched for unicode bugs. Anyone can use the build.xml
>>> there to recreate it totally from source code.
>>
>> You didn't do this as an official act of an Apache PMC. However, if
>> you had done this to Xerces, you might have gotten some (I think
>> ill-informed) trademark hassles if you didn't change the package
>> names. However, I think that's a digression.
>
> Maybe, thats ok. I could have also ignored these. Seeing how Apache
> enforces its trademark with Lucene (e.g. Zend Lucene), I'm not worried
> in the slightest. Bring the lawyers on: I'm not afraid.
>
> The best part about it: I forked this project independently, just like
> any old Joe Schmoe can fork any old github project and improve it. And
> if the licensing is ok, we are free to depend on it, and our PMC
> doesnt need to worry about trademarks for projects we arent
> responsible for.

That's right. If you, Rob Muir, just so happen to set up a github
project with a fix to Xerces, and I, Benson Margulies, just happen to
help you publish it to Maven Central, then the PMC, without (much)
fear of hassle, can consume it as a binary dependency. But some folks
will want to be sure that we have drawn a bright line between this
process and the formal activities of the PMC.

>
>
> --
> lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 3:02 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>
> That's right. If you, Rob Muir, just so happen to set up a github
> project with a fix to Xerces, and I, Benson Margulies, just happen to
> help you publish it to Maven Central, then the PMC, without (much)
> fear of hassle, can consume it as a binary dependency. But some folks
> will want to be sure that we have drawn a bright line between this
> process and the formal activities of the PMC.
>

They may want that: but that doesn't mean they will get it.

Personally I'm pretty frustrated at the insinuation that anything we
did for 3.6 is questionable, we spent a shitload of time trying to
make sure there was no .jar files in source release, no fake maven
releases of other peoples code, no sketchy links to anything on apache
infra, or anything like that.

Seeing that I (and a bunch of other people) stayed up for many long
nighters to clean this crap up, I'm willing to stand up for it: so
like I said I have no concerns at all.

If anyone in the PMC is concerned with how the dependencies for 3.6
were managed in that release, then speak your mind. But nobody voted
against it so I can only hope that you have a minority opinion here.

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 3:09 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 3:02 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>
>> That's right. If you, Rob Muir, just so happen to set up a github
>> project with a fix to Xerces, and I, Benson Margulies, just happen to
>> help you publish it to Maven Central, then the PMC, without (much)
>> fear of hassle, can consume it as a binary dependency. But some folks
>> will want to be sure that we have drawn a bright line between this
>> process and the formal activities of the PMC.
>>
>
> They may want that: but that doesn't mean they will get it.
>
> Personally I'm pretty frustrated at the insinuation that anything we
> did for 3.6 is questionable, we spent a shitload of time trying to
> make sure there was no .jar files in source release, no fake maven
> releases of other peoples code, no sketchy links to anything on apache
> infra, or anything like that.
>
> Seeing that I (and a bunch of other people) stayed up for many long
> nighters to clean this crap up, I'm willing to stand up for it: so
> like I said I have no concerns at all.
>
> If anyone in the PMC is concerned with how the dependencies for 3.6
> were managed in that release, then speak your mind. But nobody voted
> against it so I can only hope that you have a minority opinion here.

Rob, did you mean 'Benson' by 'you' in that sentence? If so, I'm
really sorry if I've given you the impression I think that that
there's anything amiss with 3.6.0. All I meant to say was that, over
the long haul, if the PMC was very obviously using the dev list to
plan, manage, and execute many things like what you did with Jetty,
someone might criticize.

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 3:23 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
> Rob, did you mean 'Benson' by 'you' in that sentence? If so, I'm
> really sorry if I've given you the impression I think that that
> there's anything amiss with 3.6.0. All I meant to say was that, over
> the long haul, if the PMC was very obviously using the dev list to
> plan, manage, and execute many things like what you did with Jetty,
> someone might criticize.
>

Really ??? But they won't criticize the fact someone put
http://code.google.com/p/language-detection/ into maven central?

http://search.maven.org/#artifactdetails|com.cybozu.labs|langdetect|1.1-20120112|jar

"Borg planet" maven central is somehow "more ok", despite the fact
that my github fork (with its ant download + patch process) is clearly
more legally traceable?

What makes sonatype.com so special? Do they offer legal protection
above and beyond my github fork that I am unaware of?!

--
lucidimagination.com

---------------------------------------------------------------------

> Really ??? But they won't criticize the fact someone put
> http://code.google.com/p/language-detection/ into maven central?
> http://search.maven.org/#artifactdetails|com.cybozu.labs|langdetect|1.1-20120112|jar

I don't think even people who first accused Lucene of releasing
commons-csv artifacts can address your question here. And I think if
you published commons-csv on github they'd be equally upset.

Dawid

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 3:53 PM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>> Really ??? But they won't criticize the fact someone put
>> http://code.google.com/p/language-detection/ into maven central?
>> http://search.maven.org/#artifactdetails|com.cybozu.labs|langdetect|1.1-20120112|jar
>
> I don't think even people who first accused Lucene of releasing
> commons-csv artifacts can address your question here. And I think if
> you published commons-csv on github they'd be equally upset.
>

Its also a good point that what i do on my github account, the apache
board has no control over (despite how powerful they might think they
are), nor does the lucene pmc have any responsibility over.

the lucene PMC just ensure we depend upon projects with compatible licenses.

--
lucidimagination.com

---------------------------------------------------------------------

> Its also a good point that what i do on my github account, the apache
> board has no control over (despite how powerful they might think they

To my best knowledge this is the case, yes. People may not like the
fact that you re-publish the same code under the same packages but
there's really no rule that forbids it (and why would there be any)?

> the lucene PMC just ensure we depend upon projects with compatible licenses.

Which doesn't mean we need to be disrespectful to other projects, no
matter where they originate from. And I think we're not, the quick
response to commons-csv issue is the best example of this.

Dawid

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 4:32 PM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>> Its also a good point that what i do on my github account, the apache
>> board has no control over (despite how powerful they might think they
>
> To my best knowledge this is the case, yes. People may not like the
> fact that you re-publish the same code under the same packages but
> there's really no rule that forbids it (and why would there be any)?
>
>> the lucene PMC just ensure we depend upon projects with compatible licenses.
>
> Which doesn't mean we need to be disrespectful to other projects, no
> matter where they originate from. And I think we're not, the quick
> response to commons-csv issue is the best example of this.
>

But what i do in my github account is totally personal, its open source.

If these projects don't want their code forked... dont offer it up on
github under the Apache 2.0 license?

... this close to forking every single apache project on his github
just to prove the point.

--
lucidimagination.com

---------------------------------------------------------------------

> If these projects don't want their code forked... dont offer it up on
> github under the Apache 2.0 license?

I wouldn't be so sure that github's idea is to create endless forks,
each ending up in a separate distribution :) I think it's to
facilitate pulling changes back to the origin?

Dawid

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 4:54 PM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
>
> I wouldn't be so sure that github's idea is to create endless forks,
> each ending up in a separate distribution :) I think it's to
> facilitate pulling changes back to the origin?
>

Not so sure, the idea is that forking is part of the design, so if
some project doesnt pull the change for, i don't know, like five
years, maybe your fork becomes the more popular version :)

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 3:53 PM, Dawid Weiss
<[EMAIL PROTECTED]> wrote:
> I don't think even people who first accused Lucene of releasing
> commons-csv artifacts can address your question here. And I think if
> you published commons-csv on github they'd be equally upset.
>

Ok: so we just use https://github.com/Gasol/commons-csv instead :)

 (picking random github fork of commons-csv)

--
lucidimagination.com

---------------------------------------------------------------------

On Mon, Apr 23, 2012 at 11:17 AM, Benson Margulies
<[EMAIL PROTECTED]> wrote:
> On Mon, Apr 23, 2012 at 2:10 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
>> On Mon, Apr 23, 2012 at 2:07 PM, Benson Margulies <[EMAIL PROTECTED]> wrote:
>>>>
>>>> How do we handle situations like
>>>> https://issues.apache.org/jira/browse/XERCESJ-1257 (open for 5 years,
>>>> still not incorporated in a release), without doing hackish things
>>>> (releasing someone elses code, sucking their entire codebase into
>>>> ours), or relying upon Uwe Schindler to come up with some clever
>>>> workaround?
>>>
>>> As far as I can tell, you have exactly the same issues solving this
>>> issue with and without Maven.
>>>
>>
>> I don't think thats really the case at all. If i want to release a
>> patched jar with ivy its pretty trivial.
>>
>> I simply make a project in github or google code with a build.xml that
>> downloads their code, patches it, and produces the patched result. its
>> still an open source project: its basically a fork. I put the binaries
>> i generate under Downloads and point ivy directly at that jar.
>>
>> Done.
>>
>> How to do this with maven?
>
> If it's a maven project, you do the same thing, patching the pom to
> shade to change the packages if you want, or just changing the
> coordinates.
>
> You publish the results to central via OSSRH under your coordinates.
>

but note that in either case if you use the same package/namespace you
are likely to upset the people who control that package/namespace

ryan

---------------------------------------------------------------------

On Apr 23, 2012, at 1:00 PM, Benson Margulies wrote:

> when I see ' viral' in one of these debates, I get frustrated. please
> state an actual practical problem, not a scare word.

It's not really a scare word - its just the reason I have been against Maven as part of our core project - I mean viral in how it has infiltrated beyond what was original discussed.

I'd go into all of the reasons why I think Maven is what it is to the Lucene/Solr projects, but I have in past discussions and I'm not looking to repeat past discussions. This debate is not about coloring anyones view about Maven. All of the committers and PMC members already have their views and won't be cowed by 'scare' words.

This is essentially about finding out where the committers/PMC members stand - you can see where I do ;)

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------

ok, sorry, I read viral as the classic anti-maven slam. sorry for the noise.

On Apr 23, 2012, at 1:14 PM, Mark Miller <[EMAIL PROTECTED]> wrote:

>
> On Apr 23, 2012, at 1:00 PM, Benson Margulies wrote:
>
>> when I see ' viral' in one of these debates, I get frustrated. please
>> state an actual practical problem, not a scare word.
>
> It's not really a scare word - its just the reason I have been against Maven as part of our core project - I mean viral in how it has infiltrated beyond what was original discussed.
>
> I'd go into all of the reasons why I think Maven is what it is to the Lucene/Solr projects, but I have in past discussions and I'm not looking to repeat past discussions. This debate is not about coloring anyones view about Maven. All of the committers and PMC members already have their views and won't be cowed by 'scare' words.
>
> This is essentially about finding out where the committers/PMC members stand - you can see where I do ;)
>
> - Mark Miller
> lucidimagination.com
>
>
>
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>

---------------------------------------------------------------------

On Apr 23, 2012, at 10:14 AM, Michael McCandless wrote:

> I think, to fix this, we (the Apache Lucene PMC) should stop
> officially posting artifacts to Maven ourselves.  I think it's great
> if Steve (and/or others) continue to do so, just outside of the Apache
> Lucene project and outside of the Lucene PMC.  This would mean the
> Maven build/deploy is fully downstream from our official releases,
> just like how the numerous other package managers/repositories (yum,
> apt, pkg, etc.) distribute our release artifacts.

+1 - Love Steve, love his work - it really is fantastic - he's a wizard when it comes to this stuff.

I have always been a "Maven belongs downstream" supporter though - and I still believe that is the way to go. If the majority of committers don't agree, that's okay - but if the majority of us are not interested in the viral nature of Maven, I think we should change things.

We predicted when this first came up that Maven would have a much larger affect on us then was predicted by some. That is why I thought we should vote at the time to support it internally or not. I think now that we can observe many of the ramifications of our previous choice, it's easy to see that Maven affects a lot. It's a burden on committers, it's a burden on the PMC. Thank goodness Steve has been around to help mitigate it, but its still affected us and he may not be around forever. That's my vote (no surprise).

- Mark Miller
lucidimagination.com


---------------------------------------------------------------------

Le 24 avr. 2012 à 03:10, Benson Margulies a écrit :

> On Mon, Apr 23, 2012 at 8:42 PM, Chris Male <[EMAIL PROTECTED]> wrote:
>> I'm with Steve here, -1.  I think the arguments have already been well
>> articulated and the issue seems to be about dependency management in general
>> not just about Maven.  However I also absolutely agree with Robert that we
>> need to make this easier somehow.
>>
>> One thing I thought, is there any way we can get our own repo? In
>> organisations that use Maven for their projects, that's how they tend to
>> handle this situation.  They have their own repo for dependencies they need
>> to manage and they use repo1 for the rest.   As an ASF project, do we get
>> any help in this regards?
>
> Chris,
>
> I'm getting a bit tired of reading my own writing here, and I bet a
> lot of other people are, too, so I'll try to answer your question
> concisely and then disappear from this for at least 12 hours.
>
> The rule we are dealing with goes as follows, stated pseudo-biblically:
>
> * No PMC shall publish a JAR file containing modified versions of some
> other PMC's output unless you change the package names.

Does this apply to any binary release in apache's dist, any tgz ? Or is it just for maven ?

Nicolas
---------------------------------------------------------------------

On Tue, Apr 24, 2012 at 9:26 AM, Nicolas Lalevée
<[EMAIL PROTECTED]> wrote:

>> The rule we are dealing with goes as follows, stated pseudo-biblically:
>>
>> * No PMC shall publish a JAR file containing modified versions of some
>> other PMC's output unless you change the package names.
>
> Does this apply to any binary release in apache's dist, any tgz ? Or is it just for maven ?
>
> Nicolas
>

In our case, the problem was only maven. Because otherwise in the .tgz
no commons-csv jar file exists, except INSIDE the solr.war in the
binary release (along with a few other apache .jar's that actually
make that .war actually work).

Putting someone's jar inside a lib/ folder or war as a dependency is
hardly 'publishing'.

Its maven that requires 'publishing' and that causes all the fuss.

I've tried to explain this over and over again, but maven supporters
just get defensive and claim its not a maven problem: but it is!
--
lucidimagination.com

---------------------------------------------------------------------

>
> Putting someone's jar inside a lib/ folder or war as a dependency is
> hardly 'publishing'.
>
> Its maven that requires 'publishing' and that causes all the fuss.
>
> I've tried to explain this over and over again, but maven supporters
> just get defensive and claim its not a maven problem: but it is!

>
> I've tried to explain this over and over again, but maven supporters
> just get defensive and claim its not a maven problem: but it is!

Rob, I'm sorry about my density, but it took quite a long time to for
focus on this specific contrast: whether or not publication is
required to get a patched binary into a binary release (as opposed to
making it the default thing-in-the-classpath of people who declare
some dependency).

Maven does not, in fact, require publication in this case. An
ephemeral jar can travel into a war or an assembly without ever being
published.

Maven is guilty as charged that the simple, obvious, way of dealing
with this requires publication. In the pressurized circumstances of
3.6.0, no one was going to get past that. Now, we can if you want.

---------------------------------------------------------------------

On Tue, Apr 24, 2012 at 9:57 AM, Benson Margulies <[EMAIL PROTECTED]> wrote:

> Maven does not, in fact, require publication in this case. An
> ephemeral jar can travel into a war or an assembly without ever being
> published.
>

OK, I opened https://issues.apache.org/jira/browse/SOLR-3405: maven
artifacts should match the binary packaging.

Since our binary package doesn't expose all these jars (just ships a
.war), I think the maven artifacts should look the same way.

This means maven artifacts are consistent with our binary releases and
don't expose additional surface area (inner details of war
dependencies).

If things were done like this originally, I think no one would have
gotten upset.

--
lucidimagination.com

---------------------------------------------------------------------

On Tue, Apr 24, 2012 at 9:57 AM, Benson Margulies <[EMAIL PROTECTED]> wrote:

> Maven is guilty as charged that the simple, obvious, way of dealing
> with this requires publication. In the pressurized circumstances of
> 3.6.0, no one was going to get past that. Now, we can if you want.
>

well FWIW: we got past it for 3.6.0 by importing their source code and
renaming the packages to o.a.s.internal.xxxxx

Thats basically my question about this bugfix scenario, is this our
only real option for maven (suck in all the source code and repackage
it)?

I would like to have other options!

--
lucidimagination.com

---------------------------------------------------------------------

Responding to the thread as a whole, having read it with great interest.

I'd be interested to know what packagers for distributions such as
debian and fedora do with systems that patch 3rd party dependencies.

I'll guess that if it is internalized as mentioned below that there is
no problem.

But I have no idea what would be the case if one were to say that
such-and-so part of Lucene did not work with a 3rd party external
dependency (say icu) unless it were patched in a particular way.

BTW, fedora 16 has lucene 2.9.4 and does not have solr. Looks like F17
will have the same. I'd also be interested to know why they don't have
3.x or solr.

-- DM

On 04/24/2012 10:02 AM, Robert Muir wrote:
> On Tue, Apr 24, 2012 at 9:57 AM, Benson Margulies<[EMAIL PROTECTED]>  wrote:
>
>> Maven is guilty as charged that the simple, obvious, way of dealing
>> with this requires publication. In the pressurized circumstances of
>> 3.6.0, no one was going to get past that. Now, we can if you want.
>>
> well FWIW: we got past it for 3.6.0 by importing their source code and
> renaming the packages to o.a.s.internal.xxxxx
>
> Thats basically my question about this bugfix scenario, is this our
> only real option for maven (suck in all the source code and repackage
> it)?
>
> I would like to have other options!
>
---------------------------------------------------------------------

Hi,

> BTW, fedora 16 has lucene 2.9.4 and does not have solr. Looks like F17 will
> have the same. I'd also be interested to know why they don't have 3.x or solr.

I think that unrelated here, but the answer is simple: Those distributions don't have Lucene JARS to provide "Lucene" as a library to the user, but they provide Lucene as dependency for other packages . And those software is often very outdated and Lucene 3.0 is no longer backwards compatible. Linux distribs will adda "Lucene3" package once it is needed as a dependency.

Uwe
---------------------------------------------------------------------

On Tue, Apr 24, 2012 at 10:45 AM, Uwe Schindler <[EMAIL PROTECTED]> wrote:
> Hi,
>
>> BTW, fedora 16 has lucene 2.9.4 and does not have solr. Looks like F17 will
>> have the same. I'd also be interested to know why they don't have 3.x or solr.
>
> I think that unrelated here, but the answer is simple: Those distributions don't have Lucene JARS to provide "Lucene" as a library to the user, but they provide Lucene as dependency for other packages . And those software is often very outdated and Lucene 3.0 is no longer backwards compatible. Linux distribs will adda "Lucene3" package once it is needed as a dependency.
>

or their packaging system is just out of date.

for example, in freebsd both are at 3.5 (so much more current)

http://www.freebsd.org/cgi/cvsweb.cgi/ports/textproc/lucene/
http://www.freebsd.org/cgi/cvsweb.cgi/ports/textproc/apache-solr/

--
lucidimagination.com

---------------------------------------------------------------------

This thread is impossible for me to keep up with!  Some responses:

> In fact, I expect if there is real demand, a downstream contributor
> would easily pick this up.

Exactly: just like the numerous useful Linux repositories... it's
clearly not this PMC's job to push to all of those repositories.  Why
should it be our job to push to this one (Maven central)?

I completely agree many users legitimately find the Maven artifacts
useful, and I fully expect if we (the PMC) stop officially pushing
artifacts to Maven as part of our officially released artifacts that
someone else will step up to do so, just like other people step up to
publish our artifacts in the popular Linux repositories.

> Do you, Mike, fully understand all of Lucene/Solr's code?

No.

> If the answer is no, then how can you be comfortable voting for a
> release?

Because I feel there is "enough" overall coverage on the PMC, across
all of us, for all of Lucene/Solr's functionality.

I don't feel the same about the artifacts we push to Maven, the
"policies" of Maven central, nor the build process that produces those
artifacts.  Benson (and others) getting involved will certainly help
with that over time...

Steve, I realize you work wonderful magic with Maven, but it's still
magic and it entails clear risks / distractions to the Lucene project.

The recent board brouhaha was just one example of "what can go wrong
when a PMC votes on/publishes artifacts into something it doesn't
understand" (though: I do agree, there were additional "factors" at
play, as Benson described...).

I do agree the response for the followon (3.6.0) release was swift and
amazing (big thank you to Robert & others!): we've either masqueraded
(commons-csv), worked around bugs (XERCESJ-1257), forked & pulled
patched versions from github (Tomcat) or "released anew" via github
(noggit) so that our deps are now clean/legal.

So, yes, we've fixed the current "known risk" on publishing Maven
artifacts... but the risk and distraction nevertheless remains, in my
opinion.  We should be spending our time building a great search
engine/app, not on distractions like this.

That said, unless others speak up, it does look like there's no point
calling a vote here (I appear to have the minority opinion here), so
we will likely continue officially releasing into Maven for now (and I
can accept that).

> I don't know how maven is of relevance here.

I disagree.

As Robert described, anyone may fork commons-csv, Xerces, Tomcat, etc,
on github, patch them with their critical bug fixes and send pull
requests to get those fixes back upstream (eventually, hopefully).

This is the beauty of git/github.  It's fully decentralized: fork
away.

Your fork is "out there" on the wild wild internet, available for
anyone to use, or, not.  And, no, you don't have to change the package
names: the point of the fork is clear (eg "Xerces 2.9.1 with Unicode
surrogates bug fixed", "Tomcat with unicode bugs fixed").

It could be over time your fork becomes popular.  Anyone parsing
Wikipedia's XML exports will need that patch on XERCESJ-1257.  Maybe
you patch a few other unicode releated issues over time, etc.  This
can then eventually be useful information for the origin
project... it's like voting with your feet (hmm I guess fingertips).

The Maven repository, in contrast, is centralized, has certain
"rules", and so the mere act of putting your dependency in there
without masquerading the package names is (reasonably?) seen as a
stealing someone else's namespace.

I think hiding/masquerading dependencies (shade/jarjar/etc.) is a bad
hack (and I'd love to stop doing this for Solr's commons-csv dep).

> I think more important than our code being correct is that our
> distributions work correctly.

+1

> And as a side note: Uwe's ideas bring up a nice potential compromise
> for simplification: maybe just lucene (as an API) should be in maven
> and not solr (as an application?). Its worth thinking about: solr has
> many many more third-party dependencies.  Dependencies are really

+1
OSGi does seem promising on paper... but I don't understand it (and
the number of patches/iterations on the issue is spooky).  It
(LUCENE-1344) is currently the top-voted Jira issue for Lucene...

But I'm not sure it really helps (yet)?  Not everyone/enough of our
users consumes Lucene/Solr via OSGi...

Mike McCandless

http://blog.mikemccandless.com

On Tue, Apr 24, 2012 at 7:12 AM, Michael McCandless
<[EMAIL PROTECTED]> wrote:
> This thread is impossible for me to keep up with!

It's a very complex topic, and I wish I could find more concise ways
to talk about it.

Anyhow, since you all seem to have a consensus not to push the rewind
button back to the days of 'just the source', I'll leave this offer
hanging in the air: I'd like to help make lives easier around here. I
can't give Rob and Steve back any of the time they spent exploring all
the twisty passages in 3.6.0, but there may be other possibilities
looking toward 4.0. After folks have recovered from this deluge, if
someone wants to post threads of the form: "Is there an alternative to
giant annoyance X" we can see what can be come up with.

---------------------------------------------------------------------