Hi,

when looking at just a very quick glance at some of the newer
grouping/join features, I found myself a little confused about what is
exactly what, and I think users might too.

I discussed some of this with hossman, and it only seemed to make me
even more totally confused about:
* difference between field collapsing and grouping
* difference between nested documents and the index-time join
* difference between index-time-join/nested documents and single-pass
index-time grouping. Is the former only a more general case of the
latter?
* difference between the above joinish capabilities and solr's join
impl... other than the single-pass/index-time limitation (which is
really an implementation detail), I'm talking about use cases.

I think its especially interesting since the join module depends on
the grouping module.

So I am curious if we should:
* add docs (maybe with simple examples) in the package.html or
otherwise that differentiate what these guys are, or at least agree on
some consistent terminology and define it somewhere? I feel like
people have explained to me the differences in all these things
before, but then its easy to forget.
* should we rename the join module to nested? or combine it with
grouping as a subdocument module? or something else?

---------------------------------------------------------------------

Hi,

On Fri, Jul 1, 2011 at 2:30 PM, Robert Muir <[EMAIL PROTECTED]> wrote:

> Hi,
>
> when looking at just a very quick glance at some of the newer
> grouping/join features, I found myself a little confused about what is
> exactly what, and I think users might too.
>
> I discussed some of this with hossman, and it only seemed to make me
> even more totally confused about:
> * difference between field collapsing and grouping
> * difference between nested documents and the index-time join
> * difference between index-time-join/nested documents and single-pass
> index-time grouping. Is the former only a more general case of the
> latter?
> * difference between the above joinish capabilities and solr's join
> impl... other than the single-pass/index-time limitation (which is
> really an implementation detail), I'm talking about use cases.
>
> I think its especially interesting since the join module depends on
> the grouping module.
>
> So I am curious if we should:
> * add docs (maybe with simple examples) in the package.html or
> otherwise that differentiate what these guys are, or at least agree on
> some consistent terminology and define it somewhere? I feel like
> people have explained to me the differences in all these things
> before, but then its easy to forget.
> * should we rename the join module to nested? or combine it with
> grouping as a subdocument module? or something else?
>

What about, dear I say, a document-relation module? Joining documents across
queries, in docblocks, into a group, or as part of a nest, all seem to be
about relationships between documents.

With all of these concepts in the single module, we can then definitely work
on the package.htmls to make it clear the purpose of each part (and maybe
list other known phrases for the same concepts, such as field collapsing and
grouping).
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
--
Chris Male | Software Developer | JTeam BV.| www.jteam.nl

I think joining and grouping are two different functions, and we
should keep different modules for them...

On Thu, Jun 30, 2011 at 10:30 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> Hi,
>
> when looking at just a very quick glance at some of the newer
> grouping/join features, I found myself a little confused about what is
> exactly what, and I think users might too.

They are confusing!

> I discussed some of this with hossman, and it only seemed to make me
> even more totally confused about:
> * difference between field collapsing and grouping

I like the name grouping better here: I think field collapsing
undersells (it's only one specific way to use grouping).  EG, grouping
w/o collapsing is useful (eg, Best Buy grouping hits by product
category and showing the top 5 in each).

> * difference between nested documents and the index-time join

Similarly I think nested docs undersells index-time join: you can
join (either during indexing or during searching) in many different
ways, and nested docs is just one use case.

EG, maybe your docs are doctors but during indexing you join to a city
table with facts about that city (each doctor's office is in a
specific city) and then you want to run queries like "city's avg
annual temp > 60 and doctor has good bedside manner" or something.

> * difference between index-time-join/nested documents and single-pass
> index-time grouping. Is the former only a more general case of the
> latter?

Grouping is purely a presentation concern -- you are not altering
which docs hit; you are simply changing how you pick which hits to
display ("top N by group").  So we only have collectors here.

The "generic" (requires 2 passes) collectors can group on anything at
search time; the "doc block" collector requires that you indexed all
docs in each group as a block.

Join is both about restricting matches and also presentation of hits,
because your query needs to match fields from different [logical]
tables (so, the module has a Query and a Collector).  When you get the
results back, you may or may not be interested in retaining the table
structure in your result set (ie, you may not have selected fields
from the child table).

Similarly, "generic" joining (in Solr/ElasticSearch today but I'd like
to factor into the join module) can do any join at search time, while
the "doc block" collector requires that you did the necessary join(s)
during indexing.

> * difference between the above joinish capabilities and solr's join
> impl... other than the single-pass/index-time limitation (which is
> really an implementation detail), I'm talking about use cases.

Solr's/ElasticSearch's join is more general because you can join
anything at search time (even, across 2 different indexes), vs doc
block join where you must pick which joins you will ever want to use
and then build the index accordingly.

You can also mix the two.  Maybe you do certain joins while indexing,
but then at search time you do other joins "generically".  That's
fine.  (Same is true for grouping).

> I think its especially interesting since the join module depends on
> the grouping module.

The join module does currently depend on the grouping module, but for
a silly reason: just for the TopGroups, to represent the returned
hits.  We could move TopGroups/GroupDocs into common (thus justifying
its generic name!)?  Then both join and grouping modules depend on
common.

Really TopGroups is just a TopDocs that allows some recursion (ie,
each hit may in turn be another TopDocs).  But TopGroups is limited
now to only depth 2 recursion... we need to fix this for nested
grouping.  Really we just need a recursive TopDocs here....

> So I am curious if we should:
> * add docs (maybe with simple examples) in the package.html or
> otherwise that differentiate what these guys are, or at least agree on
> some consistent terminology and define it somewhere? I feel like
> people have explained to me the differences in all these things
> before, but then its easy to forget.

Well, each module's package.html has a start here, but I agree we
should do more.

I think what would be best is a smallish but feature complete demo, ie
pull together some easy-to-understand sample content and the build a
small demo app around it.  We could then show how to use grouping for
field collapsing (and for other use cases), joining for nested docs
(and for other use cases), etc.

Mike McCandless

http://blog.mikemccandless.com

>> I think what would be best is a smallish but feature complete demo,

For the nested stuff I had a reasonable demo on LUCENE-2454 that was based
around resumes - that use case has the one-to-many characteristics that lends
itself to nested e.g. a person has many different qualifications and records of
employment.
This scenario was illustrated
here: http://www.slideshare.net/MarkHarwood/proposal-for-nested-document-support-in-lucene

I also had the "book search" type scenario where a book has many sections and
for the purposes of efficient highlighting/summarisation  these sections were
treated as child docs which could be read quickly (rather than highlighting a
whole book)

I'm not sure what the "parent" was in your doctor and cities example, Mike. If a
doctor is in only one city then there is no point making city a child doc as the
one city info can happily be combined with the doctor info into a single
document with no conflict (doctors have different properties to cities).
If the city is the parent with many child doctor docs that makes more sense but
feels like a less likely use case e.g. "find me a city with doctor x and a
different doctor y"
Searching for a person with excellent java and prefrerably good lucene skills
feels like a more real-world example.

It feels like documenting some of the trade-offs behind index design choices is
useful too e.g. nesting is not too great for very volatile content with
constantly changing children while search-time join is more costly in RAM and
2-pass processing

Cheers
Mark

----- Original Message ----
From: Michael McCandless <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Fri, 1 July, 2011 13:51:04
Subject: Re: revisit naming for grouping/join?

I think joining and grouping are two different functions, and we
should keep different modules for them...

On Thu, Jun 30, 2011 at 10:30 PM, Robert Muir <[EMAIL PROTECTED]> wrote:
> Hi,
>
> when looking at just a very quick glance at some of the newer
> grouping/join features, I found myself a little confused about what is
> exactly what, and I think users might too.

They are confusing!

> I discussed some of this with hossman, and it only seemed to make me
> even more totally confused about:
> * difference between field collapsing and grouping

I like the name grouping better here: I think field collapsing
undersells (it's only one specific way to use grouping).  EG, grouping
w/o collapsing is useful (eg, Best Buy grouping hits by product
category and showing the top 5 in each).

> * difference between nested documents and the index-time join

Similarly I think nested docs undersells index-time join: you can
join (either during indexing or during searching) in many different
ways, and nested docs is just one use case.

EG, maybe your docs are doctors but during indexing you join to a city
table with facts about that city (each doctor's office is in a
specific city) and then you want to run queries like "city's avg
annual temp > 60 and doctor has good bedside manner" or something.

> * difference between index-time-join/nested documents and single-pass
> index-time grouping. Is the former only a more general case of the
> latter?

Grouping is purely a presentation concern -- you are not altering
which docs hit; you are simply changing how you pick which hits to
display ("top N by group").  So we only have collectors here.

The "generic" (requires 2 passes) collectors can group on anything at
search time; the "doc block" collector requires that you indexed all
docs in each group as a block.

Join is both about restricting matches and also presentation of hits,
because your query needs to match fields from different [logical]
tables (so, the module has a Query and a Collector).  When you get the
results back, you may or may not be interested in retaining the table
structure in your result set (ie, you may not have selected fields
from the child table).

Similarly, "generic" joining (in Solr/ElasticSearch today but I'd like
to factor into the join module) can do any join at search time, while
the "doc block" collector requires that you did the necessary join(s)
during indexing.
Solr's/ElasticSearch's join is more general because you can join
anything at search time (even, across 2 different indexes), vs doc
block join where you must pick which joins you will ever want to use
and then build the index accordingly.

You can also mix the two.  Maybe you do certain joins while indexing,
but then at search time you do other joins "generically".  That's
fine.  (Same is true for grouping).
The join module does currently depend on the grouping module, but for
a silly reason: just for the TopGroups, to represent the returned
hits.  We could move TopGroups/GroupDocs into common (thus justifying
its generic name!)?  Then both join and grouping modules depend on
common.

Really TopGroups is just a TopDocs that allows some recursion (ie,
each hit may in turn be another TopDocs).  But TopGroups is limited
now to only depth 2 recursion... we need to fix this for nested
grouping.  Really we just need a recursive TopDocs here....
Well, each module's package.html has a start here, but I agree we
should do more.

I think what would be best is a smallish but feature complete demo, ie
pull together some easy-to-understand sample content and the build a
small demo app around it.  We could then show how to use grouping for
field collapsing (and for other use cases), joining for nested docs
(and for other use cases), etc.

Mike McCandless

http://blog.mikemccandless.com

To unsubscribe, e-mail: [EMAIL PROTECTED]
[EMAIL PROTECTED]

On Fri, Jul 1, 2011 at 8:51 AM, Michael McCandless
<[EMAIL PROTECTED]> wrote:

> The join module does currently depend on the grouping module, but for
> a silly reason: just for the TopGroups, to represent the returned
> hits.  We could move TopGroups/GroupDocs into common (thus justifying
> its generic name!)?  Then both join and grouping modules depend on
> common.

Just a suggestion: maybe they belong in the lucene core? And maybe the
stuff in the common module belongs in lucene core's util package?

I guess I'm suggesting we try to keep our modules as flat as possible,
with as little dependencies as possible. I think we really already
have a 'common' module, thats the lucene core. If multiple modules end
up relying upon the same functionality, especially if its something
simple like an abstract class (Analyzer) or a utility thing (these
mutable integers, etc), then thats a good sign it belongs in core
apis.

I think we really need to try to nuke all these dependencies between
modules: its great to add them as a way to get refactoring started,
but ultimately we should try to clean up: because we don't want a
complex 'graph' of dependencies but instead something dead-simple. I
made a total mess with the analyzers module at first, i think
everything depended on it! but now we have nuked almost all
dependencies on this thing, except for where it makes sense to have
that concrete dependency (benchmark, demo, solr).

>
> I think what would be best is a smallish but feature complete demo, ie
> pull together some easy-to-understand sample content and the build a
> small demo app around it.  We could then show how to use grouping for
> field collapsing (and for other use cases), joining for nested docs
> (and for other use cases), etc.
>

For the same reason listed above, I think we should take our
contrib/demo and consolidate 'examples' across various places into
this demo module. The reason is:
* examples typically depend upon 'concrete' stuff, but in general core
stuff should work around interfaces/abstract classes: e.g. the
faceting module has an analyzers dependency only because of its
examples.
* examples might want to integrate modules, e.g. an example of how to
integrate faceting and grouping or something like that.
* examples are important: i think if the same question comes up on the
user list often, we should consider adding an example.

---------------------------------------------------------------------

On 7/1/2011 at 10:02 AM, Robert Muir wrote:
> [...] I think we should take our contrib/demo and consolidate 'examples'
> across various places into this demo module. The reason is:
>
> * examples typically depend upon 'concrete' stuff, but in general core
>   stuff should work around interfaces/abstract classes: e.g. the faceting
>   module has an analyzers dependency only because of its examples.
>
> * examples might want to integrate modules, e.g. an example of how to
>   integrate faceting and grouping or something like that.
>
> * examples are important: i think if the same question comes up on the
>   user list often, we should consider adding an example.

+1

On Fri, Jul 1, 2011 at 10:02 AM, Robert Muir <[EMAIL PROTECTED]> wrote:
> On Fri, Jul 1, 2011 at 8:51 AM, Michael McCandless
> <[EMAIL PROTECTED]> wrote:
>
>> The join module does currently depend on the grouping module, but for
>> a silly reason: just for the TopGroups, to represent the returned
>> hits.  We could move TopGroups/GroupDocs into common (thus justifying
>> its generic name!)?  Then both join and grouping modules depend on
>> common.
>
> Just a suggestion: maybe they belong in the lucene core? And maybe the
> stuff in the common module belongs in lucene core's util package?

+1

If we can generalize TopDocs so that we parameterize the type of each
hit, ie it could be a leaf (single doc + score (ScoreDoc) + maybe
field values (FieldDoc)) or another TopDocs, then we don't need the
separate TopGroups anymore.

> I guess I'm suggesting we try to keep our modules as flat as possible,
> with as little dependencies as possible. I think we really already
> have a 'common' module, thats the lucene core. If multiple modules end
> up relying upon the same functionality, especially if its something
> simple like an abstract class (Analyzer) or a utility thing (these
> mutable integers, etc), then thats a good sign it belongs in core
> apis.

I like this approach.

> I think we really need to try to nuke all these dependencies between
> modules: its great to add them as a way to get refactoring started,
> but ultimately we should try to clean up: because we don't want a
> complex 'graph' of dependencies but instead something dead-simple. I
> made a total mess with the analyzers module at first, i think
> everything depended on it! but now we have nuked almost all
> dependencies on this thing, except for where it makes sense to have
> that concrete dependency (benchmark, demo, solr).

Good!

>> I think what would be best is a smallish but feature complete demo, ie
>> pull together some easy-to-understand sample content and the build a
>> small demo app around it.  We could then show how to use grouping for
>> field collapsing (and for other use cases), joining for nested docs
>> (and for other use cases), etc.
>>
>
> For the same reason listed above, I think we should take our
> contrib/demo and consolidate 'examples' across various places into
> this demo module. The reason is:
> * examples typically depend upon 'concrete' stuff, but in general core
> stuff should work around interfaces/abstract classes: e.g. the
> faceting module has an analyzers dependency only because of its
> examples.
> * examples might want to integrate modules, e.g. an example of how to
> integrate faceting and grouping or something like that.
> * examples are important: i think if the same question comes up on the
> user list often, we should consider adding an example.

+1

I think especially now that we have very new "interesting" modules
(facet, join, grouping), we really need corresponding examples to
showcase all of this.

Mike

---------------------------------------------------------------------

On Fri, Jul 1, 2011 at 9:28 AM, mark harwood <[EMAIL PROTECTED]> wrote:
>>> I think what would be best is a smallish but feature complete demo,
>
> For the nested stuff I had a reasonable demo on LUCENE-2454 that was based
> around resumes - that use case has the one-to-many characteristics that lends
> itself to nested e.g. a person has many different qualifications and records of
> employment.
> This scenario was illustrated
> here: http://www.slideshare.net/MarkHarwood/proposal-for-nested-document-support-in-lucene
>
> I also had the "book search" type scenario where a book has many sections and
> for the purposes of efficient highlighting/summarisation  these sections were
> treated as child docs which could be read quickly (rather than highlighting a
> whole book)

I think both resumes and book search, and also others like the
variants of a product SKU, would all make good examples for the nested
docs use case.

> I'm not sure what the "parent" was in your doctor and cities example, Mike. If a
> doctor is in only one city then there is no point making city a child doc as the
> one city info can happily be combined with the doctor info into a single
> document with no conflict (doctors have different properties to cities).
> If the city is the parent with many child doctor docs that makes more sense but
> feels like a less likely use case e.g. "find me a city with doctor x and a
> different doctor y"
> Searching for a person with excellent java and prefrerably good lucene skills
> feels like a more real-world example.

In my example the city was parent -- I raised this example to explain
that index-time joining is more general than just nested docs (ie, I
think we should keep the name "join" for this module... also because
we should factor out more general search-time-only join capabilities
into it).

> It feels like documenting some of the trade-offs behind index design choices is
> useful too e.g. nesting is not too great for very volatile content with
> constantly changing children while search-time join is more costly in RAM and
> 2-pass processing

+1, especially once we've factored out generic joins.

Mike

---------------------------------------------------------------------

: In my example the city was parent -- I raised this example to explain
: that index-time joining is more general than just nested docs (ie, I
: think we should keep the name "join" for this module... also because
: we should factor out more general search-time-only join capabilities
: into it).

i think that may be the wrong approach to take when discussing "examples",
while it's great to say there are dozens of usecases that these features
can all support in dozens of diff ways" we should relaly focus on
naming/deming these use cases in the ways where they really make the most
sense.

In otherwords, i don't think we should say "All of these types of problems
are different types of nails, and all of these modules are specialty
hammers that are slightly distinct from eachother in how they work, but
you can use any of these hammers on any of these nails"  instead we should
say "here are some specialty hammers, you can use them for lots of
types of nails, ut for each hammer here is the type of nail where it
really shines"
"block-index-join" as i understand it requires all the docs you want to
join up to be in one contigious range of docids in the index, so if you want to
re-index one doc in a block you have to re-index the entire block -- so
the city/doctor example doesn't sound like a good generic example of
when/why to use this (because a doctor might change his office
hours, or address -- maybe even leavong the city completely, while a
city might change it's population w/o the doctor being affected at all.

The "book and pages" example seems much more appropriate, since in the
real world these things change in lock step -- pages aren't added/removed to
a book; pages don't change w/o the book itself being fundementally
changed.  the fields of a page document are the text of that page, and
that is inheriently data about the book -- the fields of a doctor
document are metadata about the doctor, and that is not inheriently data
about the city the doctor lives in.

as for the name ... i understand why it's called "module/join" and i
understand why the classes are called "BlockJoinQuery" and
"BlockJoinCollector" but i don't think those names really stand out and
convey to end users what they do and how/why they are useful.

Personally i think better names would be "modules/subdocuments",
"ParentDocumentQuery" and "ChildDocumentsCollector"

I know mcccandless isn't a fan of the name "Nested Documents" because this
functionality *can* be used for use cases where the data being modeled is
not strictly organized in a nested relationship, but that doesn't mean
it's *optimal* or easy for a user to apply to other usecases, because they
have to design their model (and their indexing strategy) in such a way
that they think them as nested or hierarchical documents.  

Naming it "module/subdocuments" would not only emphasis the usecase where
it really shines, it would more importantly draw attention to how users
have to model their data in order to take advantage of it -- and using
"ParentDocument" and "ChildDocuments" in the names of the Query/Collector
would make it clear what they "match" on relative the underlying query
that they wrap/collect

it would also help distibguish from more general joins like what solr
does today -- it seems like that should eventually take the name
"module/join"

At a minum we should rename what we have now "modules/block-join" or
"modules/index-join" (but the later is confusing) and eventually add
"modules/query-join"  (yes, yes, block joins provide a query, btu the
differnce is when you you have to make a decision about how you want to
join your model, at index time or at query time.
-Hoss

---------------------------------------------------------------------

OK I'm sold!

I agree: let's rename this new module according to the most likely use
case, not according to its "logical function", and I agree nested
documents is the compelling use case here.  Then fully generic joins
can go to a new module/join.

Maybe modules/nesteddocuments (I think that's more descriptive than
subdocuments)?

How about NestedDocumentQuery?  And NestedDocumentCollector?

See, you can use NestedDocumentQuery but collect it with any ordinary
collector if you don't care about the "nesting" (ie, you are only
interested in matches in the parent document space).  The
NestedDocumentCollector also collects all the nested docs matching
each parent hit.

You can of course still use this Query/Collector for any kind of
join, as long as your app is able to do this join at indexing time
and index all joined docs to a single row of the primary table as a
doc block.  But this will presumably be a less common use case so
I agree we should just name this feature according to its common use
case.

Mike McCandless

http://blog.mikemccandless.com

On Mon, Jul 4, 2011 at 1:34 PM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:
>
> : In my example the city was parent -- I raised this example to explain
> : that index-time joining is more general than just nested docs (ie, I
> : think we should keep the name "join" for this module... also because
> : we should factor out more general search-time-only join capabilities
> : into it).
>
> i think that may be the wrong approach to take when discussing "examples",
> while it's great to say there are dozens of usecases that these features
> can all support in dozens of diff ways" we should relaly focus on
> naming/deming these use cases in the ways where they really make the most
> sense.
>
> In otherwords, i don't think we should say "All of these types of problems
> are different types of nails, and all of these modules are specialty
> hammers that are slightly distinct from eachother in how they work, but
> you can use any of these hammers on any of these nails"  instead we should
> say "here are some specialty hammers, you can use them for lots of
> types of nails, ut for each hammer here is the type of nail where it
> really shines"
>
>
> "block-index-join" as i understand it requires all the docs you want to
> join up to be in one contigious range of docids in the index, so if you want to
> re-index one doc in a block you have to re-index the entire block -- so
> the city/doctor example doesn't sound like a good generic example of
> when/why to use this (because a doctor might change his office
> hours, or address -- maybe even leavong the city completely, while a
> city might change it's population w/o the doctor being affected at all.
>
> The "book and pages" example seems much more appropriate, since in the
> real world these things change in lock step -- pages aren't added/removed to
> a book; pages don't change w/o the book itself being fundementally
> changed.  the fields of a page document are the text of that page, and
> that is inheriently data about the book -- the fields of a doctor
> document are metadata about the doctor, and that is not inheriently data
> about the city the doctor lives in.
>
> as for the name ... i understand why it's called "module/join" and i
> understand why the classes are called "BlockJoinQuery" and
> "BlockJoinCollector" but i don't think those names really stand out and
> convey to end users what they do and how/why they are useful.
>
> Personally i think better names would be "modules/subdocuments",
> "ParentDocumentQuery" and "ChildDocumentsCollector"
>
> I know mcccandless isn't a fan of the name "Nested Documents" because this
> functionality *can* be used for use cases where the data being modeled is
> not strictly organized in a nested relationship, but that doesn't mean
> it's *optimal* or easy for a user to apply to other usecases, because they
> have to design their model (and their indexing strategy) in such a way
> that they think them as nested or hierarchical documents.

: Maybe modules/nesteddocuments (I think that's more descriptive than
: subdocuments)?

either way ... subdocuments has the advantage of being a shorter directory
name.  

i kinda wonder about first impressions and the entomology of "nested" ...
it makes me think of bird nests and russion dolls, neither of which
really convey the point: nesting in birds is about protecting/incubating
and is only a single layer; while russian nesting dolls are singular
wrappers arround wrappers arround wrappers.

subdocuments seems like it might better because it conveys more of a
hierarchical nature (to me anyway).

: How about NestedDocumentQuery?  And NestedDocumentCollector?
:
: See, you can use NestedDocumentQuery but collect it with any ordinary
: collector if you don't care about the "nesting" (ie, you are only
: interested in matches in the parent document space).  The
: NestedDocumentCollector also collects all the nested docs matching
: each parent hit.

Hmmm...

My suggestion of ParentDocumentQuery was based on the understanding that
the simplest usecase was...

  Query inner = getSomethingThatMatchesSomeChildDocs();
  Filter parents = someFilterThatMatcheAllKnownParentDocs()
  Query outer = new ParentDocumentQuery(inner, parents)
  TopDocs results = searcher.search(outer)

...and in this case "results" will contain the parents of the child
documents that match inner.  is that correct?

if so, then indepenent of the Collector, "ParentDocumentQuery" (or
ParentDocumentQueryWrapper) still seems like it makes the most sense.

For the Collector, i realize now that i totally missunderstood it's api --
for some reason i thought it would wrap another Collector and proxy to the
inner collector only the parents, independently collecting/recording the
groups of parent->children info which could be asked for later.  

"ChildDocumentsCollector" definitely doesn't make ense -- it's not
just collecting children, it's collecting Groups made up of parents
and children ... GroupCollector is obviously too general though ... i
would toss out "ParentChildrenTopGroupCollector" to make it clear that:
  a) what you can get out of it are instances of TopGroups
  b) the Groups consists of Parents and Children

...but that may be trying to convey too much in a classname.  

I certianly wouldn't complain about NestedDocumentCollector or
SubDocumentCollector if people like those better.
-Hoss

---------------------------------------------------------------------

On Mon, Jul 4, 2011 at 3:38 PM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:
>
> : Maybe modules/nesteddocuments (I think that's more descriptive than
> : subdocuments)?
>
> either way ... subdocuments has the advantage of being a shorter directory
> name.

Yeah both are rather long...

Maybe modules/nested? modules/nesteddocs?

> i kinda wonder about first impressions and the entomology of "nested" ...
> it makes me think of bird nests and russion dolls, neither of which
> really convey the point: nesting in birds is about protecting/incubating
> and is only a single layer; while russian nesting dolls are singular
> wrappers arround wrappers arround wrappers.
>
> subdocuments seems like it might better because it conveys more of a
> hierarchical nature (to me anyway).

Hmm... sub feels like it undersells, ie emphasizes "under" or
"inferior to" and de-emphasizes the strong cooperation w/ the parent.

Also, the nesting is not just one level -- it can support an arbitrary
star join.  So you can join from main to table1 and then from table1
to table2 (parent, child, grandchild).  You can also join to multiple
child tables from the main table.

I think nested/nesting has strong enough meaning among programmers
that most will understand what it means in this context.

> : How about NestedDocumentQuery?  And NestedDocumentCollector?
> :
> : See, you can use NestedDocumentQuery but collect it with any ordinary
> : collector if you don't care about the "nesting" (ie, you are only
> : interested in matches in the parent document space).  The
> : NestedDocumentCollector also collects all the nested docs matching
> : each parent hit.
>
> Hmmm...
>
> My suggestion of ParentDocumentQuery was based on the understanding that
> the simplest usecase was...
>
>  Query inner = getSomethingThatMatchesSomeChildDocs();
>  Filter parents = someFilterThatMatcheAllKnownParentDocs()
>  Query outer = new ParentDocumentQuery(inner, parents)
>  TopDocs results = searcher.search(outer)
>
> ...and in this case "results" will contain the parents of the child
> documents that match inner.  is that correct?

Correct.

> if so, then indepenent of the Collector, "ParentDocumentQuery" (or
> ParentDocumentQueryWrapper) still seems like it makes the most sense.

Hmm, but that doesn't convey that it handles this nesting, ie, that
it's joining child docs with parent docs.

Also, these queries can be nested (from 2nd join in the star join),
and so it could be ChildAndGrandChildrenQuery.

I guess Wrapper would make sense since it wraps a query matching the
nested docs.  I think Document is redundant/implied?

Maybe NestedQueryWrapper?

> For the Collector, i realize now that i totally missunderstood it's api --
> for some reason i thought it would wrap another Collector and proxy to the
> inner collector only the parents, independently collecting/recording the
> groups of parent->children info which could be asked for later.
>
> "ChildDocumentsCollector" definitely doesn't make ense -- it's not
> just collecting children, it's collecting Groups made up of parents
> and children ... GroupCollector is obviously too general though ... i
> would toss out "ParentChildrenTopGroupCollector" to make it clear that:
>  a) what you can get out of it are instances of TopGroups
>  b) the Groups consists of Parents and Children
>
> ...but that may be trying to convey too much in a classname.

I agree we want Top in the name, since it's collecting Top hits
according to provided Sort... I don't think we should put Groups in
the name just because this class (TopGroups) is used to represent the
returned hits.  Really in this context they aren't "groups" in the
grouping module sense; they are the nested docs (parent + children),
just using TopGroups to represent that for now.

In fact, once we generalize TopDocs so that the type of each hit can
be parameterized then this collector would return TopDocs<NestedDoc>
and each NestedDoc would have parent docID, maybe sort field values,
and then the TopDocs<ScoreDoc> holding the child hits.  (But I'm
scared of the generics required here!).

So I guess I would keep Top but drop Groups, and replace
ParentChildren with NestedDocs and move the Top in front:
TopNestedDocsCollector.

Mike

On Tue, Jul 5, 2011 at 6:46 AM, Michael McCandless
<[EMAIL PROTECTED]> wrote:
> On Mon, Jul 4, 2011 at 3:38 PM, Chris Hostetter
> <[EMAIL PROTECTED]> wrote:
>>
>> : Maybe modules/nesteddocuments (I think that's more descriptive than
>> : subdocuments)?
>>
>> either way ... subdocuments has the advantage of being a shorter directory
>> name.
>
> Yeah both are rather long...

subdocs ?

---------------------------------------------------------------------

>>
>> either way ... subdocuments has the advantage of being a shorter directory
>> name.
>
> Yeah both are rather long...
>
> Maybe modules/nested? modules/nesteddocs?
>

I like modules/nested

---------------------------------------------------------------------

: Maybe modules/nested? modules/nesteddocs?

modules/subdocs
modules/nesteddocs
modules/nested

None of them scream "this is the perfect name" to me, but none of them
scream "dear lord this is a terrible idea" either.

Instinct says "All other factors being equal, pick the shortest name"

: Hmm... sub feels like it undersells, ie emphasizes "under" or
: "inferior to" and de-emphasizes the strong cooperation w/ the parent.

hmmm, maybe.  Coming from a background where the taxonomy was king i would
never think that about "subdocuments", but i respect that my perception
may be unique that way.

: Also, the nesting is not just one level -- it can support an arbitrary
: star join.  So you can join from main to table1 and then from table1
: to table2 (parent, child, grandchild).  You can also join to multiple
: child tables from the main table.

Sure, and i agree my russian doll image fits with that part -- my concern
is that "nesting" in that context suggests a singler child (which may have
it's own singular child, etc...).

6 of 1, half dozen of the other.

: > if so, then indepenent of the Collector, "ParentDocumentQuery" (or
: > ParentDocumentQueryWrapper) still seems like it makes the most sense.
:
: Hmm, but that doesn't convey that it handles this nesting, ie, that
: it's joining child docs with parent docs.

you lost me there ... i feel like i'm not understanding your use of
"joining"

If...
  * x is the parent of x1, x2, etc...
  * y is the parent of y1, y2, etc...
  * queryA matches w5, x1, x2, and z3
  * queryB = BlockJoinQuery(queryA)

...then doesn't queryB only match w, x, and z?

Isn't it just a query that wraps another query and returns the parents of
the docs matched by the wrapped query?

: Also, these queries can be nested (from 2nd join in the star join),
: and so it could be ChildAndGrandChildrenQuery.

a) wouldn't that require you to wrap multiple of them? (ie: new
BlockJoinQuery(new BlockJoinQuery(childQ, ...), ...)

b) the use of child suggests that if the base query matches parents, then
the wrapper will match children ... i think you really want it to clarify
that the relationship goes the other way -- it returns parents (and grand
parents, etc..) of the wrapped query ?

: I guess Wrapper would make sense since it wraps a query matching the
: nested docs.  I think Document is redundant/implied?
:
: Maybe NestedQueryWrapper?

ugh... that seems like it might be really confusing ... "nested what?" ...
"nested query?" .. "of course it's a wrapper, it wraps a nested query"

another reason why naming they query after the 'parents' might better
explain what it does.

if i'm missunderstanding the docs, and it can go all the way up the
taxonomy w/o having to use nest instances inside other instances then
maybe "AncestorDocumentQueryWrapper" or "OuterDocumentQueryWrapper" could
make sense?  

Dare i suggest "WrappingDocumentQueryWrapper" ?

(i hate naming shit ... good names are too fucking hard .. and i can't
find any antonyms for "nested" in the context we mean)

: In fact, once we generalize TopDocs so that the type of each hit can
: be parameterized then this collector would return TopDocs<NestedDoc>
...
: So I guess I would keep Top but drop Groups, and replace
: ParentChildren with NestedDocs and move the Top in front:
: TopNestedDocsCollector.

yep, yep ... sounds just ... just feel like we need
something better then anything we've come up for so far for the query,
something to adequately explain that it (essenially) does the inverse of
the collector -- going up the taxonomy and matching the parents/wrappers
of the nested documents matched by the base query.
-Hoss

---------------------------------------------------------------------

> : Maybe modules/nested? modules/nesteddocs?
>
> modules/subdocs
> modules/nesteddocs
> modules/nested
>
> None of them scream "this is the perfect name" to me, but none of them
> scream "dear lord this is a terrible idea" either.
>
> Instinct says "All other factors being equal, pick the shortest name"
>
> : Hmm... sub feels like it undersells, ie emphasizes "under" or
> : "inferior to" and de-emphasizes the strong cooperation w/ the parent.
>    
How about modules/superdoc?

It wouldn't undersell, at least :)

-Sokolov

and SuperDocQueryWrapper, etc...

---------------------------------------------------------------------

: How about modules/superdoc?
:
: It wouldn't undersell, at least :)

you make a frightening compelling argument

modules/superdoc - "Classes for dealing with Data which can be
modeled as a nested hierarchy of documents which can be contained in
super documents (which may themselves be contained in larger super
documents)"

SuperDocQueryWraper - given a query which matches documents, wraps that
query and returns the super documents

TopSuperDocCollector - when used to collect matches of a query,
collects an SuperDoc instances for each matching document, containing the
matching nested document if that doc has any.

-Hoss

---------------------------------------------------------------------

Of course you could take this whole hierarchical thing down the XML route i.e.
Lucene as an XML store.
Everyone knows what XML is and it is open to representing lots of things and
there is a lot of data already out there in this form.
I believe this is MarkLogic's schtick.

There are several disadvantages though -  XML may bring with it too much baggage
in the form of existing standards e.g. people wanting XQuery support, includes,
schemas etc.
XML attributes vs nested elements and how they map to physical Lucene docs is
also not necessarily a neat fit or one that can easily be automated.
Existing XML query languages are probably too binary in their matching logic (it
matches or it doesn't - no relevance ranking etc)  and also too restrictive in
their query types so I wouldn't bother implementing the  XQuery syntax
----- Original Message ----
From: Chris Hostetter <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Tue, 5 July, 2011 21:58:35
Subject: Re: revisit naming for grouping/join?
: Maybe modules/nested? modules/nesteddocs?

    modules/subdocs
    modules/nesteddocs
    modules/nested

None of them scream "this is the perfect name" to me, but none of them
scream "dear lord this is a terrible idea" either.

Instinct says "All other factors being equal, pick the shortest name"

: Hmm... sub feels like it undersells, ie emphasizes "under" or
: "inferior to" and de-emphasizes the strong cooperation w/ the parent.

hmmm, maybe.  Coming from a background where the taxonomy was king i would
never think that about "subdocuments", but i respect that my perception
may be unique that way.

: Also, the nesting is not just one level -- it can support an arbitrary
: star join.  So you can join from main to table1 and then from table1
: to table2 (parent, child, grandchild).  You can also join to multiple
: child tables from the main table.

Sure, and i agree my russian doll image fits with that part -- my concern
is that "nesting" in that context suggests a singler child (which may have
it's own singular child, etc...).

6 of 1, half dozen of the other.

: > if so, then indepenent of the Collector, "ParentDocumentQuery" (or
: > ParentDocumentQueryWrapper) still seems like it makes the most sense.
:
: Hmm, but that doesn't convey that it handles this nesting, ie, that
: it's joining child docs with parent docs.

you lost me there ... i feel like i'm not understanding your use of
"joining"

If...
  * x is the parent of x1, x2, etc...
  * y is the parent of y1, y2, etc...
  * queryA matches w5, x1, x2, and z3
  * queryB = BlockJoinQuery(queryA)

...then doesn't queryB only match w, x, and z?

Isn't it just a query that wraps another query and returns the parents of
the docs matched by the wrapped query?

: Also, these queries can be nested (from 2nd join in the star join),
: and so it could be ChildAndGrandChildrenQuery.

a) wouldn't that require you to wrap multiple of them? (ie: new
BlockJoinQuery(new BlockJoinQuery(childQ, ...), ...)

b) the use of child suggests that if the base query matches parents, then
the wrapper will match children ... i think you really want it to clarify
that the relationship goes the other way -- it returns parents (and grand
parents, etc..) of the wrapped query ?

: I guess Wrapper would make sense since it wraps a query matching the
: nested docs.  I think Document is redundant/implied?
:
: Maybe NestedQueryWrapper?

ugh... that seems like it might be really confusing ... "nested what?" ...
"nested query?" .. "of course it's a wrapper, it wraps a nested query"

another reason why naming they query after the 'parents' might better
explain what it does.

if i'm missunderstanding the docs, and it can go all the way up the
taxonomy w/o having to use nest instances inside other instances then
maybe "AncestorDocumentQueryWrapper" or "OuterDocumentQueryWrapper" could
make sense?  

Dare i suggest "WrappingDocumentQueryWrapper" ?

(i hate naming shit ... good names are too fucking hard .. and i can't
find any antonyms for "nested" in the context we mean)

    ...

yep, yep ... sounds just ... just feel like we need
something better then anything we've come up for so far for the query,
something to adequately explain that it (essenially) does the inverse of
the collector -- going up the taxonomy and matching the parents/wrappers
of the nested documents matched by the base query.
-Hoss

For additional commands, e-mail: [EMAIL PROTECTED]

On Tue, Jul 5, 2011 at 4:58 PM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:

> : > if so, then indepenent of the Collector, "ParentDocumentQuery" (or
> : > ParentDocumentQueryWrapper) still seems like it makes the most sense.
> :
> : Hmm, but that doesn't convey that it handles this nesting, ie, that
> : it's joining child docs with parent docs.
>
> you lost me there ... i feel like i'm not understanding your use of
> "joining"
>
> If...
>  * x is the parent of x1, x2, etc...
>  * y is the parent of y1, y2, etc...
>  * queryA matches w5, x1, x2, and z3
>  * queryB = BlockJoinQuery(queryA)
>
> ...then doesn't queryB only match w, x, and z?
>
> Isn't it just a query that wraps another query and returns the parents of
> the docs matched by the wrapped query?

Yes, it is... though, I think "normally" one would do a new
BlockJoinQuery per table (ie one for x, one for y, one for z); this
way you can pull child hits per-table.  But I believe your way (doing
3 joins w/ one BlockJoinQuery) will work fine (ie give the right
parent hits).

And this is the same thing as "joining" (relating the child matches
"up" to the corresponding parents), ie, this query "finishes" the join
done during indexing.

> : Also, these queries can be nested (from 2nd join in the star join),
> : and so it could be ChildAndGrandChildrenQuery.
>
> a) wouldn't that require you to wrap multiple of them? (ie: new
> BlockJoinQuery(new BlockJoinQuery(childQ, ...), ...)

Yes it would, ie, just like doing multiple joins.

> b) the use of child suggests that if the base query matches parents, then
> the wrapper will match children ... i think you really want it to clarify
> that the relationship goes the other way -- it returns parents (and grand
> parents, etc..) of the wrapped query ?

I think we want to emphasize both, that it expects child matches from
the wrapped query and then returns corresponding ("joined") parent
matches.

Also... I think we are over-thinking the name ;)  We can't convey
*everything* in this name; as long as the name makes it clear that
you'll want to consider this / read its javadocs whenever doing
something with "nested docs", I think that's sufficient.  I think
NestedQueryWrapper (maybe NestedDocsQuery) and NestedDocsCollector are
good enough, at least better than the functional-driven names they now
have...

> : I guess Wrapper would make sense since it wraps a query matching the
> : nested docs.  I think Document is redundant/implied?
> :
> : Maybe NestedQueryWrapper?
>
> ugh... that seems like it might be really confusing ... "nested what?" ...
> "nested query?" .. "of course it's a wrapper, it wraps a nested query"

Yeah I think maybe drop the Wrapper and add Docs: NestedDocsQuery?

> another reason why naming they query after the 'parents' might better
> explain what it does.

I think parents is under-stating it, since it's really a bidirectional
thing.  Ie the BlockJoinQuery interacts with both parent and child
hits.

> if i'm missunderstanding the docs, and it can go all the way up the
> taxonomy w/o having to use nest instances inside other instances then
> maybe "AncestorDocumentQueryWrapper" or "OuterDocumentQueryWrapper" could
> make sense?
>
> Dare i suggest "WrappingDocumentQueryWrapper" ?

Too long :)

> (i hate naming shit ... good names are too fucking hard .. and i can't
> find any antonyms for "nested" in the context we mean)

Naming is the hardest part :)

> : In fact, once we generalize TopDocs so that the type of each hit can
> : be parameterized then this collector would return TopDocs<NestedDoc>
>        ...
> : So I guess I would keep Top but drop Groups, and replace
> : ParentChildren with NestedDocs and move the Top in front:
> : TopNestedDocsCollector.
>
> yep, yep ... sounds just ... just feel like we need
> something better then anything we've come up for so far for the query,
> something to adequately explain that it (essenially) does the inverse of
> the collector -- going up the taxonomy and matching the parents/wrappers

Honestly at this point I'm tempted to just stick with what we have
(the functionally driven names, instead of the dominant use case
driven name).

At its heart, this query is performing a join (well, finishing the
join that was done during indexing), and despite our efforts to more
descriptively capture the dominant use case, I don't think we're
succeeding.  We are basically struggling to find ways to explain what
a join does, into these class names.

Mike McCandless

http://blog.mikemccandless.com

Right -- searching XML docs is a great fit for
BlockJoinQuery/Collector (and XML doc is really just a single-doc star
join).

But I agree picking XML search as the "dominant use case we use to
name this module and its classes" is dangerous because we don't (yet?)
do all the other things (eg XQuery) that one would expect with XML
search.

I think this is also another good reason to fall back to the
functional names (because this Query/Collector is just "finishing" a
join, where the join was done during indexing) instead of the use-case
names, ie, "XML search" is also a dominant use case here, just liked
"nested docs".

SKU search, obviously a common use case in e-commerce, is another example.

So maybe we should not be trying to name these classes on any single
use case... maybe instead name them by their function ("join"), and
then make good examples that show how each of these use cases can be
handled as index-time joins.

Mike McCandless

http://blog.mikemccandless.com

On Wed, Jul 6, 2011 at 5:09 AM, mark harwood <[EMAIL PROTECTED]> wrote:
> Of course you could take this whole hierarchical thing down the XML route i.e.
> Lucene as an XML store.
> Everyone knows what XML is and it is open to representing lots of things and
> there is a lot of data already out there in this form.
> I believe this is MarkLogic's schtick.
>
> There are several disadvantages though -  XML may bring with it too much baggage
> in the form of existing standards e.g. people wanting XQuery support, includes,
> schemas etc.
> XML attributes vs nested elements and how they map to physical Lucene docs is
> also not necessarily a neat fit or one that can easily be automated.
> Existing XML query languages are probably too binary in their matching logic (it
> matches or it doesn't - no relevance ranking etc)  and also too restrictive in
> their query types so I wouldn't bother implementing the  XQuery syntax
>
>
>
>
> ----- Original Message ----
> From: Chris Hostetter <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Tue, 5 July, 2011 21:58:35
> Subject: Re: revisit naming for grouping/join?
>
>
> : Maybe modules/nested? modules/nesteddocs?
>
>    modules/subdocs
>    modules/nesteddocs
>    modules/nested
>
> None of them scream "this is the perfect name" to me, but none of them
> scream "dear lord this is a terrible idea" either.
>
> Instinct says "All other factors being equal, pick the shortest name"
>
> : Hmm... sub feels like it undersells, ie emphasizes "under" or
> : "inferior to" and de-emphasizes the strong cooperation w/ the parent.
>
> hmmm, maybe.  Coming from a background where the taxonomy was king i would
> never think that about "subdocuments", but i respect that my perception
> may be unique that way.
>
> : Also, the nesting is not just one level -- it can support an arbitrary
> : star join.  So you can join from main to table1 and then from table1
> : to table2 (parent, child, grandchild).  You can also join to multiple
> : child tables from the main table.
>
> Sure, and i agree my russian doll image fits with that part -- my concern
> is that "nesting" in that context suggests a singler child (which may have
> it's own singular child, etc...).
>
> 6 of 1, half dozen of the other.
>
> : > if so, then indepenent of the Collector, "ParentDocumentQuery" (or
> : > ParentDocumentQueryWrapper) still seems like it makes the most sense.
> :
> : Hmm, but that doesn't convey that it handles this nesting, ie, that
> : it's joining child docs with parent docs.
>
> you lost me there ... i feel like i'm not understanding your use of
> "joining"
>
> If...
>  * x is the parent of x1, x2, etc...
>  * y is the parent of y1, y2, etc...
>  * queryA matches w5, x1, x2, and z3
>  * queryB = BlockJoinQuery(queryA)
>
> ...then doesn't queryB only match w, x, and z?
>
> Isn't it just a query that wraps another query and returns the parents of
> the docs matched by the wrapped query?
>
> : Also, these queries can be nested (from 2nd join in the star join),

On Tue, Jul 5, 2011 at 5:44 PM, Mike Sokolov <[EMAIL PROTECTED]> wrote:
>
>> : Maybe modules/nested? modules/nesteddocs?
>>
>>        modules/subdocs
>>        modules/nesteddocs
>>        modules/nested
>>
>> None of them scream "this is the perfect name" to me, but none of them
>> scream "dear lord this is a terrible idea" either.
>>
>> Instinct says "All other factors being equal, pick the shortest name"
>>
>> : Hmm... sub feels like it undersells, ie emphasizes "under" or
>> : "inferior to" and de-emphasizes the strong cooperation w/ the parent.
>>
>
> How about modules/superdoc?
>
> It wouldn't undersell, at least :)

I agree it's no longer under selling :)

But I like this even less than sub!  First, I think it has the same
problems that sub has since it's just symmetric: it's too un-equal, ie
implies one side is "superior" and "above" the other side, when in
fact joining (XML search, product SKUs, nested docs, etc.) are really
symmetric.  The nested parts of the doc are just as valid a part of
the document as the non-nested part.

Second, I don't like the "super-ness" of super (ie, in the sense of
supercalifragilisticexpialidocious or superman or superwoman) -- it's
too generic, ie, like "best" or "awesome".

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------

On 07/06/2011 08:47 AM, Michael McCandless wrote:
>> How about modules/superdoc?
>>
>> It wouldn't undersell, at least :)
>>      
> I agree it's no longer under selling :)
>
> But I like this even less than sub!  First, I think it has the same
> problems that sub has since it's just symmetric: it's too un-equal, ie
> implies one side is "superior" and "above" the other side,
I basically agree, although I think there is an asymmetry in that this
is a many-one relation?  The main improvement this name makes is the
removal of the plural in the other options (doc vs docs).  And it's
shorter than huperduperdoc :)  But otoh nothing I've seen here really
captures all that much about index-time vs query-time join, which seems
to be the main distinction (why you can't just call it "join")?  If
you're still in the market for names here are a few: StructureJoin,
IntrinsicJoin, TreeJoin; Branch? Just brainstorming loosely.  Frankly
Nest* seems well enough.

-Sokolov

---------------------------------------------------------------------

: Also... I think we are over-thinking the name ;)  We can't convey
: *everything* in this name; as long as the name makes it clear that
: you'll want to consider this / read its javadocs whenever doing
: something with "nested docs", I think that's sufficient.  I think
: NestedQueryWrapper (maybe NestedDocsQuery) and NestedDocsCollector are
: good enough, at least better than the functional-driven names they now
: have...

Yeah, that's fair ... i'm not in love with NestedDocsQuery and
NestedDocsCollector but i agree they are better then what we have now.

: Honestly at this point I'm tempted to just stick with what we have
: (the functionally driven names, instead of the dominant use case
: driven name).
:
: At its heart, this query is performing a join (well, finishing the
: join that was done during indexing), and despite our efforts to more
: descriptively capture the dominant use case, I don't think we're
: succeeding.  We are basically struggling to find ways to explain what
: a join does, into these class names.

I really think it's a bad idea to use "Join" in the name ... i understand
that to you this is a "join", but as you say it's really just finishing a
join that was already done at index time -- for most users "join" is
going to have the connotation of a SQL join where you don't have to
normalize the data in advance (ie: build the index with all the docs you
want ot join in a block) and we shouldn't use it unless we are talking
about a truely generic query time join -- particularly if we are going to
use examples i nthe doc that seem like the kind of think you would do with
a query time join in SQL.

i know you feel like "nested" (or "subdocs" or "parent") undersells the
*possible* usecases of this feature, but the thing to remember is that
even in the use cases where the real life data isn't something you might
think of as being organized in a "nested" or "hierarchical" model, in
order to use this feature the user must map their source data model to a
Lucene Document model that *does* capture a hierarchy relationship so they
can index their data in in the appropraite way.  X and Y may not be in a
hierarchy, but if you want to join them like this, then the Document for X
and the Document for Y must be thought of as being in a hierarchy and
indexed in lock step with eachother.

"Block" just doesn't feel like it really conveys this ... but anything
along the "Nested", "Parent", "Subdoc", line of terminology would at least
give some point of refrence to the idea that the *Document* model in
Lucene needs to be organized in this way -- and i think it's really
important that the name make that clear.

-Hoss

---------------------------------------------------------------------

From my external POV on this debate, it seems as though the main point of contention is naming the nature of the relation between documents.  

Instead of doing that, a name that says that there is some form of relation, but leaving open its nature, might work: something like "docrelation"?  (Avoiding the "related documents" IR concept would be important here.)

Steve

> -----Original Message-----
> From: Chris Hostetter [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, July 06, 2011 2:59 PM
> To: [EMAIL PROTECTED]
> Subject: Re: revisit naming for grouping/join?
>
>
> : Also... I think we are over-thinking the name ;)  We can't convey
> : *everything* in this name; as long as the name makes it clear that
> : you'll want to consider this / read its javadocs whenever doing
> : something with "nested docs", I think that's sufficient.  I think
> : NestedQueryWrapper (maybe NestedDocsQuery) and NestedDocsCollector are
> : good enough, at least better than the functional-driven names they now
> : have...
>
> Yeah, that's fair ... i'm not in love with NestedDocsQuery and
> NestedDocsCollector but i agree they are better then what we have now.
>
> : Honestly at this point I'm tempted to just stick with what we have
> : (the functionally driven names, instead of the dominant use case
> : driven name).
> :
> : At its heart, this query is performing a join (well, finishing the
> : join that was done during indexing), and despite our efforts to more
> : descriptively capture the dominant use case, I don't think we're
> : succeeding.  We are basically struggling to find ways to explain what
> : a join does, into these class names.
>
> I really think it's a bad idea to use "Join" in the name ... i understand
> that to you this is a "join", but as you say it's really just finishing a
> join that was already done at index time -- for most users "join" is
> going to have the connotation of a SQL join where you don't have to
> normalize the data in advance (ie: build the index with all the docs you
> want ot join in a block) and we shouldn't use it unless we are talking
> about a truely generic query time join -- particularly if we are going to
> use examples i nthe doc that seem like the kind of think you would do
> with
> a query time join in SQL.
>
> i know you feel like "nested" (or "subdocs" or "parent") undersells the
> *possible* usecases of this feature, but the thing to remember is that
> even in the use cases where the real life data isn't something you might
> think of as being organized in a "nested" or "hierarchical" model, in
> order to use this feature the user must map their source data model to a
> Lucene Document model that *does* capture a hierarchy relationship so
> they
> can index their data in in the appropraite way.  X and Y may not be in a
> hierarchy, but if you want to join them like this, then the Document for
> X
> and the Document for Y must be thought of as being in a hierarchy and
> indexed in lock step with eachother.
>
> "Block" just doesn't feel like it really conveys this ... but anything
> along the "Nested", "Parent", "Subdoc", line of terminology would at
> least
> give some point of refrence to the idea that the *Document* model in
> Lucene needs to be organized in this way -- and i think it's really
> important that the name make that clear.
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]