With a brand new setup, per the demo/tutorial, with fields slightly changed in the config and data, posting XML records results in a simple qiery being able to find records.
But records imported via a plain jane DIH request can only be found using 'q=*:*' queries.

There's no filtering, tokenizing, blah blah. It's the factory settings. The installation is as new at this as we are :-)

Anyone have any ideas why we can't query for DIH handled records? Do they have some magic juju done to them that XML Posts don't, or visa versa?

Dennis Gearon

Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'

EARTH has a Right To Life,
  otherwise we all die.

The phrase that jumps out is "with fields slightly modified". I'm
guessing that your modifications are off by a little. Here's what
I'd check first:
1> check the case. Sometimes the DB <-> field link is case
sensitive.
2> Look in your index via the admin page and look at your actual
fields as reported there. Are they really what you expect?
3> Try your query with &debugQuery=on. Is what you get back
what you expect?
4> Sometimes your browser cache will fool you, try the force-refresh
combination on your browser.

There's no magic here, nothing special or different about DIH
imported data than any other sort. So it's almost certainly
some innocent-seeming change that's not, typo, incorrect
assumption, etc.

If none of that works, you need to post your schema changes and
your query results (with &debugQuery=on). Particularly, post
the fieldType definitions as well as your field definitions...

Best
Erick

On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon <[EMAIL PROTECTED]>wrote:

> With a brand new setup, per the demo/tutorial, with fields slightly changed
> in the config and data, posting XML records results in a simple qiery being
> able to find records.
>
>
> But records imported via a plain jane DIH request can only be found using
> 'q=*:*' queries.
>
> There's no filtering, tokenizing, blah blah. It's the factory settings. The
> installation is as new at this as we are :-)
>
> Anyone have any ideas why we can't query for DIH handled records? Do they
> have some magic juju done to them that XML Posts don't, or visa versa?
>
> Dennis Gearon
>
> Signature Warning
> ----------------
> It is always a good idea to learn from your own mistakes. It is usually a
> better idea to learn from others’ mistakes, so you do not have to make them
> yourself. from '
> http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>
> EARTH has a Right To Life,
>  otherwise we all die.
>

Thanks for answering on a Sunday :-)

What happens if the query returns extra fields that are not specified in the config.xml?

And does the unque field have to be named 'id'?

Dennis Gearon

Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'

EARTH has a Right To Life,
  otherwise we all die.
--- On Sun, 10/10/10, Erick Erickson <[EMAIL PROTECTED]> wrote:

> From: Erick Erickson <[EMAIL PROTECTED]>
> Subject: Re: Records from DIH not easily queried for
> To: [EMAIL PROTECTED]
> Date: Sunday, October 10, 2010, 8:11 AM
> The phrase that jumps out is "with
> fields slightly modified". I'm
> guessing that your modifications are off by a little.
> Here's what
> I'd check first:
> 1> check the case. Sometimes the DB <-> field link
> is case
> sensitive.
> 2> Look in your index via the admin page and look at
> your actual
> fields as reported there. Are they really what you expect?
> 3> Try your query with &debugQuery=on. Is what you
> get back
> what you expect?
> 4> Sometimes your browser cache will fool you, try the
> force-refresh
> combination on your browser.
>
> There's no magic here, nothing special or different about
> DIH
> imported data than any other sort. So it's almost
> certainly
> some innocent-seeming change that's not, typo, incorrect
> assumption, etc.
>
> If none of that works, you need to post your schema changes
> and
> your query results (with &debugQuery=on). Particularly,
> post
> the fieldType definitions as well as your field
> definitions...
>
> Best
> Erick
>
> On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon <[EMAIL PROTECTED]>wrote:
>
> > With a brand new setup, per the demo/tutorial, with
> fields slightly changed
> > in the config and data, posting XML records results in
> a simple qiery being
> > able to find records.
> >
> >
> > But records imported via a plain jane DIH request can
> only be found using
> > 'q=*:*' queries.
> >
> > There's no filtering, tokenizing, blah blah. It's the
> factory settings. The
> > installation is as new at this as we are :-)
> >
> > Anyone have any ideas why we can't query for DIH
> handled records? Do they
> > have some magic juju done to them that XML Posts
> don't, or visa versa?
> >
> > Dennis Gearon
> >
> > Signature Warning
> > ----------------
> > It is always a good idea to learn from your own
> mistakes. It is usually a
> > better idea to learn from others’ mistakes, so you
> do not have to make them
> > yourself. from '
> > http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> >
> > EARTH has a Right To Life,
> >  otherwise we all die.
> >
>

Dennis,

The unique field does not have to be named "id".
Fields are specified in schema.xml.  There is no config.xml, although there is
solrconfig.xml.
If a query is returning fields that you do not have defined in schema.xml, it's
time to stop working on weekends! ;)  Seriously, though, that cannot be
happening.  Maybe you can show the relevant part of your schema file, your
response, and your request?
Otis----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/

----- Original Message ----
> From: Dennis Gearon <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Sun, October 10, 2010 8:47:29 PM
> Subject: Re: Records from DIH not easily queried for
>
> Thanks for answering on a Sunday :-)
>
> What happens if the query returns  extra fields that are not specified in the
>config.xml?
>
> And does the unque  field have to be named 'id'?
>
> Dennis Gearon
>
> Signature  Warning
> ----------------
> It is always a good idea to learn from your own  mistakes. It is usually a
>better idea to learn from others’ mistakes, so you do  not have to make them
>yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
>
> EARTH  has a Right To Life,
>   otherwise we all die.
>
>
> --- On Sun,  10/10/10, Erick Erickson <[EMAIL PROTECTED]>  wrote:
>
> > From: Erick Erickson <[EMAIL PROTECTED]>
> >  Subject: Re: Records from DIH not easily queried for
> > To: [EMAIL PROTECTED]
> >  Date: Sunday, October 10, 2010, 8:11 AM
> > The phrase that jumps out is  "with
> > fields slightly modified". I'm
> > guessing that your  modifications are off by a little.
> > Here's what
> > I'd check  first:
> > 1> check the case. Sometimes the DB <-> field  link
> > is case
> > sensitive.
> > 2> Look in your index via the  admin page and look at
> > your actual
> > fields as reported there. Are  they really what you expect?
> > 3> Try your query with  &debugQuery=on. Is what you
> > get back
> > what you  expect?
> > 4> Sometimes your browser cache will fool you, try  the
> > force-refresh
> > combination on your browser.
> >
> >  There's no magic here, nothing special or different about
> > DIH
> >  imported data than any other sort. So it's almost
> > certainly
> > some  innocent-seeming change that's not, typo, incorrect
> > assumption,  etc.
> >
> > If none of that works, you need to post your schema  changes
> > and
> > your query results (with &debugQuery=on).  Particularly,
> > post
> > the fieldType definitions as well as your  field
> > definitions...
> >
> > Best
> > Erick
> >
> > On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon
<[EMAIL PROTECTED]>wrote:
> >
> > > With a brand new setup, per the demo/tutorial, with
> > fields  slightly changed
> > > in the config and data, posting XML records  results in
> > a simple qiery being
> > > able to find  records.
> > >
> > >
> > > But records imported via a plain  jane DIH request can
> > only be found using
> > > 'q=*:*'  queries.
> > >
> > > There's no filtering, tokenizing, blah blah.  It's the
> > factory settings. The
> > > installation is as new at  this as we are :-)
> > >
> > > Anyone have any ideas why we can't  query for DIH
> > handled records? Do they
> > > have some magic juju  done to them that XML Posts
> > don't, or visa versa?
> > >
> >  > Dennis Gearon
> > >
> > > Signature Warning
> > >  ----------------
> > > It is always a good idea to learn from your  own
> > mistakes. It is usually a
> > > better idea to learn from  others’ mistakes, so you
> > do not have to make them
> > > yourself.  from '
> > > http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> >  >
> > > EARTH has a Right To Life,
> > >  otherwise we all  die.
> > >
> >
>

hmmmm,
     I am talking about the query for the DIH (Data Import Handler?) that queries the database to find records/documents to index. We originally were trying to import aout 20 fields, but have whittled down the schema file (my bad calling in config) to only try a few fields.

I myself have been STUDYING this whole Solr/Lucene product, while actually DOING the backend and interface to the front end. HE is doing the search engine and I'm trying to apply my limited knowledge and ask questions here.

I should have a pow wow with him so we present a united front to ask questions. I'll start a new thread when we do that. Thanks for the help, Otis. :-)

And for Solr/Lucene.

Dennis Gearon

Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'

EARTH has a Right To Life,
  otherwise we all die.
--- On Sun, 10/10/10, Otis Gospodnetic <[EMAIL PROTECTED]> wrote:

> From: Otis Gospodnetic <[EMAIL PROTECTED]>
> Subject: Re: Records from DIH not easily queried for
> To: [EMAIL PROTECTED]
> Date: Sunday, October 10, 2010, 8:47 PM
> Dennis,
>
> The unique field does not have to be named "id".
> Fields are specified in schema.xml.  There is no
> config.xml, although there is
> solrconfig.xml.
> If a query is returning fields that you do not have defined
> in schema.xml, it's
> time to stop working on weekends! ;)  Seriously,
> though, that cannot be
> happening.  Maybe you can show the relevant part of
> your schema file, your
> response, and your request?
>
>
> Otis----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>
>
>
> ----- Original Message ----
> > From: Dennis Gearon <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Sun, October 10, 2010 8:47:29 PM
> > Subject: Re: Records from DIH not easily queried for
> >
> > Thanks for answering on a Sunday :-)
> >
> > What happens if the query returns  extra fields
> that are not specified in the
> >config.xml?
> >
> > And does the unque  field have to be named 'id'?
> >
> > Dennis Gearon
> >
> > Signature  Warning
> > ----------------
> > It is always a good idea to learn from your own 
> mistakes. It is usually a
> >better idea to learn from others’ mistakes, so you
> do  not have to make them
> >yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> >
> > EARTH  has a Right To Life,
> >   otherwise we all die.
> >
> >
> > --- On Sun,  10/10/10, Erick Erickson <[EMAIL PROTECTED]> 
> wrote:
> >
> > > From: Erick Erickson <[EMAIL PROTECTED]>
> > >  Subject: Re: Records from DIH not easily
> queried for
> > > To: [EMAIL PROTECTED]
> > >  Date: Sunday, October 10, 2010, 8:11 AM
> > > The phrase that jumps out is  "with
> > > fields slightly modified". I'm
> > > guessing that your  modifications are off by
> a little.
> > > Here's what
> > > I'd check  first:
> > > 1> check the case. Sometimes the DB <->
> field  link
> > > is case
> > > sensitive.
> > > 2> Look in your index via the  admin page
> and look at
> > > your actual
> > > fields as reported there. Are  they really
> what you expect?
> > > 3> Try your query with 
> &debugQuery=on. Is what you
> > > get back
> > > what you  expect?
> > > 4> Sometimes your browser cache will fool you,
> try  the
> > > force-refresh
> > > combination on your browser.
> > >
> > >  There's no magic here, nothing special or
> different about
> > > DIH
> > >  imported data than any other sort. So it's
> almost
> > > certainly
> > > some  innocent-seeming change that's not,
> typo, incorrect
> > > assumption,  etc.
> > >
> > > If none of that works, you need to post your
> schema  changes
> > > and
> > > your query results (with
> &debugQuery=on).  Particularly,

Well, found the problem, us of course.

We were using string instead of text for the field type in the schema config file. So it wasn't tokenizing words or doing other 'search by word' enabling preprocessing before storing the document in the index. We could have only found whole sentences.

Now it works! But, now the long road to tuning it to find what we WANT it to find . . . begins.

That and getting what we want out of geospatial. We're just starting on that.
Dennis Gearon

Signature Warning
----------------
It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'

EARTH has a Right To Life,
  otherwise we all die.
--- On Sun, 10/10/10, Erick Erickson <[EMAIL PROTECTED]> wrote:

> From: Erick Erickson <[EMAIL PROTECTED]>
> Subject: Re: Records from DIH not easily queried for
> To: [EMAIL PROTECTED]
> Date: Sunday, October 10, 2010, 8:11 AM
> The phrase that jumps out is "with
> fields slightly modified". I'm
> guessing that your modifications are off by a little.
> Here's what
> I'd check first:
> 1> check the case. Sometimes the DB <-> field link
> is case
> sensitive.
> 2> Look in your index via the admin page and look at
> your actual
> fields as reported there. Are they really what you expect?
> 3> Try your query with &debugQuery=on. Is what you
> get back
> what you expect?
> 4> Sometimes your browser cache will fool you, try the
> force-refresh
> combination on your browser.
>
> There's no magic here, nothing special or different about
> DIH
> imported data than any other sort. So it's almost
> certainly
> some innocent-seeming change that's not, typo, incorrect
> assumption, etc.
>
> If none of that works, you need to post your schema changes
> and
> your query results (with &debugQuery=on). Particularly,
> post
> the fieldType definitions as well as your field
> definitions...
>
> Best
> Erick
>
> On Sun, Oct 10, 2010 at 10:55 AM, Dennis Gearon <[EMAIL PROTECTED]>wrote:
>
> > With a brand new setup, per the demo/tutorial, with
> fields slightly changed
> > in the config and data, posting XML records results in
> a simple qiery being
> > able to find records.
> >
> >
> > But records imported via a plain jane DIH request can
> only be found using
> > 'q=*:*' queries.
> >
> > There's no filtering, tokenizing, blah blah. It's the
> factory settings. The
> > installation is as new at this as we are :-)
> >
> > Anyone have any ideas why we can't query for DIH
> handled records? Do they
> > have some magic juju done to them that XML Posts
> don't, or visa versa?
> >
> > Dennis Gearon
> >
> > Signature Warning
> > ----------------
> > It is always a good idea to learn from your own
> mistakes. It is usually a
> > better idea to learn from others’ mistakes, so you
> do not have to make them
> > yourself. from '
> > http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036'
> >
> > EARTH has a Right To Life,
> >  otherwise we all die.
> >
>