Hi

I have a index of size 1 Tb.. And I prepared this by setting up a background
script to index records. The index was fine last 2 days, and i have not
disturbed the process. Suddenly when i queried the index i get this
response, where the value of numFound is negative. Can any one say why/how
this occurs and also the cure.

<response>
    <lst name="responseHeader">
        <int name="status">0</int>
        <int name="QTime">0</int>
        <lst name="params">
            <str name="indent">on</str>
            <str name="start">0</str>
            <str name="q">*:*</str>
            <str name="version">2.2</str>
            <str name="rows">10</str>
        </lst>
    </lst>
    <result name="response" numFound="-2125433775" start="0"/>
</response>

Regards
Senthil Kumar M R

--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398.html
Sent from the Solr - User mailing list archive at Nabble.com.

Hi!
Can you please show your hardware parameters, version of Solr, that you're
using and schema.xml file?

thanks.

--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986408.html
Sent from the Solr - User mailing list archive at Nabble.com.

The details are below

Solr : 3.5
Using a Schema file with 53 fields and 8 fields indexed among them.
OS : CentOS 5.4 64 Bit
Java : 1.6.0 64 Bit
Apache Tomcat : 7.0.22

Intel(R) Xeon(R) CPU L5518  @ 2.13GHz (16 Processors)

/dev/mapper/index 5.9T  1.9T  4.0T  33% /Index

Had around 2 Billion Records, when I queried it last time (2 days back)

Do I need to run the checkIndex tool?

Regards
Senthil Kumar M R

--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986421.html
Sent from the Solr - User mailing list archive at Nabble.com.

Hm... Have you any errors in logs? During search, during indexing?

--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986426.html
Sent from the Solr - User mailing list archive at Nabble.com.

Is this for a single-shard or multi-shard index?

There is a 2^31-1 limit for a single Lucene index since document numbers are
"int" (32-bit signed in Java) in Lucene, but with Solr shards you can have a
multiple of that, based on number of shards.

If you are multi-shard, maybe one of the shards grew too large.

-- Jack Krupansky

-----Original Message-----
From: tosenthu
Sent: Monday, May 28, 2012 8:15 AM
To: [EMAIL PROTECTED]
Subject: Negative value in numFound

Hi

I have a index of size 1 Tb.. And I prepared this by setting up a background
script to index records. The index was fine last 2 days, and i have not
disturbed the process. Suddenly when i queried the index i get this
response, where the value of numFound is negative. Can any one say why/how
this occurs and also the cure.

<response>
    <lst name="responseHeader">
        <int name="status">0</int>
        <int name="QTime">0</int>
        <lst name="params">
            <str name="indent">on</str>
            <str name="start">0</str>
            <str name="q">*:*</str>
            <str name="version">2.2</str>
            <str name="rows">10</str>
        </lst>
    </lst>
    <result name="response" numFound="-2125433775" start="0"/>
</response>

Regards
Senthil Kumar M R

--
View this message in context:
http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398.html
Sent from the Solr - User mailing list archive at Nabble.com.

There was an Out Of Memory.. But still the indexing was happening further..

--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986437.html
Sent from the Solr - User mailing list archive at Nabble.com.

In some cases multi-shard architecture might significantly slow down the
search process at this index size...

By the way, how much RAM do you use?

--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986438.html
Sent from the Solr - User mailing list archive at Nabble.com.

Hi

It is a multicore but when i searched the shards query even then i get this
response

<result name="response" numFound="-390662429" start="0">

which is again a negative value.

Might be the total number of records may be > 2147483647 (2^31-1), But is
this limitation documented anywhere. What is the strategy to over come this
situation. Expectation of my application is to have 12 billion records. So
please suggest me a strategy for my situation.

Regards
Senthil Kumar M R
                                    
--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986439.html
Sent from the Solr - User mailing list archive at Nabble.com.

OOM is a problem.

You need more RAM and more machines, and maybe more  shards.

-- Jack Krupansky

-----Original Message-----
From: tosenthu
Sent: Monday, May 28, 2012 11:29 AM
To: [EMAIL PROTECTED]
Subject: Re: Negative value in numFound

There was an Out Of Memory.. But still the indexing was happening further..

--
View this message in context:
http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986437.html
Sent from the Solr - User mailing list archive at Nabble.com.

"numFound="-390662429""

That suggests that you have at least two shards which each have > 2G docs
(2^31-1).

How many shards do you have and how big do you think they should be in terms
of number of documents?

Are you being careful to distribute your update requests between shards so
that no shard grows too large? That gets back to the preceding question.

-- Jack Krupansky

-----Original Message-----
From: tosenthu
Sent: Monday, May 28, 2012 11:34 AM
To: [EMAIL PROTECTED]
Subject: Re: Negative value in numFound

Hi

It is a multicore but when i searched the shards query even then i get this
response

<result name="response" numFound="-390662429" start="0">

which is again a negative value.

Might be the total number of records may be > 2147483647 (2^31-1), But is
this limitation documented anywhere. What is the strategy to over come this
situation. Expectation of my application is to have 12 billion records. So
please suggest me a strategy for my situation.

Regards
Senthil Kumar M R

--
View this message in context:
http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986439.html
Sent from the Solr - User mailing list archive at Nabble.com.

The RAM is about 14.5G. Allocated for Tomcat..

I have now 2 shards. But I was in an impression i can handle it with couple
of Shards. But in this case i need to have shards which can only grow up
2^31-1 records and many such shards to support 12 Billion records.

I will try to have more cores and distribute update between them. Then comes
my next question. Is there a possibility to restrict by any configuration
for a core to reject updates based on the number of records. And is there a
possibility to split a index into 2 or more based on a query.

Any how my network will have 2 SOLR servers to participate in indexing and
search.. Probably i need to have at least 6 cores distributed across these
machines to support 12 Billion Records. What is you say?
--
View this message in context: http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986453.html
Sent from the Solr - User mailing list archive at Nabble.com.

I think 100 million documents is a realistic number for a single shard.
Maybe 250 million depending on your data. But I would say that beyond that
is being unrealistic. In some cases, even 50 million might be too much for a
single shard, depending on the data and query usage. Sure, maybe depending
on your data 2 billion documents might work, but I wouldn't bet on it. And
even if you manage to index 500 million or more documents on a single shard,
memory and performance for production query loads would be questionable.
Query capacity also depends on things like number of faceted fields (i.e.,
the field cache), string field size, number of unique terms in each field,
solr query cache, and highlighting of large fields. Not to mention wanting
to have enough capacity so that the number of documents can grow over time.

As an experiment, index 250 million documents in one shard and see how
typical queries perform, and how much JVM memory you use and still have
available. Make sure to try quite a few queries (using a script), especially
if any fields are faceted or highlighted. Then you can decide whether you
feel comfortable trying a larger shard size or if a smaller size is needed.

-- Jack Krupansky

-----Original Message-----
From: tosenthu
Sent: Monday, May 28, 2012 1:25 PM
To: [EMAIL PROTECTED]
Subject: Re: Negative value in numFound

The RAM is about 14.5G. Allocated for Tomcat..

I have now 2 shards. But I was in an impression i can handle it with couple
of Shards. But in this case i need to have shards which can only grow up
2^31-1 records and many such shards to support 12 Billion records.

I will try to have more cores and distribute update between them. Then comes
my next question. Is there a possibility to restrict by any configuration
for a core to reject updates based on the number of records. And is there a
possibility to split a index into 2 or more based on a query.

Any how my network will have 2 SOLR servers to participate in indexing and
search.. Probably i need to have at least 6 cores distributed across these
machines to support 12 Billion Records. What is you say?
--
View this message in context:
http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986453.html
Sent from the Solr - User mailing list archive at Nabble.com.

You went over the max limit for number of docs.

On Monday, May 28, 2012, tosenthu wrote:

> Hi
>
> I have a index of size 1 Tb.. And I prepared this by setting up a
> background
> script to index records. The index was fine last 2 days, and i have not
> disturbed the process. Suddenly when i queried the index i get this
> response, where the value of numFound is negative. Can any one say why/how
> this occurs and also the cure.
>
> <response>
>    <lst name="responseHeader">
>        <int name="status">0</int>
>        <int name="QTime">0</int>
>        <lst name="params">
>            <str name="indent">on</str>
>            <str name="start">0</str>
>            <str name="q">*:*</str>
>            <str name="version">2.2</str>
>            <str name="rows">10</str>
>        </lst>
>    </lst>
>    <result name="response" numFound="-2125433775" start="0"/>
> </response>
>
> Regards
> Senthil Kumar M R
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Bill Bell
[EMAIL PROTECTED]
cell 720-256-8076

"... is this limitation documented anywhere..."

Kind of, but not very well, at least at the Lucene level.

The Lucene File Formats page says "Lucene uses a Java int to refer to
document numbers, and the index file format uses an Int32 on-disk to store
document numbers. This is a limitation of both the index file format and the
current implementation." It also says that "The first document added to an
index is numbered zero." Since Java Integer.MAX_INT is 2^31-1, that means
the maximum number of documents in a single Lucene (or Solr) index is 2^31.

See:
http://lucene.apache.org/core/3_6_0/fileformats.html

And the Lucene IndexSearcher API uses "int" for document number and number
of documents in index.
See:
http://lucene.apache.org/core/3_6_0/api/core/org/apache/lucene/search/IndexSearcher.html

There is a little discussion of the limit issue here:
https://issues.apache.org/jira/browse/LUCENE-2420

I am not aware of any explicit mention of the single-index Lucene document
limit at the Solr level.

-- Jack Krupansky

-----Original Message-----
From: tosenthu
Sent: Monday, May 28, 2012 11:34 AM
To: [EMAIL PROTECTED]
Subject: Re: Negative value in numFound

Hi

It is a multicore but when i searched the shards query even then i get this
response

<result name="response" numFound="-390662429" start="0">

which is again a negative value.

Might be the total number of records may be > 2147483647 (2^31-1), But is
this limitation documented anywhere. What is the strategy to over come this
situation. Expectation of my application is to have 12 billion records. So
please suggest me a strategy for my situation.

Regards
Senthil Kumar M R

--
View this message in context:
http://lucene.472066.n3.nabble.com/Negative-value-in-numFound-tp3986398p3986439.html
Sent from the Solr - User mailing list archive at Nabble.com.