I have 128 tables of mysql 5.x and each table have 3,5000 rows.
When I start dataimport(indexing) in Solr, it takes 5 minutes for one
table.
But When Solr indexs 20th table, it takes around 10 minutes for one table.
And then When it indexs 40th table, it takes around 20 minutes for one
table.

Solr has some performance problem for too many documents?
Should I set some configuration?

You wrote "3,5000", but is that 35 hundred (3,500) or 35 thousand (35,000)??

Your numbers seem far worse than what many people typically see with Solr
and DIH.

Is the database running on the same machine?

Check the Solr log file to see if some errors (or warnings) might be
occurring frequently.

Check the log for the first table from when it starts to when it ends. How
often is it committing (according to the log)? Does there seem to be any odd
activity during that period?

-- Jack Krupansky

-----Original Message-----
From: Jihyun Suh
Sent: Tuesday, June 05, 2012 9:25 PM
To: [EMAIL PROTECTED] ; [EMAIL PROTECTED]
Subject: Solr, I have perfomance problem for indexing.

I have 128 tables of mysql 5.x and each table have 3,5000 rows.
When I start dataimport(indexing) in Solr, it takes 5 minutes for one
table.
But When Solr indexs 20th table, it takes around 10 minutes for one table.
And then When it indexs 40th table, it takes around 20 minutes for one
table.

Solr has some performance problem for too many documents?
Should I set some configuration?

Which Solr do you run?

On Tue, Jun 5, 2012 at 8:02 PM, Jack Krupansky <[EMAIL PROTECTED]> wrote:
> You wrote "3,5000", but is that 35 hundred (3,500) or 35 thousand (35,000)??
>
> Your numbers seem far worse than what many people typically see with Solr
> and DIH.
>
> Is the database running on the same machine?
>
> Check the Solr log file to see if some errors (or warnings) might be
> occurring frequently.
>
> Check the log for the first table from when it starts to when it ends. How
> often is it committing (according to the log)? Does there seem to be any odd
> activity during that period?
>
> -- Jack Krupansky
>
> -----Original Message----- From: Jihyun Suh
> Sent: Tuesday, June 05, 2012 9:25 PM
> To: [EMAIL PROTECTED] ; [EMAIL PROTECTED]
> Subject: Solr, I have perfomance problem for indexing.
>
>
> I have 128 tables of mysql 5.x and each table have 3,5000 rows.
> When I start dataimport(indexing) in Solr, it takes 5 minutes for one
> table.
> But When Solr indexs 20th table, it takes around 10 minutes for one table.
> And then When it indexs 40th table, it takes around 20 minutes for one
> table.
>
> Solr has some performance problem for too many documents?
> Should I set some configuration?

--
Lance Norskog
[EMAIL PROTECTED]

Each table has 35,000 rows. (35 thousands).
I will check the log for each step of indexing.

I run Solr 3.5.
2012/6/6 Jihyun Suh <[EMAIL PROTECTED]>

> I have 128 tables of mysql 5.x and each table have 3,5000 rows.
> When I start dataimport(indexing) in Solr, it takes 5 minutes for one
> table.
> But When Solr indexs 20th table, it takes around 10 minutes for one table.
> And then When it indexs 40th table, it takes around 20 minutes for one
> table.
>
> Solr has some performance problem for too many documents?
> Should I set some configuration?
>
>

what is your db schema ? do you need to import all the schema ? (128
joined tables ??)
or are the tables all independant ? (if so dump them out and import
them in using csv)

cheers lee c

On 7 June 2012 02:32, Jihyun Suh <[EMAIL PROTECTED]> wrote:
> Each table has 35,000 rows. (35 thousands).
> I will check the log for each step of indexing.
>
> I run Solr 3.5.
>
>
> 2012/6/6 Jihyun Suh <[EMAIL PROTECTED]>
>
>> I have 128 tables of mysql 5.x and each table have 3,5000 rows.
>> When I start dataimport(indexing) in Solr, it takes 5 minutes for one
>> table.
>> But When Solr indexs 20th table, it takes around 10 minutes for one table.
>> And then When it indexs 40th table, it takes around 20 minutes for one
>> table.
>>
>> Solr has some performance problem for too many documents?
>> Should I set some configuration?
>>
>>

You haven't really told us much about what you're doing here. As Lee
hints, we don't know much about the details of *how* you are doing this.

But unless you're doing something odd, Solr shouldn't be the bottleneck
here. Often when a database import is slow, the problem is in the data-
acquisition bit. That is, your SQL query for some reason gets
slow. That said, with DIH it can be hard to know exactly.

You might want to consider using SolrJ instead of DIH. We've found that
as the import process gets more complex, using SolrJ is often easier. See:
http://www.lucidimagination.com/blog/2012/02/14/indexing-with-solrj/
Best
Erick

On Thu, Jun 7, 2012 at 5:26 AM, Lee Carroll
<[EMAIL PROTECTED]> wrote:
> what is your db schema ? do you need to import all the schema ? (128
> joined tables ??)
> or are the tables all independant ? (if so dump them out and import
> them in using csv)
>
> cheers lee c
>
> On 7 June 2012 02:32, Jihyun Suh <[EMAIL PROTECTED]> wrote:
>> Each table has 35,000 rows. (35 thousands).
>> I will check the log for each step of indexing.
>>
>> I run Solr 3.5.
>>
>>
>> 2012/6/6 Jihyun Suh <[EMAIL PROTECTED]>
>>
>>> I have 128 tables of mysql 5.x and each table have 3,5000 rows.
>>> When I start dataimport(indexing) in Solr, it takes 5 minutes for one
>>> table.
>>> But When Solr indexs 20th table, it takes around 10 minutes for one table.
>>> And then When it indexs 40th table, it takes around 20 minutes for one
>>> table.
>>>
>>> Solr has some performance problem for too many documents?
>>> Should I set some configuration?
>>>
>>>