I used RSYNC before, and 20Gb replica took less than an hour (20-40
minutes); now, HTTP, and it takes 5-6 hours...
Admin screen shows 952Kb/sec average speed; 100Mbps network, full-duplex; I
am using Tomcat Native for APR. 10x times slow...
-Fuad
http://www.tokenizer.ca

On Sat, Jan 2, 2010 at 5:48 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
> I used RSYNC before, and 20Gb replica took less than an hour (20-40
> minutes); now, HTTP, and it takes 5-6 hours...
> Admin screen shows 952Kb/sec average speed; 100Mbps network, full-duplex; I
> am using Tomcat Native for APR. 10x times slow...

Hmmm, did you try w/o native APR?

-Yonik
http://www.lucidimagination.com

Hi Yonik,

I tried... I set APR to improve performance... server is slow while replica;
but "top" shows only 1% of I/O wait... it is probably environment specific;
but the same happened in my home-based network, rsync was 10 times faster...
I don't know details of HTTP-replica, it could be base64 or something like
that; RAM-buffer, flush to disk, etc.
-Fuad
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Yonik
> Seeley
> Sent: January-02-10 5:52 PM
> To: [EMAIL PROTECTED]
> Subject: Re: SOLR: Replication
>
> On Sat, Jan 2, 2010 at 5:48 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
> > I used RSYNC before, and 20Gb replica took less than an hour (20-40
> > minutes); now, HTTP, and it takes 5-6 hours...
> > Admin screen shows 952Kb/sec average speed; 100Mbps network, full-
> duplex; I
> > am using Tomcat Native for APR. 10x times slow...
>
> Hmmm, did you try w/o native APR?
>
> -Yonik
> http://www.lucidimagination.com

On Sat, Jan 2, 2010 at 11:35 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
> I tried... I set APR to improve performance... server is slow while replica;
> but "top" shows only 1% of I/O wait... it is probably environment specific;

So you're saying that stock tomcat (non-native APR) was also 10 times slower?

> but the same happened in my home-based network, rsync was 10 times faster...
> I don't know details of HTTP-replica, it could be base64 or something like
> that; RAM-buffer, flush to disk, etc.

The HTTP replication is using binary.
If you look here, it was benchmarked to be nearly as fast as rsync:
http://wiki.apache.org/solr/SolrReplication

It does do a fsync to make sure that the files are on disk after
downloading, but that shouldn't make too much difference.

-Yonik
http://www.lucidimagination.com

Thank you Yonik, excellent WIKI! I'll try without APR, I believe it's
environmental issue; 100Mbps switched should do 10 times faster (current
replica speed is 1Mbytes/sec)
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Yonik
> Seeley
> Sent: January-03-10 10:03 AM
> To: [EMAIL PROTECTED]
> Subject: Re: SOLR: Replication
>
> On Sat, Jan 2, 2010 at 11:35 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
> > I tried... I set APR to improve performance... server is slow while
> replica;
> > but "top" shows only 1% of I/O wait... it is probably environment
> specific;
>
> So you're saying that stock tomcat (non-native APR) was also 10 times
> slower?
>
> > but the same happened in my home-based network, rsync was 10 times
> faster...
> > I don't know details of HTTP-replica, it could be base64 or something
> like
> > that; RAM-buffer, flush to disk, etc.
>
> The HTTP replication is using binary.
> If you look here, it was benchmarked to be nearly as fast as rsync:
> http://wiki.apache.org/solr/SolrReplication
>
> It does do a fsync to make sure that the files are on disk after
> downloading, but that shouldn't make too much difference.
>
> -Yonik
> http://www.lucidimagination.com

Related to the difference between rsync and native Solr replication -
we are seeing issues with Solr 1.4 where search queries that come in
during a replication request hang for excessive amount of time (up to
100's of seconds for a result normally that takes ~50 ms).

We are replicating pretty often (every 90 sec for multiple cores to
one slave server), but still did not think that replication would make
the master server unable to handle search requests.  Is there some
configuration option we are missing which would handle this situation
better?

Thanks,

Peter

On Sun, Jan 3, 2010 at 11:27 AM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
> Thank you Yonik, excellent WIKI! I'll try without APR, I believe it's
> environmental issue; 100Mbps switched should do 10 times faster (current
> replica speed is 1Mbytes/sec)
>
>
>> -----Original Message-----
>> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Yonik
>> Seeley
>> Sent: January-03-10 10:03 AM
>> To: [EMAIL PROTECTED]
>> Subject: Re: SOLR: Replication
>>
>> On Sat, Jan 2, 2010 at 11:35 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:
>> > I tried... I set APR to improve performance... server is slow while
>> replica;
>> > but "top" shows only 1% of I/O wait... it is probably environment
>> specific;
>>
>> So you're saying that stock tomcat (non-native APR) was also 10 times
>> slower?
>>
>> > but the same happened in my home-based network, rsync was 10 times
>> faster...
>> > I don't know details of HTTP-replica, it could be base64 or something
>> like
>> > that; RAM-buffer, flush to disk, etc.
>>
>> The HTTP replication is using binary.
>> If you look here, it was benchmarked to be nearly as fast as rsync:
>> http://wiki.apache.org/solr/SolrReplication
>>
>> It does do a fsync to make sure that the files are on disk after
>> downloading, but that shouldn't make too much difference.
>>
>> -Yonik
>> http://www.lucidimagination.com
>
>
>

--
Peter M. Wolanin, Ph.D.
Momentum Specialist,  Acquia. Inc.
[EMAIL PROTECTED]

On Sun, Jan 3, 2010 at 2:55 PM, Peter Wolanin <[EMAIL PROTECTED]> wrote:
> Related to the difference between rsync and native Solr replication -
> we are seeing issues with Solr 1.4 where search queries that come in
> during a replication request hang for excessive amount of time (up to
> 100's of seconds for a result normally that takes ~50 ms).
>
> We are replicating pretty often (every 90 sec for multiple cores to
> one slave server), but still did not think that replication would make
> the master server unable to handle search requests.  Is there some
> configuration option we are missing which would handle this situation
> better?

Hmmm, any other clues about what's happening during this time?
If it's not a bug, it could simply be that reading a large index to
serve it to a slave could throw out the important parts of the OS
cache that caused searches to be faster.

If it is a bug, well then we certainly want to get to the bottom of it!

-Yonik
http://www.lucidimagination.com

We are looking for a call back to know when replication has finished after
we force a replication using
http://slave_host:port/solr/replication?command=fetchindex. What is the best
way to go about doing this?  We are thinking of forcing the replication then
pulling the command=details page of the slaves to compare its version to
master.

Also any issues that might be involved with this, for instance if there is a
replication going on when we try to force one what happens?