Hi, we are currently evaluating Solr and have been browsing the archives for
one particular issue but can¹t seem to find the answer, so please forgive me
if I¹m asking a repetitive question.  We like the idea of having multiple
slave servers serving up queries and a master performing updates.  However
the the issue for us there is no redundancy for the master.  So a couple of
questions:

1. Can there be multiple masters (or update servers) sharing the same index
files, performing updates at the same time (ie. Hosting the index on a SAN)?

2. Is there a recommended architecture utilizing a SAN.   (For example 2
slaves and 2 masters sharing a SAN).  We current don¹t have that many
records ­ prob about a million and growing.  We are mainly concerned about
redundancy, then performance.

Thanks
-Rakesh

As far as I know only one machine can write to an index at a time.  
More than that and I got corrupted indexes.

Thanks!

Matthew Runo
Software Developer
Zappos.com
702.943.7833

On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:

>
> Hi, we are currently evaluating Solr and have been browsing the  
> archives for
> one particular issue but can’t seem to find the answer, so please  
> forgive me
> if I’m asking a repetitive question.  We like the idea of having  
> multiple
> slave servers serving up queries and a master performing updates.  
> However
> the the issue for us there is no redundancy for the master.  So a  
> couple of
> questions:
>
> 1. Can there be multiple masters (or update servers) sharing the  
> same index
> files, performing updates at the same time (ie. Hosting the index on  
> a SAN)?
>
> 2. Is there a recommended architecture utilizing a SAN.   (For  
> example 2
> slaves and 2 masters sharing a SAN).  We current don’t have that many
> records – prob about a million and growing.  We are mainly concerned  
> about
> redundancy, then performance.
>
> Thanks
> -Rakesh
>
>
>

Thanks for the input, much appreciated.
-Rakesh

On 7/29/08 12:18 PM, "Matthew Runo" <[EMAIL PROTECTED]> wrote:

> As far as I know only one machine can write to an index at a time.
> More than that and I got corrupted indexes.
>
> Thanks!
>
> Matthew Runo
> Software Developer
> Zappos.com
> 702.943.7833
>
> On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
>
>>
>> Hi, we are currently evaluating Solr and have been browsing the
>> archives for
>> one particular issue but can¹t seem to find the answer, so please
>> forgive me
>> if I¹m asking a repetitive question.  We like the idea of having
>> multiple
>> slave servers serving up queries and a master performing updates.
>> However
>> the the issue for us there is no redundancy for the master.  So a
>> couple of
>> questions:
>>
>> 1. Can there be multiple masters (or update servers) sharing the
>> same index
>> files, performing updates at the same time (ie. Hosting the index on
>> a SAN)?
>>
>> 2. Is there a recommended architecture utilizing a SAN.   (For
>> example 2
>> slaves and 2 masters sharing a SAN).  We current don¹t have that many
>> records ­ prob about a million and growing.  We are mainly concerned
>> about
>> redundancy, then performance.
>>
>> Thanks
>> -Rakesh
>>
>>
>>
>
>

You could implement a script that woiuld control which master server is
indexing and put them behind something like a NAT.

I use that that control my master redundancy.

2008/7/29 Rakesh Godhani <[EMAIL PROTECTED]>

> Thanks for the input, much appreciated.
> -Rakesh
>
>
>
> On 7/29/08 12:18 PM, "Matthew Runo" <[EMAIL PROTECTED]> wrote:
>
> > As far as I know only one machine can write to an index at a time.
> > More than that and I got corrupted indexes.
> >
> > Thanks!
> >
> > Matthew Runo
> > Software Developer
> > Zappos.com
> > 702.943.7833
> >
> > On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
> >
> >>
> >> Hi, we are currently evaluating Solr and have been browsing the
> >> archives for
> >> one particular issue but can¹t seem to find the answer, so please
> >> forgive me
> >> if I¹m asking a repetitive question.  We like the idea of having
> >> multiple
> >> slave servers serving up queries and a master performing updates.
> >> However
> >> the the issue for us there is no redundancy for the master.  So a
> >> couple of
> >> questions:
> >>
> >> 1. Can there be multiple masters (or update servers) sharing the
> >> same index
> >> files, performing updates at the same time (ie. Hosting the index on
> >> a SAN)?
> >>
> >> 2. Is there a recommended architecture utilizing a SAN.   (For
> >> example 2
> >> slaves and 2 masters sharing a SAN).  We current don¹t have that many
> >> records ­ prob about a million and growing.  We are mainly concerned
> >> about
> >> redundancy, then performance.
> >>
> >> Thanks
> >> -Rakesh
> >>
> >>
> >>
> >
> >
>
>
>
--
Alexander Ramos Jardim

After Matthew's comment I was thinking about putting them both behind a load
balancer, with the LB directing all traffic to one until it fails and then
kick over to the other one.

In your architectures I'm guessing the masters share the same physical
index, but do the slaves share the same index as the masters or do you use
rsync or some other mechanism to distribute copies.

Thanks
-Rakesh
On 7/29/08 5:07 PM, "Alexander Ramos Jardim"
<[EMAIL PROTECTED]> wrote:

> You could implement a script that woiuld control which master server is
> indexing and put them behind something like a NAT.
>
> I use that that control my master redundancy.
>
> 2008/7/29 Rakesh Godhani <[EMAIL PROTECTED]>
>
>> Thanks for the input, much appreciated.
>> -Rakesh
>>
>>
>>
>> On 7/29/08 12:18 PM, "Matthew Runo" <[EMAIL PROTECTED]> wrote:
>>
>>> As far as I know only one machine can write to an index at a time.
>>> More than that and I got corrupted indexes.
>>>
>>> Thanks!
>>>
>>> Matthew Runo
>>> Software Developer
>>> Zappos.com
>>> 702.943.7833
>>>
>>> On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
>>>
>>>>
>>>> Hi, we are currently evaluating Solr and have been browsing the
>>>> archives for
>>>> one particular issue but can¹t seem to find the answer, so please
>>>> forgive me
>>>> if I¹m asking a repetitive question.  We like the idea of having
>>>> multiple
>>>> slave servers serving up queries and a master performing updates.
>>>> However
>>>> the the issue for us there is no redundancy for the master.  So a
>>>> couple of
>>>> questions:
>>>>
>>>> 1. Can there be multiple masters (or update servers) sharing the
>>>> same index
>>>> files, performing updates at the same time (ie. Hosting the index on
>>>> a SAN)?
>>>>
>>>> 2. Is there a recommended architecture utilizing a SAN.   (For
>>>> example 2
>>>> slaves and 2 masters sharing a SAN).  We current don¹t have that many
>>>> records ­ prob about a million and growing.  We are mainly concerned
>>>> about
>>>> redundancy, then performance.
>>>>
>>>> Thanks
>>>> -Rakesh
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>

I use the snapshooter with rsync

2008/7/29 Rakesh Godhani <[EMAIL PROTECTED]>

> After Matthew's comment I was thinking about putting them both behind a
> load
> balancer, with the LB directing all traffic to one until it fails and then
> kick over to the other one.
>
> In your architectures I'm guessing the masters share the same physical
> index, but do the slaves share the same index as the masters or do you use
> rsync or some other mechanism to distribute copies.
>
> Thanks
> -Rakesh
>
>
>
>
> On 7/29/08 5:07 PM, "Alexander Ramos Jardim"
> <[EMAIL PROTECTED]> wrote:
>
> > You could implement a script that woiuld control which master server is
> > indexing and put them behind something like a NAT.
> >
> > I use that that control my master redundancy.
> >
> > 2008/7/29 Rakesh Godhani <[EMAIL PROTECTED]>
> >
> >> Thanks for the input, much appreciated.
> >> -Rakesh
> >>
> >>
> >>
> >> On 7/29/08 12:18 PM, "Matthew Runo" <[EMAIL PROTECTED]> wrote:
> >>
> >>> As far as I know only one machine can write to an index at a time.
> >>> More than that and I got corrupted indexes.
> >>>
> >>> Thanks!
> >>>
> >>> Matthew Runo
> >>> Software Developer
> >>> Zappos.com
> >>> 702.943.7833
> >>>
> >>> On Jul 28, 2008, at 11:25 AM, Rakesh Godhani wrote:
> >>>
> >>>>
> >>>> Hi, we are currently evaluating Solr and have been browsing the
> >>>> archives for
> >>>> one particular issue but can¹t seem to find the answer, so please
> >>>> forgive me
> >>>> if I¹m asking a repetitive question.  We like the idea of having
> >>>> multiple
> >>>> slave servers serving up queries and a master performing updates.
> >>>> However
> >>>> the the issue for us there is no redundancy for the master.  So a
> >>>> couple of
> >>>> questions:
> >>>>
> >>>> 1. Can there be multiple masters (or update servers) sharing the
> >>>> same index
> >>>> files, performing updates at the same time (ie. Hosting the index on
> >>>> a SAN)?
> >>>>
> >>>> 2. Is there a recommended architecture utilizing a SAN.   (For
> >>>> example 2
> >>>> slaves and 2 masters sharing a SAN).  We current don¹t have that many
> >>>> records ­ prob about a million and growing.  We are mainly concerned
> >>>> about
> >>>> redundancy, then performance.
> >>>>
> >>>> Thanks
> >>>> -Rakesh
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >>
> >
>
>
>
--
Alexander Ramos Jardim

Hi Rakesh,

Solr ships with a set of collection distribution scripts (rsync based) which
can be used to setup master-slave replication. I believe most people use
them for transferring index from master to slaves and use separate local
filesystems on their servers.

Take a look at http://wiki.apache.org/solr/CollectionDistribution

On Wed, Jul 30, 2008 at 3:10 AM, Rakesh Godhani <[EMAIL PROTECTED]> wrote:

> After Matthew's comment I was thinking about putting them both behind a
> load
> balancer, with the LB directing all traffic to one until it fails and then
> kick over to the other one.
If both masters use the same index on the SAN then this may not work because
if a failure happens during indexing then the other master will try to open
an IndexWriter on the shared index and may fail with LockObtainedException
because the other master will have already obtained a lock on it. Note that
lock is only obtained by masters and not by slaves. Probably someone who
knows more about this part can confirm.

>
>
> In your architectures I'm guessing the masters share the same physical
> index, but do the slaves share the same index as the masters or do you use
> rsync or some other mechanism to distribute copies.
>

If you want to use a SAN

--
Regards,
Shalin Shekhar Mangar.

SAN (or NAS if you can live with its performance) is one way.  Oh course, the SAN would have to have redundant hw, too - no use in having multiple and redundant disks if you only have 1 controller, for example.  Another options (that I have not tried) is DRBD.  Whatever the case, you need to have only 1 active master.  Have a look at one of the comments towards the end of SOLR-561 for a decent description of how this could be set up.  ZooKeeper might help with active/standby coordination, too, I imagine.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----
> From: Rakesh Godhani <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Monday, July 28, 2008 2:25:00 PM
> Subject: Multiple Update servers
>
>
> Hi, we are currently evaluating Solr and have been browsing the archives for
> one particular issue but can¹t seem to find the answer, so please forgive me
> if I¹m asking a repetitive question.  We like the idea of having multiple
> slave servers serving up queries and a master performing updates.  However
> the the issue for us there is no redundancy for the master.  So a couple of
> questions:
>
> 1. Can there be multiple masters (or update servers) sharing the same index
> files, performing updates at the same time (ie. Hosting the index on a SAN)?
>
> 2. Is there a recommended architecture utilizing a SAN.   (For example 2
> slaves and 2 masters sharing a SAN).  We current don¹t have that many
> records ­ prob about a million and growing.  We are mainly concerned about
> redundancy, then performance.
>
> Thanks
> -Rakesh