Hi,

How exactly does SolrCloud handle split brain situations?

Imagine a cluster of 10 nodes.
Imagine 3 of them being connected to the network by some switch and imagine the out port of this switch dies.
When that happens, these 3 nodes will be disconnected from the other 7 nodes and we'll have 2 clusters, one with 3 nodes and one with 7 nodes and we'll have a split brain situation.  
Imagine we had 3 ZK nodes in the original 10-node cluster, 2 of which are connected to the dead switch and are thus aware only of the 3 node cluster now, and 1 ZK instance which is on a different switch and is thus aware only of the 7 node cluster.

At this point how exactly does ZK make SolrCloud immune to split brain?
Does LBHttpSolrServer play a key role here? (I see LBHttpSolrServer mentioned only once on http://wiki.apache.org/solr/SolrCloud and with a question mark next to it)
Thanks,
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm

On 6/15/2012 12:49 PM, Otis Gospodnetic wrote:
> Hi,
>
> How exactly does SolrCloud handle split brain situations?
>
> Imagine a cluster of 10 nodes.
> Imagine 3 of them being connected to the network by some switch and imagine the out port of this switch dies.
> When that happens, these 3 nodes will be disconnected from the other 7 nodes and we'll have 2 clusters, one with 3 nodes and one with 7 nodes and we'll have a split brain situation.  
> Imagine we had 3 ZK nodes in the original 10-node cluster, 2 of which are connected to the dead switch and are thus aware only of the 3 node cluster now, and 1 ZK instance which is on a different switch and is thus aware only of the 7 node cluster.
>
> At this point how exactly does ZK make SolrCloud immune to split brain?

A quorum of N/2+1 nodes is required to operate (that's also the reason you need at least 3 to begin with)

Zookeeper avoids split brain using Paxos (or something very like it - I can't remember if they extended it or modified and/or what they call it).

So you will only ever see one Zookeeper cluster - the smaller partition will be down. There is a proof for Paxos if I remember right.

Zookeeper then acts as the system of record for Solr. Solr won't auto form its own new little clusters - *the* cluster is modeled in Zookeeper and that's the cluster. So Solr does not find it self organizing new mini clusters on partition splits.

When we lose our connection to Zookeeper, update requests are no longer accepted, because we may have a stale cluster view and not know it for a long period of time.
On Jun 15, 2012, at 12:49 PM, Otis Gospodnetic wrote:

> Hi,
>
> How exactly does SolrCloud handle split brain situations?
>
> Imagine a cluster of 10 nodes.
> Imagine 3 of them being connected to the network by some switch and imagine the out port of this switch dies.
> When that happens, these 3 nodes will be disconnected from the other 7 nodes and we'll have 2 clusters, one with 3 nodes and one with 7 nodes and we'll have a split brain situation.  
> Imagine we had 3 ZK nodes in the original 10-node cluster, 2 of which are connected to the dead switch and are thus aware only of the 3 node cluster now, and 1 ZK instance which is on a different switch and is thus aware only of the 7 node cluster.
>
> At this point how exactly does ZK make SolrCloud immune to split brain?
>
>
> Does LBHttpSolrServer play a key role here? (I see LBHttpSolrServer mentioned only once on http://wiki.apache.org/solr/SolrCloud and with a question mark next to it)
>
>
> Thanks,
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm

- Mark Miller
lucidimagination.com

Hi,
 
> Zookeeper avoids split brain using Paxos (or something very like it - I 

> can't remember if they extended it or modified and/or what they call it).
>
> So you will only ever see one Zookeeper cluster - the smaller partition will be
> down. There is a proof for Paxos if I remember right.
>
> Zookeeper then acts as the system of record for Solr. Solr won't auto form
> its own new little clusters - *the* cluster is modeled in Zookeeper and
> that's the cluster. So Solr does not find it self organizing new mini
> clusters on partition splits.
>
> When we lose our connection to Zookeeper, update requests are no longer
> accepted, because we may have a stale cluster view and not know it for a long
> period of time.
Does this work even when outside clients (apps for indexing or searching) send their requests directly to individual nodes?
Let's use the example from my email where we end up with 2 groups of nodes: 7-node group with 2 ZK nodes on the same network and 3-node group with 1 ZK node on the same network.

If a client sends a request to a node in the 7-node group what happens?
And if a client sends a request to a node in the 3-node group what happens?

Yury wrote:
> A quorum of N/2+1 nodes is required to operate (that's also the reason you need at least 3 to begin with)

N=3 (ZK nodes), right?
So in that case we need at least 3/2+1 => 2.5 ZK nodes to operate.  So in my example neither the 7-node group nor the 3-node group will operate (does that mean request rejection or something else?) because neither sees 2.5 ZK nodes?

Thanks,
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 
> On Jun 15, 2012, at 12:49 PM, Otis Gospodnetic wrote:
>
>>  Hi,
>>
>>  How exactly does SolrCloud handle split brain situations?
>>
>>  Imagine a cluster of 10 nodes.
>>  Imagine 3 of them being connected to the network by some switch and imagine
> the out port of this switch dies.
>>  When that happens, these 3 nodes will be disconnected from the other 7
> nodes and we'll have 2 clusters, one with 3 nodes and one with 7 nodes and
> we'll have a split brain situation. 
>>  Imagine we had 3 ZK nodes in the original 10-node cluster, 2 of which are
> connected to the dead switch and are thus aware only of the 3 node cluster now,
> and 1 ZK instance which is on a different switch and is thus aware only of the 7
> node cluster.
>>
>>  At this point how exactly does ZK make SolrCloud immune to split brain?
>>
>>
>>  Does LBHttpSolrServer play a key role here? (I see LBHttpSolrServer
> mentioned only once on http://wiki.apache.org/solr/SolrCloud and with a question
> mark next to it)
>>
>>
>>  Thanks,
>>  Otis
>>  ----
>>  Performance Monitoring for Solr / ElasticSearch / HBase -
> http://sematext.com/spm
>
> - Mark Miller
> lucidimagination.com
>

On Jun 15, 2012, at 1:44 PM, Otis Gospodnetic wrote:

> Does this work even when outside clients (apps for indexing or searching) send their requests directly to individual nodes?
> Let's use the example from my email where we end up with 2 groups of nodes: 7-node group with 2 ZK nodes on the same network and 3-node group with 1 ZK node on the same network.

The 3-node group with 1 ZK would not have a functioning zk - so it would stop accepting updates. If it could serve a complete view of the index, it would though, for searches.

The 7-node group would have a working ZK it could talk to, and it would continue to accept updates as long as a node for a shard for that hash range is up. It would also of course serve searches.

In this case, hitting a box in the 3-node group for searches would start becoming stale. A smart client would no longer hit those boxes though.

If you have a 'dumb' client or load balancer, then yes - you would have to remove the bad nodes from rotation.

We could improve this or make the behavior configurable. At least initially though, we figured it was better if we kept serving searches even when we cannot talk to zookeeper.

>
> If a client sends a request to a node in the 7-node group what happens?
> And if a client sends a request to a node in the 3-node group what happens?

- Mark Miller
lucidimagination.com

Ola,

Thanks Mark!
 
>>  Does this work even when outside clients (apps for indexing or searching) 

> send their requests directly to individual nodes?
>>  Let's use the example from my email where we end up with 2 groups of
> nodes: 7-node group with 2 ZK nodes on the same network and 3-node group with 1
> ZK node on the same network.
>
> The 3-node group with 1 ZK would not have a functioning zk - so it would stop
> accepting updates. If it could serve a complete view of the index, it would
> though, for searches.
So in this case information in this 1 ZK node would tell the 3 Solr nodes whether they have all index data or if some shards are missing (i.e. were only on nodes in the other 7-node group)?
And if nodes figure out they don't have all index data they will reject search requests?  Or will they accept and perform searches, but return responses that tell the client that the searched index was not complete?

> The 7-node group would have a working ZK it could talk to, and it would continue
> to accept updates as long as a node for a shard for that hash range is up. It
> would also of course serve searches.
Right, so if the node for the shard where a doc is supposed to go to is in that 3-node group, then the indexing request will be rejected.  Is this correct?

> In this case, hitting a box in the 3-node group for searches would start
> becoming stale. A smart client would no longer hit those boxes though.
>
> If you have a 'dumb' client or load balancer, then yes - you would have
> to remove the bad nodes from rotation.
Aha, yes and yes.

> We could improve this or make the behavior configurable. At least initially
> though, we figured it was better if we kept serving searches even when we cannot
> talk to zookeeper.
Makes sense.  Do responses carry something to alert the client that "something is rotten in the state of cluster"?

Thanks,

Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 

On Jun 15, 2012, at 2:12 PM, Otis Gospodnetic wrote:

> Makes sense.  Do responses carry something to alert the client that "something is rotten in the state of cluster"?

No, I don't think so - we should probably add that to the header similar to how I assume partial results will work.

Feel free to fire up a JIRA issue for that.

- Mark Miller
lucidimagination.com

Thanks Mark, will open an issue in a bit.

But I think the following is the real meat of the Q about split brain and SolrCloud, especially when it comes to how indexing is handled during split brain:

>>  Does this work even when outside clients (apps for indexing or searching) 

> send their requests directly to individual nodes?
>>  Let's use the example from my email where we end up with 2 groups of 
> nodes: 7-node group with 2 ZK nodes on the same network and 3-node group with 1 
> ZK node on the same network.
> 
> The 3-node group with 1 ZK would not have a functioning zk - so it would stop 
> accepting updates. If it could serve a complete view of the index, it would 
> though, for searches.

So in this case information in this 1 ZK node would tell the 3 Solr nodes whether they have all index data or if some shards are missing (i.e. were only on nodes in the other 7-node group)?
And if nodes figure out they don't have all index data they will reject search requests?  Or will they accept and perform searches, but return responses that tell the client that the searched index was not complete?

> The 7-node group would have a working ZK it could talk to, and it would continue 
> to accept updates as long as a node for a shard for that hash range is up. It 
> would also of course serve searches.

Right, so if the node for the shard where a doc is supposed to go to is in that 3-node group, then the indexing request will be rejected.  Is this correct? 

Otis 
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 

----- Original Message -----
> From: Mark Miller <[EMAIL PROTECTED]>
> To: solr-user <[EMAIL PROTECTED]>
> Cc:
> Sent: Friday, June 15, 2012 2:22 PM
> Subject: Re: SolrCloud and split-brain
>
>
> On Jun 15, 2012, at 2:12 PM, Otis Gospodnetic wrote:
>
>>  Makes sense.  Do responses carry something to alert the client that
> "something is rotten in the state of cluster"?
>
> No, I don't think so - we should probably add that to the header similar to
> how I assume partial results will work.
>
> Feel free to fire up a JIRA issue for that.
>
> - Mark Miller
> lucidimagination.com
>

On Jun 15, 2012, at 3:21 PM, Otis Gospodnetic wrote:

> Thanks Mark, will open an issue in a bit.
>
> But I think the following is the real meat of the Q about split brain and SolrCloud, especially when it comes to how indexing is handled during split brain:
>
>>>   Does this work even when outside clients (apps for indexing or searching)
>
>> send their requests directly to individual nodes?
>>>   Let's use the example from my email where we end up with 2 groups of
>> nodes: 7-node group with 2 ZK nodes on the same network and 3-node group with 1
>> ZK node on the same network.
>>  
>> The 3-node group with 1 ZK would not have a functioning zk - so it would stop
>> accepting updates. If it could serve a complete view of the index, it would
>> though, for searches.
>
> So in this case information in this 1 ZK node would tell the 3 Solr nodes whether they have all index data or if some shards are missing (i.e. were only on nodes in the other 7-node group)?
> And if nodes figure out they don't have all index data they will reject search requests?  Or will they accept and perform searches, but return responses that tell the client that the searched index was not complete?

The 1 ZK node will not function, so the 3 Solr nodes will not accept updates.

If there is one replica for each shard available, search will still work. I don't think partial results has been committed yet for distrib search. In that case, we will put something in the header to indicate a full copy of the index was not available. I think we can also add something in the header if we know we cannot talk to zookeeper to let the client know it could be seeing stale state. SmartClients that talked to zookeeper would see those nodes appear as down in zookeeper and stop trying to talk to them.

>
>> The 7-node group would have a working ZK it could talk to, and it would continue
>> to accept updates as long as a node for a shard for that hash range is up. It
>> would also of course serve searches.
>
> Right, so if the node for the shard where a doc is supposed to go to is in that 3-node group, then the indexing request will be rejected.  Is this correct?

it depends on what is available - but you will need at least one replica for each shard available - eg your partition needs to have one copy of the index - otherwise updates are rejected if there are no nodes hosting a shard of the hash range. So if a replica made it into the larger partition, you will be fine - it will become the leader.

>
>
>
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm
>
>
>
> ----- Original Message -----
>> From: Mark Miller <[EMAIL PROTECTED]>
>> To: solr-user <[EMAIL PROTECTED]>
>> Cc:
>> Sent: Friday, June 15, 2012 2:22 PM
>> Subject: Re: SolrCloud and split-brain
>>
>>
>> On Jun 15, 2012, at 2:12 PM, Otis Gospodnetic wrote:
>>
>>> Makes sense.  Do responses carry something to alert the client that
>> "something is rotten in the state of cluster"?
>>
>> No, I don't think so - we should probably add that to the header similar to
>> how I assume partial results will work.
>>
>> Feel free to fire up a JIRA issue for that.
>>
>> - Mark Miller
>> lucidimagination.com
>>

- Mark Miller
lucidimagination.com

Thanks Mark.

The reason I asked this is because I saw mentions of SolrCloud being resilient to split brain because it uses ZooKeeper.
However, if my half brain understands what split brain is then I think that's not a completely true claim because one can get unlucky and get a SolrCloud cluster partitioned in a way that one or even all partitions reject indexing (and update and deletion) requests if they do not have a complete index.

In my example of a 10-node cluster that gets split into a 7-node and a 3-node partition, if neither partition ends up containing the full index (i.e. at least one copy of each shard) then neither partition will accept updates.

And here is one more Q.
* Imagine a client is adding documents and, for simplicity, imagine SolrCloud routes all these documents to the same shard, call it S.
* Imagine that both the 7-node and the 3-node partition end up with a complete index and thus both accept updates.
* This means that both the 7-node and the 3-node partition have at least one replica of shard S, lets call then S7 and S3.
* Now imagine if the client sending documents for indexing happened to be sending documents to 2 nodes, say in round-robin fashion.
* And imagine that each of these 2 nodes ended up in a different partition.

The client now keeps sending docs to these 2 nodes and both happily take and index documents in their own copies of S.
To the client everything looks normal - all documents are getting indexed.
But S7 and S3 are no longer the same - they contain different documents!

Problem, no?
What happens with somebody fixes the cluster and all nodes are back in the same 10-node cluster?  What happens to S7 and S3?
Wouldn't SolrCloud have to implement bi-directional synchronization to fix things and "unify" S7 and S3?

And if there are updates and deletes involved, things get even messier.... :(

Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 

----- Original Message -----
> From: Mark Miller <[EMAIL PROTECTED]>
> To: solr-user <[EMAIL PROTECTED]>
> Cc:
> Sent: Friday, June 15, 2012 5:07 PM
> Subject: Re: SolrCloud and split-brain
>
>
> On Jun 15, 2012, at 3:21 PM, Otis Gospodnetic wrote:
>
>>  Thanks Mark, will open an issue in a bit.
>>
>>  But I think the following is the real meat of the Q about split brain and
> SolrCloud, especially when it comes to how indexing is handled during split
> brain:
>>
>>>>    Does this work even when outside clients (apps for indexing or
> searching)
>>
>>>  send their requests directly to individual nodes?
>>>>    Let's use the example from my email where we end up with 2
> groups of
>>>  nodes: 7-node group with 2 ZK nodes on the same network and 3-node
> group with 1
>>>  ZK node on the same network.
>>>  
>>>  The 3-node group with 1 ZK would not have a functioning zk - so it
> would stop
>>>  accepting updates. If it could serve a complete view of the index, it
> would
>>>  though, for searches.
>>
>>  So in this case information in this 1 ZK node would tell the 3 Solr nodes
> whether they have all index data or if some shards are missing (i.e. were only
> on nodes in the other 7-node group)?
>>  And if nodes figure out they don't have all index data they will reject
> search requests?  Or will they accept and perform searches, but return responses
> that tell the client that the searched index was not complete?
>
> The 1 ZK node will not function, so the 3 Solr nodes will not accept updates.
>
> If there is one replica for each shard available, search will still work. I
> don't think partial results has been committed yet for distrib search. In
> that case, we will put something in the header to indicate a full copy of the
> index was not available. I think we can also add something in the header if we
> know we cannot talk to zookeeper to let the client know it could be seeing stale
> state. SmartClients that talked to zookeeper would see those nodes appear as

On Sat, Jun 16, 2012 at 5:33 AM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:

> And here is one more Q.
> * Imagine a client is adding documents and, for simplicity, imagine SolrCloud routes all these documents to the same shard, call it S.
> * Imagine that both the 7-node and the 3-node partition end up with a complete index and thus both accept updates.

According to comments from Mark this (having two functioning sides)
is not possible... only one side _can_ continue functioning (taking in
updates). Depending on how the shards are deployed over the nodes a
"side" may still not accept updates (even if that side has a working
zk setup).

> Now imagine if the client sending documents for indexing happened to be sending documents to 2 nodes, say in round-robin fashion.

In my understanding all updates are routed through a shard leader.

--
 Sami Siren

On Jun 15, 2012, at 10:33 PM, Otis Gospodnetic wrote:

> However, if my half brain understands what split brain is then I think that's not a completely true claim because one can get unlucky and get a SolrCloud cluster partitioned in a way that one or even all partitions reject indexing (and update and deletion) requests if they do not have a complete index.

That's not split brain. Split brain means that multiple partitioned clusters think they are *the* cluster and would keep accepting updates. This is a real problem because when you unsplit the cluster, you cannot reconcile conflicting updates easily! In many cases you have to ask the user to resolve the conflict.

Yes, you must have a node to serve a shard in order to index to that shard. You do not need the whole index - but if an update hashes to a shard that has no nodes hosting it, it will fail. If there is no node, the document has no where to live. Some systems do interesting things like buffer those updates to other nodes for a while - we don't plan on anything like that soon. At some point, you can only survive a loss of so many nodes before its time to give up accepting updates in any system. If you need to survive catastrophic loss of nodes, you have to have enough replicas to handle it. Whether those nodes are partitioned off from the cluster or simply die, it's all the same. You can only survive so many node loses, and replicas are your defense.

The lack of split-brain allows your cluster to remain consistent. If you allow split brain you have to use something like vector clocks and handle conflict resolution when the splits rejoin, or you will just have a lot of messed up data. You generally allow split brain when you want to favor write availability in the face of partitions, like Dynamo. But you must have a strategy for rejoining splits (like vector clocks or something) or you can never properly go back to a single, consistent cluster. We favor consistency in the face of partitions rather than write availability. It seemed like the right choice for Solr.

- Mark Miller
lucidimagination.com

Hi Mark,

Thanks.  All that is clear (I think Voldemort does a good job with hinted handoff, which I think Mark is referring to).
The part that I'm not clear about is maybe not SolrCloud-specific, and that is - what exactly prevents the two halves of a cluster that's been split from thinking they are *the* cluster?
Let's say you have a 10-node cluster, say with 10 ZK instances, one instance on each Solr node.
And say 5 of these 10 servers are on switch A and the other 5 are on switch B.
Something happens and switch A and 5 nodes on it get separated from 5 nodes on switch B.
Say that both A and B happen to have complete copies of the index.

What in Solr (or ZK) tells either A or B half that "no, you are not *the* cluster and thou shalt not accept updates"?

I'm guessing this: https://cwiki.apache.org/confluence/display/ZOOKEEPER/FailureScenarios ?

So then the Q becomes: if we have 10 ZK nodes and they split in 5 & 5 nodes, does that mean neither side will have quorum because having 10 ZKs was a bad number of ZKs to have to begin with?

Thanks,
Otis 
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 
----- Original Message -----
> From: Mark Miller <[EMAIL PROTECTED]>
> To: solr-user <[EMAIL PROTECTED]>
> Cc:
> Sent: Monday, June 18, 2012 11:05 AM
> Subject: Re: SolrCloud and split-brain
>
>
> On Jun 15, 2012, at 10:33 PM, Otis Gospodnetic wrote:
>
>>  However, if my half brain understands what split brain is then I think
> that's not a completely true claim because one can get unlucky and get a
> SolrCloud cluster partitioned in a way that one or even all partitions reject
> indexing (and update and deletion) requests if they do not have a complete
> index.
>
> That's not split brain. Split brain means that multiple partitioned clusters
> think they are *the* cluster and would keep accepting updates. This is a real
> problem because when you unsplit the cluster, you cannot reconcile conflicting
> updates easily! In many cases you have to ask the user to resolve the conflict.
>
> Yes, you must have a node to serve a shard in order to index to that shard. You
> do not need the whole index - but if an update hashes to a shard that has no
> nodes hosting it, it will fail. If there is no node, the document has no where
> to live. Some systems do interesting things like buffer those updates to other
> nodes for a while - we don't plan on anything like that soon. At some point,
> you can only survive a loss of so many nodes before its time to give up
> accepting updates in any system. If you need to survive catastrophic loss of
> nodes, you have to have enough replicas to handle it. Whether those nodes are
> partitioned off from the cluster or simply die, it's all the same. You can
> only survive so many node loses, and replicas are your defense.
>
> The lack of split-brain allows your cluster to remain consistent. If you allow
> split brain you have to use something like vector clocks and handle conflict
> resolution when the splits rejoin, or you will just have a lot of messed up
> data. You generally allow split brain when you want to favor write availability
> in the face of partitions, like Dynamo. But you must have a strategy for
> rejoining splits (like vector clocks or something) or you can never properly go
> back to a single, consistent cluster. We favor consistency in the face of
> partitions rather than write availability. It seemed like the right choice for
> Solr.
>
> - Mark Miller
> lucidimagination.com
>

>
> So then the Q becomes: if we have 10 ZK nodes and they split in 5 & 5
> nodes, does that mean neither side will have quorum because having 10 ZKs
> was a bad number of ZKs to have to begin with?
Right - from the ZooKeeper admin guide, under Clustered Setup:

"Because Zookeeper requires a majority, it is best to use an odd number of
machines."

--
- Mark

http://www.lucidimagination.com