Day 44: Leader election, quorum, split brain, fencing
When consensus goes wrong: split brain
A split brain happens when a network partition splits a cluster into two groups, and each group — unable to see the other — separately elects its own leader. Now two nodes both believe they're in charge, both accepting writes, and once the partition heals you have two divergent, conflicting histories.
A concrete example
A 5-node cluster splits into a group of 3 and a group of 2 during a network partition. The group of 3 can still form a majority (3 of 5) and elect a leader safely. The group of 2 cannot (2 of 5 is not a majority) — so it correctly refuses to elect anyone, avoiding split brain. This is exactly why quorum-based systems need an odd number of nodes: it guarantees at most one side of any partition can ever have a majority.
Fencing
Even with correct quorum logic, a former leader might not know yet that it's been deposed (it's still "alive", just partitioned) and could keep trying to act as leader. Fencing is actively preventing a demoted or unreachable node from causing damage — e.g., a fencing token that increases every time a new leader is elected, with downstream systems rejecting any write carrying an older token than the last one seen.
Leader A (term 5) writes with token=5
... network partition, Leader B elected (term 6) ...
Leader B writes with token=6
Leader A (unaware it was deposed) tries to write with token=5
Storage rejects it: "5 < 6, already seen a higher token" — stale write blockedKey terms
- Split brain
- Two or more nodes each believing they are the sole leader after a network partition.
- Quorum
- The minimum number of nodes (a strict majority) required to make a binding decision.
- Fencing
- Mechanisms that prevent a deposed or partitioned node from taking harmful action, e.g. via monotonically increasing tokens.
Why is a 5-node cluster generally preferred over a 4-node cluster for the same fault tolerance goal?
