What Breaks When a Database Becomes a Cluster
A database is easiest to picture when it is one machine.
The application sends SQL.
The database finds the row.
The database changes the row.
The database commits the transaction.
UPDATE orders
SET status = 'shipped'
WHERE id = 100;
Even on one machine, there is a lot happening underneath. The database still has to parse the query, choose a plan, use indexes, manage pages in memory, write logs, handle locks, and recover after crashes. A monolithic database is not simple.
But one important part is simple: ownership.
There is one obvious place to send the request. There is one storage engine responsible for finding the row. There is one database process ordering the writes. There is one local transaction manager deciding whether a transaction committed.
That changes when the database becomes a cluster.
The usual reason is not mysterious. One machine eventually becomes an uncomfortable boundary. The data gets too large. The write traffic grows. A single server failure becomes too expensive. Users are spread across regions. The system needs to keep working while machines fail, restart, or move.
The natural answer is to split the data across machines and keep copies of it.
That sounds like the whole solution.
split the table
replicate the pieces
send traffic to the right place
But those two choices, splitting and replication, are exactly what create the hard parts.
Once the data is split, the database has to route every operation to the right piece.
Once the data is replicated, the database has to decide which copy is allowed to coordinate a write.
Once transactions touch multiple split and replicated pieces, the database has to make several physical operations behave like one logical action.
That is where distributed SQL becomes interesting.
It is not just SQL with more servers behind it. It is the work required to preserve normal database behavior after the old single-machine assumptions stop being true.
CockroachDB is a useful system to study because it keeps the SQL interface while changing the internal shape of the database. The application sees tables, rows, indexes, and transactions. Underneath, CockroachDB works with encoded keys, ranges, replicas, leaseholders, Raft logs, MVCC versions, write intents, and transaction records.
Those pieces are easier to understand if we do not introduce them as a list.
Instead, we can start with the thing that broke.
a table is too large a shape for a cluster
Application code thinks in tables.
orders
users
payments
shipments
That is the right abstraction for writing software. If I want order 100, I should be able to ask for order 100. I should not need to know which machine stores it.
But a cluster cannot manage a table as one giant physical object.
A table may contain billions of rows. Some parts of it may be hot while others are barely touched. One section may need to move to a less busy machine. Another section may need to split because it is growing too quickly. If the database could only move whole tables around, the unit of distribution would be too coarse.
So CockroachDB stores SQL data over a distributed key-value layer.
Rows and index entries are encoded into keys. Those keys live in a sorted keyspace. The keyspace is split into ranges.
The application sees this:
orders table
The cluster manages something closer to this:
encoded SQL keys
→ sorted keyspace
→ ranges
A range is a span of keys. It is small enough to move, split, replicate, and rebalance independently.
A simplified keyspace might look like this:
Range 17:
/Table/orders/000 → /Table/orders/100
Range 18:
/Table/orders/100 → /Table/orders/500
Range 19:
/Table/orders/500 → /Table/orders/900
Now the database has a manageable unit.
It does not need to move the entire orders table. It can move Range 18. It does not need to replicate the entire table as one object. It can replicate each range. It does not need one machine to coordinate the whole table. Different ranges can be coordinated by different nodes.
This is the first transformation.
A table is the shape we use to think about data. A range is the shape the cluster uses to manage data.
But this immediately creates a new problem.
If the application asks for this:
SELECT *
FROM orders
WHERE id = 100;
which machine should receive the work?
On one database server, the answer was obvious. The request was already at the database.
In a distributed database, the request might enter through any node. That node may not store the row. It may not coordinate the range containing the row. It may only be the first node that happened to receive the SQL request.
Splitting the table into ranges solved the distribution problem, but it created the routing problem.
routing starts by turning SQL into a key problem
Suppose a client connects to node 1 and sends:
UPDATE orders
SET status = 'shipped'
WHERE id = 100;
Node 1 acts as the gateway for this request. It accepts the SQL statement, plans it, and coordinates the work needed to execute it.
But node 1 may not own the data.
So the gateway has to translate the logical request into a physical route.
Conceptually, it asks:
what key represents orders(id = 100)?
which range contains that key?
which replicas store that range?
which replica should coordinate this operation?
The first part is the SQL-to-key transformation.
The row orders(id = 100) becomes an encoded key or key span in the underlying key-value layer. The exact encoding is not the important part here. The important part is that the database can map a SQL operation to a span of the sorted keyspace.
Then the database needs to find which range owns that span.
This is where range metadata comes in.
Range metadata is the cluster's map of the keyspace. It describes which ranges exist, what start and end keys they cover, and where their replicas are located.
A simplified range descriptor might look like this:
Range 18
start key: /Table/orders/100
end key: /Table/orders/500
replicas: node 2, node 4, node 7
Now node 1 can see that the key for orders(id = 100) belongs to Range 18, and Range 18 has replicas on nodes 2, 4, and 7.
So the route is no longer:
client → database → row
It is:
client
→ node 1 as SQL gateway
→ map SQL row to key
→ find range containing key
→ find replicas for that range
This already explains why a distributed database needs more than a query planner. It needs a live routing layer.
The planner figures out what data the query needs.
The routing layer figures out where the relevant pieces currently live.
But we still have not answered the most important question.
If Range 18 has three replicas, which one should node 1 send the write to?
replication makes the destination ambiguous
Replicating a range is the obvious way to survive failures.
If Range 18 only lives on node 4 and node 4 dies, that part of the database becomes unavailable. So CockroachDB keeps multiple replicas of a range on different nodes.
Range 18
replica 1 → node 2
replica 2 → node 4
replica 3 → node 7
Now the range can survive a node failure.
But the routing problem has become more subtle.
The gateway knows the range has three replicas. What should it do with a write?
One tempting answer is: send the write to any replica.
That would be simple, but it breaks quickly.
If node 2 accepts one write while node 4 accepts a conflicting write, the replicas can create different histories. Later, when they try to synchronize, there may be no single obvious truth to merge back into.
Another tempting answer is: send the write to every replica and wait for all of them.
That avoids one replica deciding alone, but now one slow or unavailable replica can stop the whole range. If node 7 is down, every write would block even though nodes 2 and 4 are healthy.
So neither obvious solution is good.
Any replica deciding alone risks disagreement.
Every replica needing to agree hurts availability.
The system needs a coordinator for the range, and it needs a rule for how many replicas must accept a write before it is considered safe.
Those two ideas are leaseholders and Raft.
a leaseholder gives the range a coordinator
For each range, one replica currently acts as the leaseholder.
The leaseholder is the replica responsible for coordinating requests for that range.
This does not mean the leaseholder is the only copy of the data. The range still has multiple replicas. The leaseholder is just the replica that currently has authority to coordinate reads and writes for that range.
So we can extend the range descriptor:
Range 18
start key: /Table/orders/100
end key: /Table/orders/500
replicas: node 2, node 4, node 7
leaseholder: node 4
Now the gateway has an answer.
The client sent SQL to node 1.
Node 1 mapped the row to a key.
The key belongs to Range 18.
Range 18 has replicas on nodes 2, 4, and 7.
Node 4 is the leaseholder.
So node 1 routes the write to node 4.
client
→ node 1 as SQL gateway
→ Range 18 lookup
→ node 4 as leaseholder
This is the first reason leaseholders exist: they remove ambiguity from routing.
The gateway does not need to invent a write coordinator for every request. It routes to the current leaseholder for the relevant range.
The second reason is scale.
There is not one global leaseholder for the entire database. Leaseholders are assigned per range.
Range 18 → leaseholder node 4
Range 37 → leaseholder node 8
Range 52 → leaseholder node 2
This spreads coordination across the cluster. If one global leader handled every write, the database would bottleneck around that leader. By assigning authority per range, CockroachDB splits not only the data, but also the coordination responsibility.
Ranges split the data. Leaseholders split the authority.
But a leaseholder still cannot just update itself and return success.
It coordinates the write, but the range is replicated. The other replicas need to agree on the write too.
That is the next problem.
a write has to enter a shared history
Suppose node 4 is the leaseholder for Range 18.
It receives the write:
UPDATE orders
SET status = 'shipped'
WHERE id = 100;
A bad version of the system would let node 4 update its local copy and immediately return success.
That is fast, but unsafe.
If node 4 crashes before nodes 2 and 7 learn about the write, the cluster may lose a write that the client believed had committed. Worse, another replica might later become coordinator without knowing that write ever happened.
So the replicas need a shared history.
CockroachDB uses Raft for this.
Raft is a consensus protocol. In this context, its job is to make the replicas of a range agree on a sequence of commands.
The important part is not just copying data. The important part is ordering.
For Range 18, the update becomes a command in that range's Raft log.
A simplified write path looks like this:
write reaches leaseholder
→ leaseholder proposes a Raft log entry
→ replicas append the entry
→ a majority acknowledges
→ entry is committed
→ replicas apply committed entries in order
For example:
Range 18 replicas:
node 2
node 4 ← leaseholder
node 7
Node 4 proposes a log entry:
entry 42:
update orders/100 status = shipped
The replicas respond:
node 4: appended entry 42
node 2: appended entry 42
node 7: unavailable
Two out of three replicas have accepted the entry. That is a majority.
So the entry can be committed for Range 18.
This may feel strange at first. Node 7 does not have the entry yet, but the write can still commit.
The reason this is safe is that future majorities overlap with past majorities.
If node 4 crashes after the commit, node 2 still has the committed entry. A future leader cannot form a valid majority that completely ignores the history held by the committed majority.
That overlap is what prevents two different histories from both becoming valid.
So Raft gives us the middle ground we wanted earlier.
The system does not need every replica to respond, because that would make one failed node stop progress.
The system does not let one replica decide alone, because that would allow conflicting histories.
A majority gives the range both fault tolerance and a single ordered log.
At this point, a single-range write has a full path:
client sends SQL to node 1
→ node 1 maps SQL to a key
→ key belongs to Range 18
→ Range 18 metadata lists replicas
→ node 4 is the leaseholder
→ node 1 routes the write to node 4
→ node 4 proposes the write through Raft
→ node 4 and node 2 acknowledge
→ majority reached
→ write commits for Range 18
→ result returns to client
This is already a lot of machinery for one update.
But it is all forced by the two original choices.
Splitting data created the need for routing.
Replicating data created the need for consensus.
Now we need to look at what happens when one SQL transaction touches more than one range.
raft solves one range, not a whole transaction
A single-range write can be coordinated by one leaseholder and committed through one Raft group.
Real transactions are often larger.
Consider this transaction:
BEGIN;
UPDATE orders
SET status = 'shipped'
WHERE id = 100;
INSERT INTO shipments(order_id, warehouse_id)
VALUES (100, 8);
COMMIT;
The application expects this to behave like one action.
The order should not be marked as shipped unless the shipment row exists. The shipment row should not exist if the order update failed.
On one machine, the transaction manager can coordinate this locally.
In a distributed database, these two writes may touch different ranges.
orders/100
→ Range 18
→ leaseholder node 4
→ replicas node 2, node 4, node 7
shipments/100
→ Range 44
→ leaseholder node 6
→ replicas node 3, node 6, node 8
Now the transaction spans two independent coordination groups.
Range 18 has its own leaseholder and Raft log.
Range 44 has its own leaseholder and Raft log.
Raft can make Range 18 agree on Range 18's writes. Raft can make Range 44 agree on Range 44's writes. But Raft alone does not make a multi-range transaction atomic.
The database needs another layer.
It needs a way to place provisional writes on multiple ranges, track the state of the transaction, and later decide whether those writes become visible or disappear.
That is where MVCC, write intents, and transaction records enter.
mvcc gives the database multiple versions to reason with
CockroachDB uses MVCC, or Multi-Version Concurrency Control.
Instead of storing only one current value for a key, the database can store timestamped versions.
A simplified key might look like this:
orders/100 @ t10 = status: paid
orders/100 @ t20 = status: shipped
This matters because transactions need consistent views of the database while other transactions are reading and writing.
If every write simply overwrote the previous value in place, it would be much harder for concurrent transactions to read stable snapshots. One transaction might see part of another transaction's work. Another might overwrite data before an older transaction has finished reasoning about it.
MVCC lets the database separate physical storage from logical visibility.
A version may exist at a timestamp.
A transaction may read the version valid for its timestamp.
A newer write may be present without being visible to every reader.
This is especially useful in a distributed database because there is no single local timeline for the whole cluster. Different ranges are doing work on different nodes. A transaction may read from one range, write to another, and still need to appear as though it ran in a valid serial order.
MVCC gives the system versions.
But versions alone do not solve unfinished transactions.
For that, the database needs a way to mark writes as provisional.
write intents are unfinished writes
When a transaction writes a value, CockroachDB can create a write intent.
A write intent is a provisional write.
It says:
transaction T wants this value to exist,
but transaction T has not fully committed yet
For the order transaction, CockroachDB may create intents like this:
Range 18:
intent on orders/100
value: status = shipped
transaction: T
Range 44:
intent on shipments/100
value: warehouse_id = 8
transaction: T
Each intent lives on the range containing that key.
Each intent is replicated through that range's Raft group.
So the transaction leaves physical traces across the cluster before the transaction is fully committed.
This may sound dangerous, but it is exactly why the writes are marked as intents. They are not ordinary committed values yet.
If another operation encounters the intent on orders/100, it cannot simply ignore it. The transaction may have committed.
It also cannot blindly trust it. The transaction may have aborted.
So it has to ask another question:
what happened to transaction T?
That question is answered by the transaction record.
a transaction record ties the pieces together
A transaction record stores the state of a transaction.
Simplified, it can move through states like:
PENDING
STAGING
COMMITTED
ABORTED
This record matters because a transaction's writes may be spread across multiple ranges.
Range 18 can see the order intent.
Range 44 can see the shipment intent.
But neither range alone is the whole transaction.
The transaction record gives the database a shared place to check the state of transaction T.
If another operation finds an intent, it can follow the pointer back to the transaction state.
found intent on orders/100
→ created by transaction T
→ check transaction record for T
→ T is PENDING, COMMITTED, or ABORTED
If T committed, the intent can be treated as a committed value.
If T aborted, the intent can be ignored and cleaned up.
If T is still pending, the system may wait, push, or resolve the conflict depending on the situation.
This is the transaction layer above Raft.
Raft gives each range an ordered local history. Transaction records and intents let several ranges participate in one logical outcome.
A simplified multi-range transaction looks like this:
transaction T starts
T writes intent on Range 18
→ Range 18 replicates that intent through Raft
T writes intent on Range 44
→ Range 44 replicates that intent through Raft
transaction record tracks T
→ T moves toward COMMITTED or ABORTED
other operations encountering T's intents
→ check T's transaction record
→ decide whether the provisional values are visible
This is why distributed transactions are not just "Raft, but more of it."
Consensus gives agreement inside one replicated range.
Transactions need atomicity and isolation across several ranges.
That requires metadata about the transaction itself.
sometimes the correct result is a retry
The last piece is that not every transaction can safely commit on the first attempt.
Two transactions may touch overlapping keys. A transaction may read a value and later discover that another write changed what it was allowed to assume. A timestamp may need to move forward. A conflict may make the original ordering invalid.
In a serializable database, the system cannot simply commit both transactions if there is no valid serial order that explains the result.
So CockroachDB may ask the client to retry.
That retry is not just a random distributed systems annoyance. It is the database preserving the transaction model.
It means:
given the reads, writes, conflicts, and timestamps observed,
this transaction cannot safely commit as if it ran atomically
at its original timestamp
Trying again gives the transaction a fresh view of the world.
This is one of the costs of preserving strong transactional semantics across split and replicated data. The system cannot make conflicts disappear. It can detect them, order them, and sometimes reject an execution that would produce an unsafe result.
the components fit the problems
The architecture becomes easier to understand when each component is tied to the problem that forced it to exist.
First, the database splits tables into ranges because tables are too large and uneven to manage as one physical object.
SQL table
→ encoded keys
→ sorted keyspace
→ ranges
Then routing becomes necessary because a request can enter any node, while the relevant range may be coordinated somewhere else.
SQL request
→ gateway node
→ key span
→ range metadata
→ replicas
→ leaseholder
Then replication becomes necessary because one copy of a range is not enough for availability or durability.
Range 18
→ replica on node 2
→ replica on node 4
→ replica on node 7
Then Raft becomes necessary because replicas cannot accept writes independently.
leaseholder proposes command
→ majority of replicas append it
→ command commits in the range's log
→ replicas apply commands in the same order
Then transaction metadata becomes necessary because real SQL transactions may touch multiple ranges.
transaction T
→ intent on Range 18
→ intent on Range 44
→ transaction record tracks T
→ committed, aborted, or retried
The old single-machine database had simple answers.
where is the row?
inside this database
who coordinates the write?
this database process
who agrees the write happened?
the local system
what does a transaction mean?
the local transaction manager decided it
The distributed database has to rebuild those answers.
where is the row?
map SQL to keys, then keys to ranges
who coordinates the write?
the leaseholder for that range
who agrees the write happened?
a majority of the range's replicas through Raft
what does a transaction mean?
MVCC versions, write intents, and transaction records
make work across ranges behave like one action
This is why scaling a database is not just adding machines.
Adding machines is what creates the interesting problems.
Once data is split, the database needs routing.
Once data is replicated, it needs consensus.
Once transactions cross those split and replicated pieces, it needs transaction metadata and conflict handling.
CockroachDB's architecture is one answer to those problems. Ranges make data movable. Range metadata makes data findable. Leaseholders give each range a current coordinator. Raft gives replicas one ordered history. MVCC gives the system versions to reason about. Write intents and transaction records let a transaction span multiple ranges without exposing half-finished state.
The application still sends SQL.
The cluster does the distributed systems work required to make that SQL mean what the application expects.