The Single-Node Lie

Three assumptions that were free

On a single machine, three things are true and you never think about them. There is one clock, and it only moves forward. A function call either returns or throws — there is no third outcome. And when something stops responding, it has stopped: there is no meaningful distinction between “crashed” and “slow,” because your own process being slow doesn’t change while you wait for it.

Add one more machine and all three become false in ways that produce silent data loss, phantom orderings, and outages that survive the thing that triggered them. Here is the full ledger, because naming the assumption is half of catching it in a design review.

Lie #1: there is one clock

The clock on a server is two clocks wearing a trench coat. The wall clock (clock_gettime(CLOCK_REALTIME), Date.now(), time.Now()) tells you the calendar time, and it is constantly being corrected by NTP to track an external reference. Those corrections jump — forward when the clock was slow, and backward when it was fast. The monotonic clock (CLOCK_MONOTONIC, performance.now()) only ever increases, but its zero is arbitrary and it’s meaningless across machines.

Two failures fall out of this immediately. First: if you measure “how long did this take” with the wall clock and NTP steps backward mid-measurement, your duration is negative — a well-known source of timeouts that fire instantly and rate limiters that grant infinite budget. Always measure elapsed time with the monotonic clock. Second, and the one that ate the customer’s address: comparing wall-clock timestamps across machines to order events is comparing two independently-wrong numbers. A 1-second skew is ordinary. Under load it’s worse, because NTP corrections get deprioritized exactly when the machine is busy.

That distinction is abstract until you watch it form. Below, three replicas of a shopping cart take writes and gossip them around. Step through it, then use the probe to ask whether any two events are ordered — or concurrent. Watch what happens to A’s write and C’s write.

The repair: logical clocks

A Lamport clock is a single counter per process: increment on every event, and on receiving a message set your counter to max(local, received) + 1. It guarantees that if A happens-before B then stamp(A) < stamp(B). That’s enough to build a consistent total order (break ties by node id) — and it’s all you need for, say, deterministic tie-breaking. But it has a fatal blind spot for conflict detection: a smaller stamp does not prove causality. From two Lamport stamps alone you can never tell “A caused B” from “A and B were concurrent.” That information was never captured.

A vector clock captures it. Keep one counter per replica. On a local event, bump your own. On receive, take the element-wise max, then bump your own. Now the comparison is total information: A happens-before B iff every component of A is ≤ B and at least one is strictly less; if each clock leads in some component, they’re concurrent. That “each leads somewhere” case is the one LWW cannot see.

1compare(other: VectorClock): Ordering {
2  let lessSomewhere = false
3  let greaterSomewhere = false
4  for (const id of VectorClock.union(this, other)) {
5    const a = this.get(id)
6    const b = other.get(id)
7    if (a < b) lessSomewhere = true
8    else if (a > b) greaterSomewhere = true
9  }
10  // leads in both directions => causally independent
11  if (lessSomewhere && greaterSomewhere) return "concurrent"
12  if (lessSomewhere) return "before"
13  if (greaterSomewhere) return "after"
14  return "equal"
15}

The point of returning concurrent as a first-class answer is that your application now gets to decide. For a shopping cart, the right resolution is to merge (union the items) — Amazon’s Dynamo paper made exactly this choice. For a username, it might be to surface the conflict to a human. The bug in the opening ticket was never “we picked the wrong winner.” It was “we treated a concurrent pair as if one had to win.”

Lies #2 and #3: instant calls, and crashed vs. slow

These two are the same lie. On one machine a call returns or throws. Across a network there is a third outcome — silence — and you cannot tell which of three things it means: the request never arrived, it arrived and the work was done but the reply was lost, or it’s all still happening and you’re just early. The only tool you have is a timeout, and a timeout is a guess.

This is not an engineering shortcoming you can buy your way out of. The FLP result (1985) proved that in a fully asynchronous network — one with no bound on message delay — there is no algorithm that can reliably tell a crashed node from a slow one. Real networks aren’t perfectly asynchronous, so we get to cheat with timeouts. But every timeout is a bet on the maximum normal delay, and every such bet is sometimes wrong. A failure detector is just the formal name for that bet, and the useful insight is that you get to choose how it’s wrong.

What this sounds like in an interview

Exercises

1// Receive a message carrying `msg`, then record that WE observed it.
2receive(msg: VectorClock, selfId: string): VectorClock {
3  // take the element-wise max so we incorporate the sender's history
4  const merged = this.merge(msg)
5  // ...and return it.
6  return merged
7  // (we already merged in the sender's counters — what else is there?)
8}