Fieldnotes / engineering
← Load balancing

Watch & compare

Load balancing

Different ways to see the same idea. Play one cut at a time, then return to the guide to try it yourself.

Why we recommend Unequal work

Start with Astra: one continuous two-worker example makes the idle-worker problem and the fate of an already queued call easy to follow. Its later calls use the free worker twice while the original call stays queued, showing exactly what a new routing policy changes. Baseline is a useful companion about health-based eligibility; Fable adds live counts and client response timing, but its denser three-replica animation and different before/after schedules require more explanation.

Original · 21 sec

Original explanation

Load balancing distributes requests across service instances according to a routing policy.

Read the video sequence
  1. 0–6s: Requests are distributed across three backends in turn.
  2. 7–12s: The middle server fails and health checks remove it from eligible routing. The remaining servers keep receiving work.
  3. 13–19s: A successful health check confirms recovery, and the middle server rejoins routing.
About this cut

Production: Original prototype; model not recorded.

Recommended · Astra · 28 sec

Unequal work

Equal request counts can hide unequal work; route new calls using current outstanding work while existing queued calls retain their assignment.

Read the video sequence
  1. 0–5.5s In this illustrative agent tool service, round robin sends a short call to worker A and then worker B. Each worker has one execution slot; both calls complete. The counters below the workers show cumulative assignments, not current load.
  2. 5.5–10.5s A receives long call L1 while B receives short call Q3. Both workers have received two calls, but B finishes and becomes idle while A is still running.
  3. 10.5–14.25s Round robin sends Q4 to A. It waits behind L1 although worker B is idle. Equal turns do not account for different task durations.
  4. 14.25–22s The example switches to least-request routing for new calls. Q5 and then Q6 go to B, breaking the alternating pattern. A still has L1 running and Q4 waiting; the policy change does not move those calls.
  5. 22–28s L1 finishes on A. Q4 then leaves its local queue, runs on A, and completes. Both workers finish, with three calls assigned to A and four to B. The resolved scene holds, then fades to black.
About this cut

Production: gpt-6-astra.

Revision: gpt-6-astra comparative review.

Illustrative two-worker, one-slot-per-worker schedule with perfect visibility; not a production benchmark.

Least request compares the two visible candidates; it is not an exact Envoy implementation or a universal improvement guarantee.

Existing requests are neither migrated nor retried; progress bars do not predict remaining time.

Audio levels, waveform timing and spectrogram were analysed; physical listening was not performed.

Comparative review and edits by gpt-6-astra comparative review; original author provenance retained. Recommendation and evidence are recorded in reviews/comparisons/load-balancing.json.

Alternative · Fable · 32 sec

Route on in-flight load, not request count

Round robin rotates through replicas without measuring request cost. This illustrative schedule shows a small request waiting while another replica is idle; selecting a minimum in-flight count avoids that placement for later new requests. The count remains an imperfect proxy for work.

Read the video sequence
  1. 0:00.7–0:05.7 — Normal case under ROUND ROBIN. Three small '2k tokens' requests leave the client about a second apart. The balancer's pointer steps to Replica 1, 2 and 3 in turn; each replica's counter ticks to '1 in flight' and back to 0 as its slot finishes, and each response returns with a green '2.8 s' demo-elapsed badge. The footer marks the schedule illustrative; all badges measure animation elapsed time, not production latency.
  2. 0:07.3–0:12.0 — Costs diverge. A '60k tokens' request lands on Replica 1 and another on Replica 2; their slots fill slowly with a low ticking. A small request lands on Replica 3 and finishes after 0.6 s of service. At 0:11.4 the three counters pop together: every replica has had exactly one request, yet they read 1, 1 and 0 because the short one is done.
  3. 0:12.2–0:16.7 — The misroute. The next small request leaves and the rotation points at Replica 1 anyway. It lands at 0:13.5 with a buzz and a jolt; Replica 1's counter turns orange at '2 in flight' and the request waits in an empty second slot with a WAIT badge until 0:15.25; it then starts filling with an arrival cue. Replica 3's counter ripples green at 0:14.0 to show it was free. The heavy jobs finish and return grey envelopes; the delayed small response arrives last at 0:16.65 with an orange '4.5 s' demo-elapsed badge.
  4. 0:18.0–0:19.3 — The fix. The balancer flashes green and its label changes in place to LEAST IN-FLIGHT; three green dots run from the counters up to the balancer's ports and a chime confirms the counters are now inputs.
  5. 0:19.8–0:26.8 — Resolution. The first heavy request is selected while all counters are 0; rotation breaks the tie toward Replica 2. By the next selection Replica 2 has one unfinished call, so the remaining zero-count tie selects Replica 3. With the counters at 0, 1, 1 the next small request goes to Replica 1 (its counter flashes green at the pick) and returns with a '2.8 s' demo-elapsed badge at 0:24.6; the one after that goes to Replica 1 again, where round robin would have sent it to busy Replica 2, and returns with a '2.8 s' demo-elapsed badge at 0:26.75.
  6. 0:27.0–0:31.8 — The heavy jobs complete and all counters return to 0 under the LEAST IN-FLIGHT label; the stage holds, then fades to the title-only frame that also opens the video.
About this cut

Production: anthropic/claude-fable-5-1.

Revision: gpt-6-astra comparative review.

Illustrative single-execution-slot model: token sizes and 6 s / 0.6 s service times are invented. The elapsed badges are calculated from scene send/receive times, including animated transport; they are not benchmark results or guaranteed token runtimes. The two phases are different schedules, not a controlled performance replay.

The in-flight count is a proxy for load, not a measure of work (the Google SRE chapter's own caveat): two replicas with equal counts can hold very different work. Least in-flight cannot tell them apart; the cut shows it helping only after short requests finish and the counts diverge. Weighted or utilisation-based policies, health checks, connection draining and long-lived connections are not shown; the baseline cut covers health checks.

The policy label changing on the balancer is a scene event. Routing policy is configuration enforced by the balancer or client library; a label, prompt or persistent instruction does not enforce routing, and by the same token persistent prompts do not enforce permissions.

The orange second slot and WAIT badge represent a local waiting request, not parallel execution. Its bar stays empty until execution starts. In-flight counters count backend-arrived, unfinished work; transport and response travel are excluded from this toy signal.

Decoded audio peak -1.2 dBFS; all 107 scheduled 160 ms cue windows contain energy. Waveform and spectrogram were analysed; overlapping cues were not isolated and audio was not heard.

These animations use sound effects rather than narration. The written guide explains the mechanism and its limits. Model names describe how a cut was made, not a quality ranking.

What the review found