Fieldnotes / engineering

Load balancing

Explanation

One request

An application asks a server to do something. The server runs the work and returns a response. Here the request asks for a user profile.

A request is the ask. A response is the result. The server does work between them.

  1. 0s Meet the application and server.
  2. 5s The application sends a request.
  3. 10s The server processes the request.
  4. 19s The server sends a response.
  5. 25s The application receives the result.
Check your understanding

What is the server doing while the application waits?

It is processing the request. The response travels back only after that work finishes.

Share the work

A server has finite capacity. A second server can run the same service, so separate requests can be handled at the same time. This simplified example gives each server one work slot.

More servers can provide more capacity when requests can be handled independently. Someone still needs to choose a server.

  1. 0s Two servers can run the same service.
  2. 6s The first request starts on server A.
  3. 12s A second request starts on server B while A works.
  4. 24s Both finish independently.
  5. 29s Separate requests use separate capacity.
Check your understanding

Does adding a second server make one request run twice as fast?

No. Here it lets two separate requests run at once. Each request still takes its own processing time.

Take turns

A load balancer receives a request and selects a server. Round robin is a selection rule: take turns between eligible servers. Here both servers have the same configured weight.

Round robin rotates through servers: A, B, A, B. A request goes to one selected server.

  1. 0s Introduce the load balancer.
  2. 5s Request 1 routes to A.
  3. 13s Request 2 routes to B.
  4. 21s Request 3 routes to A.
  5. 29s The rule repeats A, B, A, B.
Check your understanding

After A, B, A, which server gets the next request?

B, provided both servers remain eligible and equally weighted.

Unequal work

Taking turns balances the number of assigned requests, but requests can take different amounts of work. Watch a slow report on A and a short lookup on B. This example allows one running request per server.

Equal request counts do not imply equal remaining work. Round robin can choose a busy server even while another is idle.

  1. 0s Some requests take longer.
  2. 4s A receives a slow request.
  3. 10s B receives a quick request.
  4. 18s B is ready while A remains busy.
  5. 22s Round robin selects A and request 3 waits.
  6. 31s A finishes and the waiting request starts.
Check your understanding

Why does request 3 wait even though B is ready?

The round-robin rule selects A next. In this one-slot example A is still working, so the new request waits there.

Look at active work

Another rule is least requests: choose a server with fewer requests still in progress. We start with the same imbalance as before: A has one long-running request and B has none. Counts are useful signals, not perfect measurements of cost.

Least requests uses current active-request counts when choosing where new work goes. It does not move existing work.

  1. 0s A has one active request; B has none.
  2. 8s Compare current active counts.
  3. 14s The new request goes to B.
  4. 23s B finishes while A continues.
  5. 29s Selection changes new assignments only.
Check your understanding

Does choosing B move the long-running request away from A?

No. A keeps its existing request. Only the new request goes to B.

Avoid a failed server

A selection rule also needs to know which servers are eligible. In this example a periodic health probe gets no successful response from A. After its configured timeout, the balancer marks A unavailable and sends new requests to B.

Health checks can remove a failed server from new routing decisions after detection. They do not instantly detect every failure or rescue requests already sent.

  1. 0s A stops responding.
  2. 6s The balancer sends a health probe.
  3. 13s The probe times out.
  4. 18s A is excluded from new selection.
  5. 23s A new request routes to B.
  6. 31s B returns its response through the balancer.
Check your understanding

If A fails just after a successful check, is the failure detected immediately?

No. Detection depends on the next checks, their timeouts, and configured thresholds. A request already sent can still fail.

Read the guide

Helpful first: Tool design, Queues and backpressure, Observability and tracing.

The idea

A load balancer selects a backend: one service instance able to handle a request. The routing unit matters. Transport-level balancing assigns connections; application-level balancing can assign individual requests or HTTP/2 streams. Round robin rotates through eligible instances without measuring their current workload. These distinctions are documented by gRPC and Envoy.

Where it helps

An agent that fans out repository searches can expose bottlenecks in a shared tool service. Investigate per-worker waiting time, outstanding calls, resource use, and failures before adding replicas or changing models. Those measurements help distinguish routing imbalance from exhausted capacity.

Use this chapter when concurrent agents share replicated tool services and traces reveal uneven waiting or utilization. A team learning delegation on one workstation should prioritize tool contracts, bounded concurrency, and evaluation before introducing balancing infrastructure.

Imagine two identical repository-search workers, each with one execution slot. Short file lookups usually finish quickly; a full-tree scan takes longer. Round robin can send the next lookup behind that scan while the other worker is idle. The accompanying film is an illustrative schedule, not a benchmark or a production configuration.

What you will be able to check

  • Identify whether a tool call is routed per connection or per request.
  • Use a trace to distinguish cumulative assignments, outstanding requests, and executing work.
  • Compare two routing policies under the same mixed-cost workload and explain a limitation.

What changes in the team

Today
An engineer launches searches sequentially and investigates slow jobs individually, with little simultaneous demand on the shared service.
With an agent
An agent launches bounded concurrent searches through a measured routing layer; engineers replay representative traffic before changing its policy.
Still your responsibility
Own concurrency limits, service eligibility, permissions, rollback thresholds, and correctness checks. An instruction in an agent prompt does not enforce infrastructure limits or access control.

Work through an example

  1. Map the real path from agent to tool client, proxy, and worker. Check whether multiple calls share a connection. gRPC documents how an application-aware proxy can distribute HTTP/2 streams; adding workers does not itself move an existing connection.
  2. Replay a small workload with fixed arrivals: several short lookups and one long scan. Record assignment, execution start, completion, and backend. A request waiting at a worker remains outstanding to the caller even though it is not executing.
  3. Compare round robin with a toy least-outstanding dispatcher. In the film, A has a running scan and queued Q4; B is idle. New Q5 and then Q6 go to B after the policy changes, visibly breaking rotation. Q4 stays on A until the scan finishes. Routing new work does not migrate existing work.
  4. Compare waiting and completion times, errors, and resource saturation. If all workers fill, change admission limits or capacity. Envoy documents separate limits for pending requests, outstanding requests, and retries; balancing alone supplies none of that extra capacity.
  5. Rehearse removing a worker before rollout. Envoy documents explicit drain controls and protocol-specific behavior, so test your shutdown sequence instead of assuming a successful health response or a termination signal guarantees a clean drain.

The engineering decision

Use request-aware routing when uneven request cost is an observed problem. Google SRE explains why outstanding-request counts can still misrepresent capacity, especially with unseen traffic or fast failures. Envoy’s equal-weight least-request policy samples candidates rather than necessarily scanning every host. Validate the deployed policy and version; neither algorithm promises optimal latency.

  • Counting requests instead of measuring cost: equal assignments can conceal a long-running task.
  • Treating fewer outstanding calls as guaranteed spare capacity; fast errors or other clients can make that signal misleading.
  • Replaying state-changing tool calls to fix distribution; the demo uses read-only work and establishes no retry safety.

Try it with your team

Find the idle worker

  1. Create two local single-slot worker stubs with FIFO queues; use read-only simulated tasks.
  2. Send six fixed arrivals, mixing 200 ms tasks with one 2 s task; log arrival, assignment, start, and finish.
  3. Replay exactly that schedule with round robin and least outstanding, counting both running and queued calls.
  4. Compare individual waits and maximum queue depth; repeat with both workers occupied and record why routing alone cannot remove all waiting.

You have an answer when…

  • Show an actual trace where work waits while another worker is idle, or explain why your chosen arrivals never create that condition.
  • Explain one measured difference without claiming universal improvement; confirm every request completes once and existing work is never moved.

Read further

Where is routing decided in our tool path, what is counted there, and can we demonstrate waiting on one worker while another is idle?

Original video comparisons