Fieldnotes / engineering

Queues and backpressure

Explanation

Submit a job

A producer creates a job. A queue holds jobs until a worker can execute them. Submission and completion are different events.

Accepted means the job entered the queue; it does not mean the requested work has finished.

  1. 0s A producer creates a job. A queue holds it until a worker is ready.
  2. 7s Job A enters the queue. The job has been accepted.
  3. 14s A worker takes job A. It leaves the waiting queue.
  4. 21s Execution happens at the worker. Accepted is different from complete.

Finish a job

Follow one job from waiting to running to completed. A worker reports completion only after the work succeeds.

Waiting, running and completed describe different states. Acknowledgement policy must match the effect being confirmed.

  1. 0s Replay job A from its waiting state. No work has happened yet in this replay.
  2. 7s The worker takes job A. Execution starts after it arrives.
  3. 14s The worker executes the job. Leaving the wait queue did not finish it.
  4. 21s The work finishes successfully. Now the job can be marked complete.

A backlog grows

A slow worker cannot keep up with incoming jobs. The queue absorbs a short burst, but sustained excess arrivals create a growing backlog.

A queue delays overload; it does not create execution capacity. This example has one worker.

  1. 0s New scenario: the worker is running A. More jobs arrive while it is busy.
  2. 7s B waits behind the running job. The worker has not taken B.
  3. 14s C arrives before A finishes. C will join B in the queue.
  4. 21s D arrives too. The backlog grows. Arrivals exceed completions here.

A full queue pushes back

Give the queue three waiting places. Under this blocking admission policy, submitting E waits at the producer while the queue is full.

Backpressure makes upstream production wait or slow. Dropping a job would be a different policy: load shedding.

  1. 0s All three waiting places are full. E is still with the producer.
  2. 7s Admission waits at the producer. E does not enter the full queue.
  3. 14s The worker finishes A, then takes B. One waiting place becomes free.
  4. 21s E can now enter the queue. It is accepted, not yet completed.

Choose an overload policy

A bounded queue requires an explicit full-queue policy. This lesson illustrates rejection instead of indefinite waiting.

Waiting, rejecting and dropping work have different consequences. Expose the outcome rather than calling every submission successful.

  1. 0s The queue is full again. This policy rejects new submissions.
  2. 7s The producer asks to submit a job. No waiting place is available.
  3. 14s The queue reports rejection. The producer must handle that result.
  4. 21s The rejected job was not admitted. Make the overload policy explicit.
Read the guide

Helpful first: Tool design, Timeouts and retries.

The idea

A producer submits jobs; consumers execute them. Queues separate arrival from execution, but sustained arrivals above completions grow backlog unless work is delayed, rejected or discarded. Backpressure carries capacity information upstream so producers slow or wait. Load shedding instead abandons work. These are explicit policies, not automatic consequences of adding a queue.

Where it helps

An engineer running commands manually naturally waits for results. An agent runner can fan out tool calls faster than a test service or API can finish them. Moving that workflow into automation requires explicit limits on waiting jobs, active calls and how long either may live.

Use this chapter when agents start batch jobs or concurrent tool calls against limited services. It supports safe scaling of an established workflow; it is unnecessary infrastructure for a first supervised, sequential task.

Imagine an agent planning documentation checks against an internal search tool. Its harness has one worker and three waiting places. While job B is slow, C, D and E fill the queue. Submission of F waits at the producer. When the worker removes C, F can enter. Admission of F is not completion: expose queued, running and completed outcomes separately. The video illustrates this finite batch; it makes no claim about model speed or production capacity.

What you will be able to check

  • Locate and measure waiting work separately from executing calls.
  • Demonstrate that a full queue changes producer behavior without stopping the worker.
  • Explain which limits cover backlog, in-flight deliveries and request rate.
  • Report queue admission separately from successful tool completion.

What changes in the team

Today
An engineer submits one documentation check, watches its result, then submits the next.
With an agent
The harness feeds checks through a bounded queue and fixed worker count; the producer awaits admission.
Still your responsibility
The team owns acceptable delay, cost, cancellation, retry policy and whether a result is good enough.

Work through an example

  1. Start locally. Python’s asyncio.Queue(maxsize=3) permits three waiting items. Awaiting put() on a full queue waits for get() to free a place; task_done() records completion separately. With one worker, three waiting jobs and one active job can coexist. This is not a durable broker.
  2. Keep production bounded too. Have one producer await admission before generating more jobs. Creating unlimited tasks that each wait on put() merely moves the backlog outside the queue. Large payloads also need a byte or memory budget; an item count alone is insufficient.
  3. For a RabbitMQ deployment, manual acknowledgements identify handled deliveries. Prefetch caps outstanding unacknowledged deliveries, commonly per consumer; it does not cap the broker’s waiting backlog. Publisher confirms cover broker acceptance, not successful tool execution. Redelivery remains possible, so externally visible actions need an idempotency strategy.
  4. Configure overflow deliberately. RabbitMQ queue-length limits count ready messages, excluding unacknowledged ones; the default can drop oldest messages. Choose a supported rejection policy and handle its feedback if silent loss is unacceptable. Separately choose deadlines and a finite retry budget. Increasing queue size only postpones the capacity decision.

The engineering decision

Choose limits from measured tool cost, latency and the team’s delay budget. Google SRE cautions that request counts hide different resource costs. Track oldest waiting age, active calls and completion rate alongside depth. An age threshold is a policy signal, not proof that remaining work will meet its deadline. Enforce limits in the runner; a prompt asking an agent to be careful is not enforcement.

  • Increasing worker count floods the same downstream bottleneck; a concurrency cap is not a requests-per-minute quota.
  • Immediate retries compete with useful work and can sustain overload; repeated failures need a terminal outcome.
  • Acknowledging before recording a result can lose recoverable work; acknowledging later still requires duplicate-safe side effects.

Try it with your team

Make the producer wait

  1. Use a local stub tool and six named jobs; avoid paid API calls or repository writes.
  2. Run one consumer and an asyncio.Queue(maxsize=3). Log submission start, admission, execution start and finish.
  3. Make the second job take several seconds. Submit the others from a single producer using await put().
  4. Check that the sixth admission waits for a free place, then let all jobs drain. Discuss where a deadline would cancel stale work.

You have an answer when…

  • Logs never show more than three waiting jobs or one executing job.
  • The blocked submission resumes after a dequeue, while existing work continues.
  • All six stub jobs reach a recorded outcome; admission is never mistaken for completion.

Read further

If the downstream tool slows tenfold, where does waiting accumulate, and which component stops accepting or generating more work?

Original video comparisons