Fieldnotes / engineering

Canary deployments

Explanation

Keep a control version

A canary deploys a candidate to limited traffic while the current version remains as a control. The comparison must influence the rollout decision.

A slow rollout alone is not an evaluation. Define which outcomes will cause promotion, pause or abort.

  1. 0s Version 1 is serving the application. Version 2 is a candidate change.
  2. 7s Keep the current version as a control. It gives the candidate a comparison.
  3. 14s Send limited traffic to the candidate. Observe its behavior before expanding.
  4. 21s Compare results from both versions. The result must change a rollout decision.

Limit the exposure

In this example, the candidate receives ten percent of traffic: 1,000 of 10,000 requests. The other 9,000 stay on the current version.

The percentages describe this illustrative window. A small share limits exposure, but can still harm affected users.

  1. 0s This window has 10,000 requests. The candidate receives a ten-percent share.
  2. 7s 9,000 requests go to version 1. Most traffic stays on the control.
  3. 14s 1,000 requests go to version 2. The candidate has limited exposure.
  4. 21s Measure each version separately. A small rollout is not proof of safety.

An average hides failure

Forty candidate exports fail. Everything else succeeds. Across all traffic that is 40 out of 10,000, or 0.4%, but the affected operation is much worse.

Aggregate rates can conceal failures concentrated in a small version-and-operation slice.

  1. 0s There are 10,000 total requests. Only 100 are exports handled by v2.
  2. 7s Forty candidate exports fail. The other requests succeed.
  3. 14s Overall: 40 failures out of 10,000. The headline error rate is only 0.4%.
  4. 21s An overall 1% alert would stay quiet. Split the results before judging safety.

Compare the same operation

Compare export requests on the candidate with export requests on the control. Candidate exports fail 40 out of 100 times; control exports fail zero out of 900.

The 40% candidate-export failure rate is relevant to this rollout. These authored counts are not real benchmark results.

  1. 0s Compare exports with exports. Do not mix them with unrelated requests.
  2. 7s Version 1: zero failures out of 900. Control exports have a 0% error rate.
  3. 14s Version 2: 40 failures out of 100. Candidate exports have a 40% error rate.
  4. 21s The candidate has a clear regression. The comparison argues against promotion.

Abort and verify recovery

After detecting the regression, stop sending new traffic to the candidate and verify the current version can safely resume the load.

Rerouting new requests does not undo completed writes or guarantee schema compatibility. Rollback safety needs separate planning.

  1. 0s The candidate fails the release check. Stop increasing its share.
  2. 7s Route new requests to the control. The candidate receives no new traffic.
  3. 14s Check the recovered service behavior. Do not assume abort means healthy.
  4. 21s Confirm recovery before concluding. Data changes may need separate repair.
Read the guide

Helpful first: Load balancing, Observability and tracing, Agent evals.

The idea

A canary is a limited deployment with an evaluation, not simply a slow rollout. Keep the current version as a control: the comparison must influence promotion, pause, or abort. Google SRE explains why a healthy service-wide average can conceal a failing candidate.

Where it helps

Use this when your team starts shipping agent-written service changes or versioned agent tooling. Code review and offline evaluations remain necessary; production exposure tests a different question: does this change behave acceptably for the people and workloads receiving it?

Consult this chapter when an agent-assisted change reaches a live service or a shared agent tool. It earns a supporting place because delegation changes who prepares the release, while engineers still need a defensible exposure and recovery decision.

Illustrative export-service release: an agent refactors CSV generation. Route 10% of requests to v2. In one window, v1 serves 9,000 requests and v2 serves 1,000; exports comprise 10% of each. Forty of v2’s 100 exports fail, while everything else succeeds. The overall error rate is only 0.4%, but candidate exports fail at 40% versus 0/900 on v1. An overall 1% alert would miss this regression.

What you will be able to check

  • Calculate both pooled and version-specific operation error rates from request counts.
  • Write a rollout gate with explicit pass, fail, and insufficient-evidence outcomes.
  • Demonstrate traffic returning to the stable version and identify effects that rollback cannot undo.

What changes in the team

Today
An engineer deploys a release, watches a shared dashboard, and decides whether to continue from memory and a runbook.
With an agent
Ask an agent to draft version-filtered queries, replay fixtures, and a rollout report linking each decision to counts, time windows, and the deployed artifact.
Still your responsibility
The release owner reviews those queries, sets risk limits, authorizes deployment access, and owns the promotion and recovery policy. Model-written instructions do not enforce permissions.

Work through an example

  1. Freeze the candidate artifact and change scope. For the export refactor, review the diff and offline tests first. Write down which export formats and customer cohorts must be observed; keep an owner available during the trial.
  2. Establish routing and measurement. Label observations by release and operation, compare concurrent windows, and record actual request counts. Collect a new window after exposure starts; do not reuse the earlier zero-error result as evidence for v2.
  3. Evaluate the fixture. Divide 40 by 10,000, then 40 by 100. For this exercise only, fail when candidate export errors exceed control by five percentage points after 100 candidate exports; pause with fewer candidate samples or missing control observations. A missing denominator must never be treated as a zero error rate. These are training parameters, not universal safety thresholds.
  4. Rehearse recovery. Send new traffic back to v1, preserve the failed window, and verify fresh requests. AWS describes why stored data and protocols must remain compatible with the older version; reverting code is not a repair for previously corrupted data or completed external actions.

The engineering decision

Choose exposure, observation time, and critical operations before deployment. In Argo Rollouts, analysis can continue, abort, or pause a rollout; a timed pause alone is not an evidence gate. Replica counts only approximate traffic weights without traffic management. Test what your router actually delivers.

  • Treating a low pooled error rate as approval without checking the changed operation.
  • Promoting on absent metrics or an unrepresentative handful of requests; zero observations are not zero failures.
  • Assuming small exposure isolates all harm: shared dependencies can affect the control, so retain absolute service-health checks.
  • Letting the agent revise its own acceptance threshold after seeing a failure, without a reviewed policy change.

Try it with your team

Catch the hidden export regression

  1. Create a local count table using the example’s four version/operation groups; make no production changes.
  2. Ask an agent for the pooled and export-specific calculations, then verify the denominators yourself.
  3. Implement the illustrative gate; run passing, failing, and insufficient-sample fixtures.
  4. Simulate an abort by setting candidate traffic to zero. Save the failed report alongside a fresh stable-version observation.

You have an answer when…

  • The 0.4% pooled result cannot override the 40% candidate export failure.
  • Missing samples pause the decision, and abort preserves the failure evidence instead of relabelling the candidate healthy.

Read further

Which critical operation would disappear inside your overall success rate, and what would stop promotion if its candidate sample were empty?

Original video comparisons