Watch & compare
Agent evals
Different ways to see the same idea. Play one cut at a time, then return to the guide to try it yourself.
Why we recommend Test the fix
Astra makes the grading decision concrete: rejecting an empty password is correct, but rejecting a valid password is a regression that blocks the patch. Its larger expected/actual labels read most clearly on a phone, and two passing checks lead explicitly to human review. Baseline is a useful follow-up on checking actual state rather than trusting Done; Fable shows how an incomplete grader can pass a broken artifact and how expanding the requirements exposes it.
Original · 26 sec
Original explanation
Agent evals measure task outcomes and behavior using repeatable scenarios and explicit grading rules.
Read the video sequence
- 0.0–8.0s: Trial 1 sends a close-ticket task to the agent and harness. Ticket #42 becomes CLOSED at 3.2s; the final message separately says Done at 4.65s. The grader reads CLOSED at 6.7–7.35s and records PASS at 8.0s.
- 8.0–13.0s: The pass holds. At 11.0s, trial 1 is retained as a PASS badge while the ticket resets to OPEN and the final message clears for trial 2.
- 13.0–19.8s: The second close attempt leaves the ticket OPEN. The final message still becomes Done at 16.45s. At 18.5–19.15s, the same state grader reads the actual OPEN state.
- 19.8–26.0s: The second trial fails at 19.8s. At 21.6s the result is labelled 2: FAIL beside 1: PASS. Both results hold until the full-frame fade starts at 24.0s; the final black frame matches the first.
About this cut
Production: Original prototype; model not recorded.
Recommended · Astra · 28 sec
Test the fix
Evaluate an agent-generated fix against both the requested behavior and existing behavior before advancing it to human review.
Read the video sequence
- Illustration: a coding agent proposes authentication patch A. It rejects an empty password, so the new requirement passes.
- The same patch also rejects a valid password. The existing login behavior fails even though the new test passed.
- Both results reach the evaluation gate. One required check failed, so the candidate is blocked.
- The agent produces revised patch B. Previous results clear. Fresh checks now reject the empty password and accept the valid password; both pass.
- The two passing checks advance patch B to human review. This limited example does not establish release readiness. The scene fades for a clean loop.
About this cut
Production: gpt-6-astra.
Illustrative authentication patches and two checks; no agent experiment or comprehensive security evaluation.
Patch A to B depicts revision, not repeated trials of one configuration.
Passing checks supports human review, not release authorization or permission enforcement.
Audio was measured and inspected spectrally, not physically listened to.
Self-reviewed; no comparative recommendation across variants.
Alternative · Fable · 27 sec
Assert what must not change
A grader that checks only the requested fix can miss regressions; require the existing behaviors that must be preserved as well as the new outcome.
Read the video sequence
- 0.0–8.0s: The task 'fix login bug' goes to the agent (harness v1). Its patch turns test_login from FAIL to PASS while test_signup and test_reset stay PASS. The grader, labelled required checks, runs its single assertion, login FAIL→PASS, ticks test_login and shows PASS. This is the normal case.
- 8.0–10.3s: The environment resets for a fresh trial (test_login failing again) and the agent's sublabel changes to 'harness v2', marking a change under test.
- 10.3–19.0s: The same task runs under v2. The patch fixes test_login, but test_signup and test_reset jolt and turn FAIL. The grader still checks only test_login and shows a green PASS beneath two red rows: a false pass, because nothing asserted the unrelated tests.
- 19.0–22.8s: A second assertion, others PASS→PASS, is added to the grader. Regrading the same end state ticks test_login, then marks × beside test_signup and test_reset; the second grader row turns red and the verdict pops to FAIL with a distinct low-frequency failure cue.
- 22.8–27.1s: The failing verdict holds, then the stage fades out while the title remains visible, matching the opening frame.
About this cut
Production: anthropic/claude-fable-5-1.
Revision: gpt-6-astra comparative review.
Illustrative scenario: the harness change breaking two unrelated tests is one plausible regression, not a claim about how any model, prompt or harness behaves or how often this occurs.
One trial per harness version is shown; the resolution regrades the recorded end state rather than rerunning the task from a clean environment across several trials, which is what a real suite does.
Audio was verified by analysis (final encoded peak −3.2 dBFS, spectrogram, cue timing); it was not listened to.
The three-test suite stands in for any end-state assertion set (tickets, rows, events); the video shows only the test-suite form.
These animations use sound effects rather than narration. The written guide explains the mechanism and its limits. Model names describe how a cut was made, not a quality ranking.
What the review found
- Astra 3.30–11.30s separates expected behavior, observed output, and grade: the same REJECT passes the empty-password case and fails valid login. This is more immediately actionable for a coding team than Baseline’s ticket predicate or Fable’s abstract test names.
- Baseline 4.65s versus 16.45s shows the same Done message with different actual ticket states; only the state reaches the grader. Its retained trial result and 11.00s fixture reset introduce variability that the two patch revisions in Astra do not.
- Fable 13.28–15.89s deliberately juxtaposes two failing regression tests with a passing narrow grader. At 19.00–22.83s it expands the required checks and rejects the same artifact, teaching grader quality rather than another agent retry.
- Astra 23.80–26.70s ends with 2/2 passed and Human review, the strongest explicit limit on what these checks establish. Baseline 21.60–24.00s retains two trial outcomes; Fable 22.83–25.94s holds a failed artifact, without a release claim.
- Fable throughout, especially 19.60–23.60s: revised 36px rows in wider stores replace cramped 26px type, and the required-checks label distinguishes predicates from results. Phone readability now passes, while Astra’s larger behavioral labels remain easier to scan.
- Baseline 19.80s is the actual FAIL onset, corrected in its transcript; 21.60s only adds the trial number. Astra’s 10.60–10.90s and 23.10–23.40s evidence packets route around the grade text without obscuring it. All final cuts have matching decoded first/last frames and deliberate result holds.