Fieldnotes / engineering

Observability and tracing

Explanation

One operation is a span

A span records one operation, including its start, duration and attributes. It describes observed execution rather than the intended plan.

Instrumentation creates spans. Missing instrumentation means some operations will not appear.

  1. 0s The caller invokes a remote test tool. That operation takes time.
  2. 7s Instrumentation records the start. The tool performs its work.
  3. 14s The operation ends and its span closes. The record includes duration and outcome.
  4. 21s This span describes one operation. Several related spans can form a trace.

Related spans form a trace

A trace groups related spans. They share a trace ID but each has its own span ID. Parent IDs describe the nesting relationship.

Shared trace identity links operations; distinct span identities keep individual work identifiable.

  1. 0s The agent operation starts trace T. Its children represent individual calls.
  2. 7s A model call gets its own span ID. It keeps trace T and the parent relation.
  3. 14s A tool call gets another span ID. It also belongs to trace T.
  4. 21s The trace view groups the related spans. Parent links show how they fit together.

Cross a service boundary

When an operation crosses a process boundary, tracing context must travel with the request and be extracted by the receiving service.

Propagation links the spans, but instrumentation and export must also work. Headers alone do not create a complete trace.

  1. 0s The agent calls a separate service. A local parent relation cannot cross alone.
  2. 7s Include trace context in the request. It identifies the trace and parent span.
  3. 14s The receiver extracts that context. It starts a child span for its work.
  4. 21s Both sides now share trace identity. Export their spans to see the whole path.

Visibility is not a speedup

In this illustrative sequential run, the model takes 0.6 seconds and the tool 2.4 seconds. Correct propagation makes the slow tool visible but does not shorten the work.

Use tracing to locate time. A tracing repair is an observability improvement, not automatically a performance optimization.

  1. 0s Example: two calls run sequentially. 0.6 + 2.4 = 3.0 seconds total.
  2. 7s Without linkage, the tool looks separate. Its time is harder to attribute.
  3. 14s Next run: propagate the request context. Export now links the same 2.4s tool span.
  4. 21s The run still takes 3.0 seconds. We improved visibility, not execution speed.

Missing and sensitive data

Sampling, export failures and missing instrumentation can omit spans. Recorded attributes can also contain sensitive information unless the application controls them.

A missing span is not proof that work never happened. Minimize sensitive attributes and treat traces as partial evidence.

  1. 0s Runtime operations produce tracing data. Not every detail should be recorded.
  2. 7s Remove a secret attribute from span A. Keep the span itself in the export.
  3. 14s Separately, span B fails to export. The view has A, but B is missing.
  4. 21s Use traces with their collection limits. Absence of a span is not proof of absence.
Read the guide

Helpful first: Tool design.

The idea

Observability uses runtime evidence to investigate behavior. A trace groups related operations; each span records one operation, its timing, and contextual metadata. Spans share a trace ID but have distinct span IDs. Parent IDs describe nesting. Context propagation carries these relationships across process boundaries; instrumentation and export must also work.

Where it helps

When an engineer runs commands manually, terminal output and memory supply a rough history. Delegation removes that direct supervision. A final answer or total duration cannot show which intermediate operation consumed time. A recorded trace helps locate the next investigation, but does not prove that the generated answer is correct.

Teams need inspectable execution as soon as agents perform several operations without continuous supervision. Learning to connect one task to its tool activity gives engineers a practical debugging habit before they invest in larger monitoring systems.

An illustrative coding assistant asks a model which tests to run, then calls a remote test tool. The model takes 0.6 seconds; the tool takes 2.4 seconds. Without propagated context, the tool starts a separate trace. After correcting propagation and repeating the task, its span joins the agent trace. Total time is still 3.0 seconds: visibility improved, performance did not.

What you will be able to check

  • Locate the longest relevant operation in a recorded task and distinguish its duration from total elapsed time.
  • Demonstrate propagation across one tool boundary using trace and parent IDs.
  • Explain one evidence gap and verify that a seeded secret is absent from exported telemetry.

What changes in the team

Today
Run tests manually, inspect terminal output, and reconstruct the command sequence from shell history when a task stalls.
With an agent
Instrument the agent harness and remote tool, attach a trace reference to the task record, and inspect the recorded execution before changing the workflow.
Still your responsibility
Choose useful boundaries, approve captured fields, test instrumentation, and validate the resulting code or answer independently.

Work through an example

  1. Pick one safe test task. Start a task span in the harness and end it when the task finishes or fails. Add model and tool operations beneath it. OpenTelemetry has GenAI conventions for these operations, but its client-span guidance is marked Development; check the version your instrumentation implements.
  2. Check the boundary. For HTTP, a propagator injects context into outgoing headers and extracts it at the receiver. W3C traceparent includes a trace ID, parent ID, and flags. Do this in trusted transport code; asking the model to remember an identifier does not implement propagation or grant permission.
  3. Run the same fixture with and without propagation. Inspect actual exported parent relationships, not only matching timestamps. In the connected run, select the long tool span. Its duration includes whatever that instrumented operation covers; investigate inside it before concluding that the test process itself is slow.
  4. Compare the timeline with a correlated error log and task outcome. Nested or overlapping spans must not be added as if they were sequential. If a span is absent, check instrumentation, sampling, and export before concluding the operation never happened. Turn a confirmed failure into a regression case.

The engineering decision

Start with task, model, and tool boundaries rather than every function. Record operation names, durations, error classes, and a non-sensitive task identifier. Treat output evaluation and authorization as separate controls. Prefer approved metadata over prompts, source code, credentials, or tool payloads; define access and retention before collecting richer evidence.

  • Equating a successful span or plausible final message with correct code; preserve tests and review.
  • Treating an absent span as proof of no execution when sampling may omit telemetry.
  • Exporting raw prompts or tool arguments to debug faster, exposing secrets and customer content.
  • Changing prompts to address a delay that actually belongs to a tool or an uninstrumented wait.

Try it with your team

Break and repair one trace boundary

  1. Use a sandbox harness with a deterministic model stub and an HTTP test-tool stub; add a known delay to the tool.
  2. Record one run with outgoing context disabled and one with injection and extraction enabled; retain all traces for this fixture.
  3. Compare trace IDs, parent IDs, and durations. Ask a teammate to identify the disconnected operation without seeing the code.
  4. Place a fake secret in the tool input, export telemetry, and verify that the value was omitted. Record the capture policy and one remaining uncertainty.

You have an answer when…

  • The repaired run connects the receiver span to the intended parent while keeping distinct span IDs.
  • The teammate locates the delayed tool and explains why propagation did not reduce its duration.
  • The fake secret is absent from exported data; the review records a limitation of the trace.

Read further

Can a teammate use a recorded task to locate its slow tool operation, identify a missing parent relationship, and state what the trace cannot establish?

Original video comparisons