Quick Takeaways
What you'll learn in this article
- 1
Companion code on GitHub โ the full, tested otel-mcp-agent-tracing-typescript project from this tutorial.
- 2
The agent-observability turn โ Gemini Spark and OpenTelemetry's MCP spans โ the market context for why agent tracing landed all at once this week.
- 3
Prediction: OpenTelemetry's GenAI and MCP conventions reach Stable and ship default-on by end of 2027 โ my dated, falsifiable claim on where this standardizes.
- 4
Build a parallel subagent orchestrator in TypeScript โ the same root-span discipline applied to a fleet of concurrent agents.
- 5
The Overnight Run โ a short story on the difference between watching an agent and controlling it.
Keep reading for detailed implementation, code examples, and real-world results
The week I wrote this, Google shipped a personal agent meant to run twenty-four hours a day and take actions on your behalf, checking in only before the big ones. That is the destination the whole industry has been driving toward for two years, and arriving at it quietly changes the engineering problem. An agent you supervise turn by turn does not really need instrumentation, because you are the instrumentation โ you see every step because you approve every step. An agent that works while you sleep is a different animal. It fans a single instruction out into a dozen model calls and tool invocations, most of which you will never witness, and the only way to know what it did is to have decided, in advance, to record it.
That recording is a trace, and this tutorial builds one. We will write a small, honest agent loop โ a model that thinks, a registry of tools it can call, and a controller that runs the loop until the model produces a final answer โ and then we will make the whole thing observable with OpenTelemetry. By the end you will have an agent that emits a clean span tree for every run: one root span for the invocation, a child span for each model call, a child span for each tool call, each carrying the conventional attributes that let any OpenTelemetry backend read them. And because the agent is deterministic and runs with no API keys, the tests can assert the exact shape of that tree without spending a cent or flaking on the network.
The companion code lives at CrashBytes/ByteSizedExamples/otel-mcp-agent-tracing-typescript. Clone it, run npm install && npm run demo, and watch the span tree print to your console while you follow along.
Why an agent needs a trace, not a log
Start with the failure that motivates everything else. A single user request to a tool-using agent does not produce one model call. It produces a planning step, a tool call, a follow-up model call to interpret the tool's result, maybe another tool call, and a final synthesis. Each of those is a place where things go wrong in a way that a log line cannot explain. The model can decide to call a tool that does not exist. A tool can return an error the model shrugs off and talks past. The agent can loop, calling the same tool with the same bad arguments until it exhausts its step budget and returns something confident and wrong.
None of those failures show up in the metrics most teams start with. Request counts and token totals tell you the agent ran and what it cost; they do not tell you that step four called the weather tool with an empty city and the model papered over the blank result. To see that, you need the run rendered as what it actually is: a tree.
How visible each layer of an agent run is by default (illustrative)
| layer | visibility |
|---|---|
| Tool calls | 18 |
| Model decisions | 62 |
| Token + cost totals | 90 |
| Final answer only | 98 |
A trace is the right shape because an agent run is a tree and a trace is a tree. The root is the invocation. Beneath it hang the model calls and tool calls, in the order they happened, each a span with its own start time, duration, status, and attributes. When a run misbehaves you do not scroll a wall of log lines hunting for correlation IDs; you open the trace and read the branch that turned red. This is the same move the industry made for microservices a decade ago, which is exactly where OpenTelemetry came from. Agents are the new distributed system, with the twist that the nondeterminism lives inside the nodes.
The same agent failure, two debugging surfaces
Debugging with logs
Debugging with traces
The market has noticed. Agent-observability platforms โ Braintrust, LangSmith, Arize Phoenix, Helicone, Datadog's LLM product, and others โ now compete on reading these traces. The leverage move is to emit traces in a standard, portable format so you own the data and can point it at any of them. That is what OpenTelemetry's generative-AI semantic conventions give you, and it is what we will use. For the wider context on why this all landed at once, I wrote a companion analysis on the agent-observability turn and OpenTelemetry's new MCP spans.
The concepts you need: traces, spans, conventions
Three OpenTelemetry primitives carry the whole tutorial.
A span is one timed unit of work โ it has a name, a start and end, a status, and a bag of key/value attributes. A trace is a set of spans that share a trace id and link to each other through parent/child relationships, forming the tree. A tracer is the object you ask to create spans. That is the entire mental model; everything below is just deciding which spans to create and which attributes to set.
The "which attributes" question is where the semantic conventions come in. OpenTelemetry publishes a standard vocabulary so that a model span from your agent and a model span from someone else's agent describe themselves the same way. The generative-AI conventions define gen_ai.* keys โ gen_ai.operation.name, gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and so on. As of 2026 the tool layer is covered too: Model Context Protocol attributes like mcp.tool.name and mcp.method.name describe the tool call, and they enrich the existing tool-execution span rather than duplicating it. Following the conventions is what makes your traces legible to backends you have not even chosen yet.
The shape we are building toward
1 root, N children
Every agent run becomes one agent.invoke root span with a child chat span per model call and a child mcp.tool span per tool call, all sharing one trace id.
A quick word on what this tutorial is not. It is not a guide to a specific agent framework โ there is no LangChain or LlamaIndex here, on purpose, because the point is the instrumentation pattern, which you can drop into any loop. And it is not a model-quality tutorial; the "LLM" is a deterministic fake so the focus stays on the trace. Swap the fake for a real provider and the spans do not change.
Project setup
The project is an intermediate-level TypeScript package: ECMAScript modules, strict mode, a handful of source files, and a Vitest suite. Here is the quick start.
git clone https://github.com/CrashBytes/ByteSizedExamples.git cd ByteSizedExamples/otel-mcp-agent-tracing-typescript npm install npm run demo # runs the agent, prints the span tree to the console npm test # asserts the span tree shape with an in-memory exporter
The dependencies are the OpenTelemetry packages and nothing exotic. These are the versions the project pins, all current stable releases at the time of writing:
| Package | Version | Role | | ----------------------------------------- | ---------- | --------------------------------------- | | @opentelemetry/api | ^1.9.1 | Tracer, spans, context โ the surface | | @opentelemetry/sdk-trace-node | ^2.8.0 | The Node tracer provider | | @opentelemetry/sdk-trace-base | ^2.8.0 | Processors and exporters (incl. memory) | | @opentelemetry/resources | ^2.8.0 | Service-identity resource | | @opentelemetry/semantic-conventions | ^1.41.1 | The gen_ai.* attribute constants | | @opentelemetry/exporter-trace-otlp-http | ^0.219.0 | Export to a real backend over OTLP |
The file layout keeps each responsibility in its own module:
otel-mcp-agent-tracing-typescript/ โโโ src/ โ โโโ telemetry.ts # SDK setup: provider, exporters, shutdown โ โโโ tracer.ts # getTracer + the semantic-convention constants โ โโโ llm.ts # LLM interface, FakeLLM, the traced model call โ โโโ mcp-tools.ts # Tool interface, ToolRegistry, the traced tool call โ โโโ agent.ts # the agent loop and its root span โ โโโ types.ts # shared types โ โโโ index.ts # public exports โโโ examples/demo.ts # runs the agent end to end โโโ test/ # vitest: agent, tracer, mcp-tools
Setting up telemetry
Everything starts with turning the SDK on. telemetry.ts exposes a single function, setupTelemetry, that configures a tracer provider, wires an exporter, and hands back a small handle you use to flush and shut down. Crucially it supports three exporters: console for eyeballing the tree during development, otlp for shipping to a real backend, and memory for tests that need to read the spans back as data.
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node'
import {
SimpleSpanProcessor,
ConsoleSpanExporter,
InMemorySpanExporter,
type ReadableSpan,
} from '@opentelemetry/sdk-trace-base'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { resourceFromAttributes } from '@opentelemetry/resources'
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions'
import { trace } from '@opentelemetry/api'
export interface TelemetryHandle {
exporter: 'console' | 'otlp' | 'memory'
serviceName: string
shutdown(): Promise<void>
getFinishedSpans(): ReadableSpan[]
}
export function setupTelemetry(
options: {
serviceName?: string
exporter?: 'console' | 'otlp' | 'memory'
} = {}
): TelemetryHandle {
const serviceName =
options.serviceName ?? process.env.OTEL_SERVICE_NAME ?? 'mcp-agent'
// Auto-select OTLP when an endpoint is configured, else honor the option.
const kind =
options.exporter ??
(process.env.OTEL_EXPORTER_OTLP_ENDPOINT ? 'otlp' : 'console')
const memory = kind === 'memory' ? new InMemorySpanExporter() : null
const exporter =
kind === 'otlp'
? new OTLPTraceExporter()
: (memory ?? new ConsoleSpanExporter())
const provider = new NodeTracerProvider({
resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: serviceName }),
spanProcessors: [new SimpleSpanProcessor(exporter)],
})
// The global tracer provider is process-wide and ignores re-registration,
// so disable any prior one before registering this run's provider.
trace.disable()
provider.register()
return {
exporter: kind,
serviceName,
async shutdown() {
await provider.shutdown()
},
getFinishedSpans() {
return memory ? memory.getFinishedSpans() : []
},
}
}
A few decisions here earn their keep. The exporter selection is the only place environment variables touch the code: set OTEL_EXPORTER_OTLP_ENDPOINT and the agent ships traces to your collector with no code change, which is exactly the "flip a switch in production" property you want. The provider is built with the current constructor form โ a resource describing the service identity and spanProcessors injected directly โ because the older new Resource(...) constructor and addSpanProcessor(...) method are deprecated in the version 2 SDK. And the trace.disable() call before register() is not ceremony: the global tracer provider is a process-wide singleton that silently ignores a second registration, so without disabling first, a test that calls setupTelemetry twice would keep exporting into the first run's memory exporter. That one line is the difference between tests that pass and tests that lie.
We use a SimpleSpanProcessor for clarity โ it exports each span as it ends. In production you would swap in a BatchSpanProcessor to amortize network calls; it is a one-line change and the rest of the code is identical.
The tracer and the conventions
tracer.ts is small but it is where the standards discipline lives. It hands out the tracer, defines the attribute-key constants so the rest of the code never hard-codes a string, and provides one helper for recording errors correctly.
import {
trace,
SpanStatusCode,
type Span,
type Tracer,
} from '@opentelemetry/api'
import {
ATTR_GEN_AI_OPERATION_NAME,
ATTR_GEN_AI_SYSTEM,
ATTR_GEN_AI_REQUEST_MODEL,
ATTR_GEN_AI_USAGE_INPUT_TOKENS,
ATTR_GEN_AI_USAGE_OUTPUT_TOKENS,
ATTR_GEN_AI_TOOL_NAME,
} from '@opentelemetry/semantic-conventions/incubating'
const NAME = 'otel-mcp-agent-tracing-typescript'
export function getTracer(): Tracer {
return trace.getTracer(NAME, '1.0.0')
}
// GenAI keys come from the real (incubating) conventions package.
export const GEN_AI = {
OPERATION_NAME: ATTR_GEN_AI_OPERATION_NAME, // "gen_ai.operation.name"
SYSTEM: ATTR_GEN_AI_SYSTEM, // "gen_ai.system"
REQUEST_MODEL: ATTR_GEN_AI_REQUEST_MODEL, // "gen_ai.request.model"
USAGE_INPUT_TOKENS: ATTR_GEN_AI_USAGE_INPUT_TOKENS,
USAGE_OUTPUT_TOKENS: ATTR_GEN_AI_USAGE_OUTPUT_TOKENS,
TOOL_NAME: ATTR_GEN_AI_TOOL_NAME, // "gen_ai.tool.name"
} as const
// MCP keys are not in the package yet, so we define them as literals.
export const MCP = {
TOOL_NAME: 'mcp.tool.name',
METHOD_NAME: 'mcp.method.name',
} as const
export const AGENT = { STEPS: 'agent.steps', TASK: 'agent.task' } as const
export function recordError(span: Span, err: unknown): void {
const error = err instanceof Error ? err : new Error(String(err))
span.recordException(error)
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message })
}
Two details are worth slowing down on. First, the gen_ai.* constants are imported from the package's /incubating entry point, not its root. In version 1.41 of the conventions package the generative-AI keys are still incubating, so they live behind that subpath; importing the real constants (rather than typing the strings) means your spans automatically track the spec as it stabilizes. Second, the MCP keys are local literals because they are newer than the package โ the convention exists, the constants are not published yet, and writing them as a single MCP object means there is exactly one place to delete when they ship upstream.
The recordError helper exists because recording an error on a span is two distinct acts that are easy to do by halves: attaching the exception as a span event, and setting the span's status to ERROR. Do only the first and the span looks green in your backend despite carrying an exception; do only the second and you lose the stack trace. The helper does both, every time.
Tracing the model call
Now the first real span. llm.ts defines the model interface, a deterministic fake that stands in for a real provider, and tracedChat โ the wrapper that turns a model call into a conventional gen_ai span.
import { SpanKind } from '@opentelemetry/api'
import { getTracer, GEN_AI, recordError } from './tracer.js'
import type { ChatMessage, LLMResponse } from './types.js'
export interface LLM {
readonly model: string
readonly system: string
chat(messages: ChatMessage[]): Promise<LLMResponse>
}
export async function tracedChat(
llm: LLM,
messages: ChatMessage[]
): Promise<LLMResponse> {
const tracer = getTracer()
// Span name follows the convention: the operation plus the model.
return tracer.startActiveSpan(
`chat ${llm.model}`,
{ kind: SpanKind.CLIENT },
async span => {
span.setAttribute(GEN_AI.OPERATION_NAME, 'chat')
span.setAttribute(GEN_AI.SYSTEM, llm.system)
span.setAttribute(GEN_AI.REQUEST_MODEL, llm.model)
try {
const res = await llm.chat(messages)
span.setAttribute(GEN_AI.USAGE_INPUT_TOKENS, res.usage.inputTokens)
span.setAttribute(GEN_AI.USAGE_OUTPUT_TOKENS, res.usage.outputTokens)
return res
} catch (err) {
recordError(span, err)
throw err
} finally {
span.end()
}
}
)
}
The mechanics here are the template for every traced operation in the codebase, so they are worth internalizing. startActiveSpan does two jobs at once: it creates the span and it makes that span the active span for the duration of the callback. That second job is what makes the tree assemble itself โ any span created inside the callback automatically becomes a child of this one, with no manual parent-passing. The attributes are set from the GEN_AI constants, never from raw strings. Token usage is recorded after the call returns, because you do not know it until then. And the try/catch/finally is load-bearing: errors are recorded on the span and rethrown so the caller still sees them, and span.end() runs in finally so a span never leaks open on an error path.
The FakeLLM that implements this interface is deliberately dumb and fully deterministic โ it routes on keywords ("weather" means call the weather tool, "docs" or "cite" means call the doc-search tool, otherwise produce a final answer). That determinism is a feature: it is what lets the tests assert an exact span tree. Replace FakeLLM with a class that calls a real provider and implements the same three-member interface, and not one line of the tracing changes. For the pattern of wrapping a real provider behind a swappable interface like this โ with timeouts, retries, and failover โ see the companion build of a resilient multi-provider LLM client in TypeScript.
Why a deterministic fake
0 API keys
A keyword-routed FakeLLM makes the agent run identically every time, so the test suite can assert the exact span tree instead of mocking a network and hoping.
Tracing the MCP tool call
The tool layer is the part that used to go dark, so this is the span that matters most. mcp-tools.ts defines a tool, a registry that runs tools, and โ the whole point โ wraps each invocation in an mcp.tool span.
import { SpanKind } from '@opentelemetry/api'
import { getTracer, GEN_AI, MCP, recordError } from './tracer.js'
export interface Tool {
name: string
description: string
execute(args: Record<string, unknown>): Promise<unknown>
}
export class ToolRegistry {
private readonly tools = new Map<string, Tool>()
register(tool: Tool): this {
this.tools.set(tool.name, tool)
return this
}
list(): Tool[] {
return [...this.tools.values()]
}
has(name: string): boolean {
return this.tools.has(name)
}
async call(name: string, args: Record<string, unknown>): Promise<unknown> {
const tool = this.tools.get(name)
if (!tool) throw new Error(`Unknown tool: ${name}`)
const tracer = getTracer()
return tracer.startActiveSpan(
`mcp.tool/${name}`,
{ kind: SpanKind.CLIENT },
async span => {
span.setAttribute(MCP.TOOL_NAME, name)
span.setAttribute(MCP.METHOD_NAME, 'tools/call')
span.setAttribute(GEN_AI.TOOL_NAME, name)
span.setAttribute(GEN_AI.OPERATION_NAME, 'execute_tool')
try {
const result = await tool.execute(args)
span.setAttribute('mcp.tool.result', JSON.stringify(result))
return result
} catch (err) {
recordError(span, err)
throw err // surface the failure to the agent loop
} finally {
span.end()
}
}
)
}
}
The span name mcp.tool/<name> keeps every tool call grouped and greppable while still distinguishing which tool ran. The attributes do double duty: the mcp.* keys describe the call in Model Context Protocol terms (mcp.method.name is "tools/call", the MCP method that actually executes a tool), and the gen_ai.tool.name plus gen_ai.operation.name keys describe the same call in the generative-AI vocabulary, so backends that only understand one of the two conventions still get a useful span. The result is stringified onto the span so you can see what the tool returned โ which is the single most useful thing to have when the model misreads a tool result.
The error path is the reason this matters. One of the bundled tools, get_weather, throws when handed a blank city. When that happens the catch block calls recordError, the span gets ERROR status and a recorded exception, and the error is rethrown so the agent loop can decide what to do. In your backend that tool call shows up red, named, with the exception attached โ instead of vanishing into a model span that simply produced a worse answer.
What the instrumentation now covers
The agent loop
With both leaf spans in place, the loop is almost anticlimactic โ which is the sign the abstraction is right. agent.ts opens the root span and runs think-act-observe until the model stops asking for tools.
import { SpanKind } from '@opentelemetry/api'
import { getTracer, AGENT, recordError } from './tracer.js'
import { tracedChat, type LLM } from './llm.js'
import { ToolRegistry } from './mcp-tools.js'
import type { AgentResult, ChatMessage } from './types.js'
export class Agent {
private readonly llm: LLM
private readonly tools: ToolRegistry
private readonly maxSteps: number
constructor(opts: { llm: LLM; tools: ToolRegistry; maxSteps?: number }) {
this.llm = opts.llm
this.tools = opts.tools
this.maxSteps = opts.maxSteps ?? 6
}
async run(task: string): Promise<AgentResult> {
const tracer = getTracer()
return tracer.startActiveSpan(
'agent.invoke',
{ kind: SpanKind.INTERNAL },
async root => {
root.setAttribute(AGENT.TASK, task)
const messages: ChatMessage[] = [{ role: 'user', content: task }]
let steps = 0
try {
while (steps < this.maxSteps) {
steps++
const reply = await tracedChat(this.llm, messages) // child span
if (reply.toolCall) {
const result = await this.tools.call(
reply.toolCall.name,
reply.toolCall.args
) // child span, nested under the root
messages.push({
role: 'tool',
content: JSON.stringify(result),
})
continue
}
root.setAttribute(AGENT.STEPS, steps)
return { answer: reply.content, steps }
}
root.setAttribute(AGENT.STEPS, steps)
return { answer: 'Step budget exhausted.', steps }
} catch (err) {
recordError(root, err)
throw err
} finally {
root.end()
}
}
)
}
}
Because the model call and the tool call both use startActiveSpan and both run inside the root's startActiveSpan callback, the nesting happens for free: every chat span and every mcp.tool span lands as a child of agent.invoke, sharing its trace id. The maxSteps guard is the loop's safety belt โ it is what turns a runaway "call the same tool forever" loop into a bounded run that ends with an honest "step budget exhausted" instead of hanging. And agent.steps on the root span gives you an at-a-glance answer to "how much work did this take," which is often the first number you want when a run feels too slow or too expensive. When you graduate from one agent to many, this same root-span-per-unit discipline is what keeps a fleet legible; the parallel subagent orchestrator tutorial extends the idea to concurrent workers.
Cumulative traced operations across the demo run (3 steps)
| step | calls |
|---|---|
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
Running it
npm run demo builds an agent from the FakeLLM and the two bundled tools, runs it on a sample task, and prints the resulting spans to the console. With the task "What's the weather in Paris, and cite the docs?" the agent takes three steps โ a model call that asks for the weather tool, the weather tool call, a model call that asks for doc search, the doc-search tool call, and a final model call that answers โ and the console exporter prints each span as it ends.
{ name: 'chat fake-router-v1', kind: 2,
attributes: { 'gen_ai.operation.name': 'chat', 'gen_ai.system': 'fake',
'gen_ai.request.model': 'fake-router-v1',
'gen_ai.usage.input_tokens': 14, 'gen_ai.usage.output_tokens': 9 } }
{ name: 'mcp.tool/get_weather', kind: 2,
attributes: { 'mcp.tool.name': 'get_weather', 'mcp.method.name': 'tools/call',
'gen_ai.tool.name': 'get_weather',
'gen_ai.operation.name': 'execute_tool',
'mcp.tool.result': '{"city":"Paris","summary":"partly cloudy, 20C"}' } }
{ name: 'mcp.tool/search_docs', kind: 2, attributes: { 'mcp.tool.name': 'search_docs', ... } }
{ name: 'agent.invoke', kind: 1,
attributes: { 'agent.task': "What's the weather in Paris, and cite the docs?",
'agent.steps': 3 } }
Final answer: The weather in Paris is partly cloudy at 20C. Per the docs:
OpenTelemetry models an agent run as a trace: one root span per invocation with
child spans for each model call and tool call.
Steps taken: 3
Notice the order. The leaf spans print first because they end first; the agent.invoke root prints last because it does not end until its children are done. In a real backend you would not read this as text โ you would see the tree rendered, root at the top, the two tool calls and three model calls nested beneath it, the whole run reconstructable at a glance. Point the agent at a collector by setting OTEL_EXPORTER_OTLP_ENDPOINT, and the same spans flow to Jaeger, Grafana Tempo, Datadog, or anything else that speaks OTLP.
Testing the span tree
Here is the payoff for all that determinism: you can test the shape of the observability itself. Because setupTelemetry({ exporter: 'memory' }) returns an in-memory exporter, a test can run the agent and then read the finished spans back as plain objects and assert against them.
import { describe, it, expect, afterEach } from 'vitest'
import {
setupTelemetry,
Agent,
FakeLLM,
createDefaultToolRegistry,
} from '../src/index.js'
describe('agent span tree', () => {
let handle: ReturnType<typeof setupTelemetry>
afterEach(async () => {
await handle?.shutdown()
})
it('nests model and tool spans under one agent.invoke root', async () => {
handle = setupTelemetry({ exporter: 'memory' })
const agent = new Agent({
llm: new FakeLLM(),
tools: createDefaultToolRegistry(),
})
const result = await agent.run(
"What's the weather in Paris, and cite the docs?"
)
expect(result.steps).toBe(3)
const spans = handle.getFinishedSpans()
const root = spans.find(s => s.name === 'agent.invoke')!
const children = spans.filter(
s => s.parentSpanContext?.spanId === root.spanContext().spanId
)
// Every child shares the root's trace id...
for (const child of children) {
expect(child.spanContext().traceId).toBe(root.spanContext().traceId)
}
// ...and the children are the model calls and the tool calls.
expect(children.some(s => s.name.startsWith('chat '))).toBe(true)
expect(children.some(s => s.name.startsWith('mcp.tool/'))).toBe(true)
})
})
This is a genuinely different kind of test from the unit tests you are used to. It does not assert what the agent answered โ it asserts that the agent is observable: that the run produced a single root, that every span belongs to the same trace, and that the model and tool calls are nested where a backend expects them. A separate test reads the gen_ai.* attributes off the model span and checks the exact keys and values; another drives the get_weather tool with a blank city and asserts that the resulting span carries ERROR status and a recorded exception, and that call() rethrows. The full suite is eight tests across three files, and it runs in about a second with no network.
Test suite
8 passing
Span-tree nesting, shared trace id, gen_ai attribute keys and values, and the tool error path โ all asserted against an in-memory exporter, no API keys, about one second.
One Vitest configuration detail makes this reliable. The OpenTelemetry global tracer provider is process-wide, so two test files registering their own in-memory exporters can race. Pinning Vitest to a single fork (fileParallelism: false) makes each test own its own provider cleanly, which is a small price for tests that assert telemetry without flaking.
Production considerations
The tutorial code is honest but minimal. Three changes turn it into something you would run for real.
Mind what the spans capture. This is the big one. Prompts and tool arguments are exactly the sensitive data you do not want sprayed across a tracing backend โ they can contain user messages, secrets, and PII. The conventions anticipate this: capturing message content is opt-in, and you should keep it that way. Record the shape of a call by default (which model, which tool, how many tokens, success or failure) and gate the content (the actual prompt text, the raw arguments) behind an explicit, environment-controlled flag that is off in production unless someone deliberately turns it on for a debugging session.
Batch and sample. Swap the SimpleSpanProcessor for a BatchSpanProcessor so exports do not add latency to every span, and add a sampler if your volume is high โ though for agent traces, where each run is rare and expensive relative to a web request, many teams sample at one hundred percent because every run is worth keeping.
Treat instrumentation as default-on. The endgame is that you do not remember to add tracing; it is simply on, the way request tracing is on in a modern web framework. That is also where the standards are headed โ I lay out the case in my prediction that OpenTelemetry's GenAI and MCP conventions reach Stable and ship default-on in major agent frameworks by the end of 2027. There is a compliance dimension too: regulations that require reconstructing an AI-assisted decision after the fact are, functionally, asking for exactly this span tree.
A rollout path from console to production
Console exporter
Wrap the agent loop, read the tree in your terminal, confirm the nesting is right.
OTLP to a backend
Set OTEL_EXPORTER_OTLP_ENDPOINT, batch the processor, and view real traces in Jaeger, Tempo, or Datadog.
Content gating + sampling
Keep prompt and argument content opt-in; decide a sampling rate; record only span shape by default.
Default-on
Bake the instrumentation into your agent harness so every run is observable without anyone remembering to add it.
The discipline pays a human dividend too. An agent that runs unattended and records everything it does is honest in a way an uninstrumented one cannot be โ though "honest" and "controllable" are not the same thing, a gap I explored in a short story about an engineer reading her agent's overnight trace, The Overnight Run. A trace tells you what happened. It is on you to decide, in advance, what the agent is allowed to do.
Conclusion
You now have the full pattern for making a tool-using agent observable: a tracer and a set of convention constants, a setupTelemetry switch that exports to your console in development and your collector in production, a traced model call and a traced tool call that follow the gen_ai.* and MCP conventions, an agent loop whose root span gathers it all into one tree, and tests that assert the shape of that tree without a single API key. The instrumentation is maybe a hundred lines; the leverage is that any agent loop you write from here can adopt it wholesale, and the traces it emits are portable to any backend you ever choose.
The autonomy is already here. The observability is the half that decides whether you can run it where it counts.
Related resources
- Companion code on GitHub โ the full, tested otel-mcp-agent-tracing-typescript project from this tutorial.
- The agent-observability turn โ Gemini Spark and OpenTelemetry's MCP spans โ the market context for why agent tracing landed all at once this week.
- Prediction: OpenTelemetry's GenAI and MCP conventions reach Stable and ship default-on by end of 2027 โ my dated, falsifiable claim on where this standardizes.
- Build a parallel subagent orchestrator in TypeScript โ the same root-span discipline applied to a fleet of concurrent agents.
- The Overnight Run โ a short story on the difference between watching an agent and controlling it.
