With AI, what's old is new in telemetry again

I remember a demo with an observability vendor back in 2019. We were setting up a big data app, and they showed us a cool topology map of a request, how it flowed through multiple API’s, and ultimately queried a database we were configuring. The demo was great, but it turns out that setting up tracing throughout the stack was substantially more difficult than we’d planned, and it cost a lot more to run with all of the data it was generating. Ultimately, we shelved the work.

So why is tracing so important now with AI?

An example of two requests

Let’s say there is an AI agent (a coding agent, support agent, doesn’t really matter) that intakes a request (intent) from a user, has an internal agentic loop, and ultimately should satisfy the request (intent).

This is a very simple example of the flow, where the agent is performing work. Let’s assume that you’ll have multiple users or requests that match intents. Ideally, the flow of each session looks like this:

This would be an ideal state where similar or matched intents, when clustered together, yield similar or matched outcomes when the agent does work.

However, when you factor in model non-determinism, various tool calls and definitions, and other things like specific context or data the model has access to, this ends up looking something more like:

So how do you figure out what went wrong? We could observe some things our agent may have performed like tool calls out to other API’s or databases, but we need to look at the trajectory and the reasoning chain that yielded a particular good or bad outcome, analyze them, and then modify our agent or other components.

Why does tracing help?

In the example above, we need to see the internal and external work the agent performed, the timelines the work was performed in, and in what order the work was done for a particular intent or outcome. Tracing gives us the ability to string all of these together. Even better, we can enable tracing that gives us the full prompts and responses for each turn of the agent so we can see, in detail, what actually happened and when.

In 2019, this was a nice to have, but this is becoming a requirement to budget for and implement when working with agents. Not only does it help for building better agents, but it improves the observability for things like regulations when we need to show why agents perform a particular task and what they did.

Tracing applied to our example

Let’s assume we’ve added tracing to our agent example. You can do this with OpenTelemetry and built in extensions in most agent frameworks (Google ADK example here).

Now, let’s assume we have two users that asked a question about their bill last month, and why it was so high.

Using tracing, we now have access to more data points. For example, we could see that for one customer, the agent may have properly found the customer’s information, but the bill for last month had not been processed yet. As a result, the agent only had access to information that was two months old, and the resulting outcome wasn’t correct due to data access.

In another example, we have a coding agent. In one flow, the coding agent had access to a repository but chose to use grep to search for the changes required instead of using an AST or tree based tool to find functions. By having access to the internal thoughts and steps, we can determine if, in this example, we need to further tune our MCP tool definitions or even our skill so the agent is better equipped for what to use and when.

And, we can use LLMs to improve LLM quality by providing all of this information to agents with specific tasks.

Implementing tracing

In the observability stack, access to real time traces, metrics, and logs are table-stakes for agentic development and runtimes.

Look in the frameworks you’re using to start seeing how you can implement the default tracing abilities. Keep in mind that a substantial portion of agentic system cost is actually on the observability and control around the agent, not the agent itself. Budget accordingly.

And last, make sure you’re thinking about data security. When you are exporting traces, in most cases you will need the full logs to figure out what happened. But that doesn’t mean you should necessarily give everyone or an improvement agent access to personally identifiable data, and you should be aware what you can use for training and improvement in your customer agreements and regulations (ex GDPR).

Hope you enjoyed this weeks post!

$ uv run python -m agent_telemetry.lineage --commit 9f41c8a2 --format summary
LINEAGE RECORD: commit=9f41c8a2 -> trace_id=4bf92f3577b34da6a3ce929d0e0e4736
MODEL: gemini-3.8-flash · STEPS: 6 · TOOL CALLS: 4 (read_file, edit_file, pytest)
VERIFICATION GATE: PASSED (100% pass rate in 1.42s)