Explainable AI for agents What medical imaging taught me about trusting models

An agent is software that takes a goal, plans the steps and gets on with them. It reads documents, remembers things from earlier conversations, uses other tools and writes into other systems. My team builds them for regulated work like compliance and banking, where a person has to sign off on what the agent did.

Most agents run on a model that belongs to someone else, so you can’t open it up and look. A single run can take dozens of steps, and some of them change things in the real world. On top of that, the same request can come back with a different answer the next time you send it.

In my cardiac MRI research I could look inside the model. With an agent, I mostly have to work from the outside.

When people talk about trusting agents, three different questions tend to get mixed up. I mixed them up myself for a while.

What happened?

A trace, the step-by-step log of a run, records every document the agent read and every tool it used.

Was it right?

Evals, which are tests for AI output, grade the result and sometimes the path that led to it.

Why did it do that?

Which input actually drove the decision. This is the hard one.

Most tools sold as “AI observability” answer the first two questions, and that’s valuable. A trace can show that the agent read a policy document before it flagged a payment. It can’t show whether that document is the reason it flagged it.

Even the plain record of what happened catches things a score misses. The researchers behind the Holistic Agent Leaderboard read through the logs of more than 21,000 agent runs and found agents searching online for the test itself instead of doing the task, and agents misusing credit cards while booking flights. None of that showed up in the scores.

This is the habit from my cardiac MRI research that I use most. With the scans, we hid one CNN component and watched how far the answer moved. With an agent, you do the same thing with whatever it relied on.

Everything an agent decides is built from inputs you can list: the request, the documents it found, what it pulled from memory and what its tools sent back. Remove one, run it again and compare. If the decision holds, that input didn’t matter much. If it flips, you’ve found something the agent depended on, whatever it says about its own reasoning.

One agent I ran helped a distribution company analyse and optimise its inventory, using a handful of tools. It kept failing, and the cause turned out to be one of those tools: the one that logged the agent in to the company’s ERP, the system that runs its stock and orders, had been duplicated. Once we isolated that tool, the failures stopped.

It also pays to log what the agent looked at and set aside, like the nine documents it retrieved and then ignored. In my cardiac MRI research, seeing what the model pushed against made the heatmap far more believable. With agents, those choices often tell me more about its judgement than the final answer does.

Newer models write out their thinking before they answer. It reads like an explanation, and it’s tempting to treat it as one.

In 2025, Anthropic tested this by slipping models a hint and checking whether their written reasoning admitted to using it. Claude 3.7 Sonnet mentioned a hint it had used only 25% of the time, and DeepSeek R1 39% of the time. Those were that year’s models and newer ones may do better, but I treat an agent’s reasoning the way I came to treat our heatmaps, as a useful sketch that still needs checking.

For a long time the honest answer to “what is the model thinking?” was that nobody could really say. A language model’s insides are billions of numbers, and on their own those numbers mean nothing to a person.

In 2025 Anthropic published a method called circuit tracing that starts to change that. Its researchers describe the work as a kind of biology, studying a model the way you’d study cells under a microscope. The method swaps part of the model for a simpler stand-in made of features, small pieces you can put a name to, like “Texas” or “capital city”. Then it draws an attribution graph, a map of which features fed into which on the way from the question to the answer.

The best-known example is a geography question. Asked to finish “the capital of the state containing Dallas is”, Claude 3.5 Haiku answered Austin, and the graph showed it working out Texas first, then using Texas to find the capital. When the researchers swapped the Texas feature for California, the answer changed to Sacramento.

That’s the habit from my cardiac MRI research again, change one piece and watch the answer move, only this time inside the model. It also turns up things you’d never learn from a model’s written reasoning. When finishing a rhyming couplet, the model often chose the rhyming word first and then wrote a line that would land on it.

In May 2025 Anthropic open-sourced the tools. They work with open models such as Google’s Gemma and Meta’s Llama, and Neuronpedia hosts an interactive version where you can type a prompt, look at its graph and nudge a feature to see what happens to the answer.

Anthropic is upfront that this is early work. By the team’s own estimate, the graphs gave a satisfying explanation for about a quarter of the prompts they tried, and the stand-in is only an approximation of the real model. The bigger catch for anyone building agents is that it needs the model’s weights, so it can’t be pointed at the closed models most agents run on, and it explains a single answer where an agent run can take dozens of steps.

Where I expect it to pay off first is in the small open models teams run themselves, like the guard models that check what goes in and out of an agent. Everywhere else, checking still means changing the inputs and watching what moves. It reminds me of our heatmaps, which were useful and still only ever showed part of the picture.

My cardiac MRI research left me one more lesson. The version of our model with the better average score also had a worst single mistake about four times larger than the other version’s. An average can hide one bad miss.

Agents have the same problem. Teams usually describe an agent by its success rate, say 90%. That sounds safe, but it hides what happens when you use the agent every day.

Think of a bus that’s on time 90% of the time. On any one day you’ll probably be fine. Over eight days, though, the chance that it’s on time every single day is only 43%, because each new day is one more chance for it to be late. So in most eight-day stretches, you’ll get at least one late bus.

An agent with a 90% success rate works the same way. A business that runs it every day will see it go wrong at least once in most eight-day stretches. You can put in your own agent’s numbers below.

Real agents are a little different from the bus. Their mistakes usually aren’t random, and they tend to trip on the same kinds of task again and again. So instead of everyone having a bad day now and then, some customers get a broken experience every single time. Either way, the success rate on its own hides what you’d most want to know before you ship.

Send the same question to a language model twice and you can get two different answers, even with temperature set to zero. Temperature is the setting that controls how adventurous a model is when it picks its next word. At zero it’s supposed to always pick the single most likely word, so the same question should always get the same answer.

In 2025 Thinking Machines Lab put that to the test. They asked one large model the same question 1,000 times at temperature zero and got back 80 different answers.

The cause was surprisingly mundane. Computers round numbers off a little as they add them up, so adding the same numbers in a different order can give a very slightly different total, a bit like totting up a long bill and rounding at every step. Providers also process many people’s requests together in batches, and how busy the server is changes the batch size, which changes the order the sums are done in. Usually the difference is too small to matter. When two words are almost tied for most likely, though, it can tip the choice, and once one word changes, the rest of the answer can drift with it.

Their fix was to make the maths run the same way whatever the batch size, an approach they call batch-invariant kernels. With it switched on, all 1,000 answers came back identical. It does cost some speed: one of their tests took 42 seconds instead of 26. Open-source tools for running models, such as vLLM and SGLang, now offer a deterministic mode built on the same idea.

For agents this matters more than for a chatbot. One different word early on can send an agent down a different path for every step after it, and a failure you can’t reproduce is very hard to fix. The catch is that you can only switch this on if you run the model yourself. On someone else’s API, the batching is up to them.

So I make the system around the model repeatable. I pin the exact model version wherever the provider allows it, record real responses and replay them in tests so a failing test means our own code changed, and judge the agent over many runs instead of one. I also keep the design as simple as the job allows, because every extra tool or sub-agent is one more thing to explain later. Even in my cardiac MRI research, adding more machinery made one of our models slightly worse.

I enjoy the irony. I’ve written before about what AI is doing to tech jobs, and in agent work testing has become most of the job. You write evals, read traces and chase failures, and each failure becomes a new test so it can’t come back unnoticed. Evals now run before every release the way unit tests do, and some teams have agents write their tests as fixed scripts, so a test runs the same way every time.

On a good team, the explanation for a failure usually turns up as a failing test with the trace attached, plus someone willing to re-run it with one input taken away.

A score tells you how often a model is right. An explanation is what lets you decide whether to rely on it. Agents haven’t changed that; they’ve pushed the explanation outside the model and made checking it part of everyday work.

If you’re putting agents in front of customers, I’d like to know which of those questions your team would struggle with most. Send me a message on LinkedIn or X and tell me how you’re handling it.