1 September 2026

 

An AI agent receives an invoice, checks it against a purchase order and prepares the payment. The supplier details look plausible, the amount falls within its authorised limit and the agent has access to the company’s payment system.

But somewhere inside the AI model, the information does not quite fit. Perhaps the invoice conflicts with an earlier document. Perhaps its reasoning has been diverted by a malicious instruction hidden in an attachment. The agent proceeds anyway.

Most companies could examine what happened only afterwards. They might inspect the prompt, the output and a log showing which tools the agent used. What they cannot usually see is whether the model displayed internal signs of confusion or manipulation before approving the payment.

A young field known as mechanistic interpretability is attempting to reveal such signals. Instead of observing an AI model solely from the outside, researchers examine its internal activations: the constantly changing numerical patterns created as information passes through a neural network.

The ambition is not merely to explain why a model failed after the event. Developers hope eventually to identify dangerous behaviour as it forms, and intervene before an AI agent clicks “approve”.

 

Goodfire turns AI interpretability into a business

 

In February 2026, San Francisco-based Goodfire raised $150 million in Series B funding at a reported valuation of $1.25 billion. The round followed a $50 million Series A less than a year earlier and brought the company’s total backing to more than $200 million. Investors include B Capital, Menlo Ventures, Lightspeed Venture Partners and Salesforce Ventures.

Goodfire’s proposition is that neural networks should eventually be understood, debugged and deliberately shaped more like conventional software. Its team includes researchers who previously worked on AI interpretability at OpenAI and Google DeepMind.

The company is developing tools that inspect a model’s internal representations, find components associated with particular behaviours and use those findings to alter how the model performs.

In one experiment, Goodfire used probes trained on internal model signals as rewards during reinforcement learning. It reported a 58% reduction in hallucinations on a held-out test set using a particular sampling and monitoring setup, without a decline on its standard performance benchmarks.

The company has also applied interpretability to scientific models. In work with Prima Mente, researchers examined an epigenetic foundation model and identified DNA fragment-length patterns as potential blood-based biomarkers for Alzheimer’s detection. A simple model based on these features was tested on an independent cohort, but the study was small and its authors describe it as a pilot requiring further validation. The findings are research hypotheses, not a clinical diagnostic tool.

Goodfire's Silico AI research agent is available to academic and non-profit researchers for $1,000 a month, while commercial team pricing is negotiated separately. Organisations can run it on Goodfire’s infrastructure or connect their own computing cluster. 

That makes Silico a commercial product, although not yet a conventional AI compliance dashboard that can be handed to a risk department. It is designed primarily for machine-learning researchers and organisations developing or adapting their own models. It can plan and run long interpretability experiments, train probes and sparse autoencoders, perform causal analyses and generate reports. 

 

How does mechanistic interpretability work?

 

Neural networks do not store concepts in neatly labelled locations. A single artificial neuron can respond to several unrelated ideas, while one concept may be distributed across many neurons.

Researchers therefore use techniques including sparse autoencoders. These are secondary neural networks trained to decompose dense internal activations into features that may be easier for humans to interpret. One feature might appear to respond to medical diagnoses, computer code or a particular writing style.

The technique is often described as placing a model under a microscope. The comparison is useful, provided nobody mistakes a microscope for a mind reader.

Suppose the invoice-processing agent activates a feature associated with suspicious instructions. That does not prove it has detected fraud, nor that it is about to ignore a warning. Researchers must determine what the feature represents, whether it appears consistently and whether changing it alters the model’s behaviour.

Correlation alone is not an explanation. A company could otherwise end up replacing one black box with a colourful dashboard that creates only the impression of transparency.

Google DeepMind has helped make mechanistic interpretability more accessible through Gemma Scope. The original release provided more than 400 freely available sparse autoencoders covering the layers and sublayers of its Gemma 2 2B and 9B models. Together, they contained more than 30 million learned features, although Google noted that many probably overlap.

Gemma Scope 2 extended the approach across the Gemma 3 model family, adding several types of transcoders intended to help researchers trace computations within and between layers. The tools support research into behaviours including hallucinations, jailbreaks, refusal mechanisms and whether a model’s stated chain of thought reflects its internal processing. They do not automatically deliver complete explanations of those behaviours.

This work is easier with open-weight AI models because researchers can access their parameters and activations. A company using a closed model solely through an application programming interface normally sees only what enters and leaves the system. Unless the provider exposes internal information or performs the analysis itself, customers cannot conduct the same inspections.

 

AI interpretability, observability and governance are not the same

 

The commercial opportunity becomes clearer when mechanistic interpretability is separated from two markets that already surround enterprise AI.

AI observability tools monitor inputs, outputs, latency, errors and sometimes the steps taken by an AI agent. They can show that the invoice agent called a payment system after reading a particular document.

AI governance platforms maintain model inventories, assign responsibility, organise evaluations and store evidence for auditors. They can show who approved the agent, which policies apply and whether its performance was tested.

Mechanistic interpretability goes deeper. It attempts to identify internal features and computational pathways associated with a model’s behaviour.

The first two layers are already becoming established enterprise software categories. The third remains experimental. Its commercial promise lies in revealing risks that external AI monitoring might miss, but its findings are harder to interpret and validate.

The most useful future system may combine all three. A company would know what an agent was allowed to do, retain a record of what it actually did and receive an additional warning when internal signals indicated that its behaviour was moving in an unsafe direction.

 

Anthropic traces the circuits inside AI models

 

Anthropic has produced some of the field’s most visible mechanistic interpretability research. Its work goes beyond identifying individual features and attempts to trace the circuits connecting them.

In 2025, the company demonstrated attribution graphs that partially reconstruct internal pathways contributing to a model’s answer. Researchers found evidence of shared conceptual processing across languages and examined how Claude solved some reasoning problems.

In a poetry experiment, the model appeared to select a possible rhyme in advance and then construct a line leading towards it. Intervening on the internal representation changed the planned ending.

Anthropic subsequently released circuit-tracing tools for supported open-weight models. Through an interface developed with Neuronpedia and Decode Research, users can generate graphs, examine internal features and change feature values to test how they affect an output. Anthropic is careful to say that these graphs reveal internal processing only partially.

The company has also investigated “persona vectors”: activation patterns associated with tendencies such as sycophancy, hallucination or harmful behaviour.

In experiments on Qwen 2.5-7B-Instruct and Llama 3.1-8B-Instruct, researchers used the vectors to monitor personality changes, alter model behaviour and identify training data likely to encourage unwanted traits. The tests therefore concerned two open models, rather than demonstrating a deployed monitoring feature in Claude.

Later work identified an “assistant axis” associated with whether a model retained its trained assistant character or moved towards alternative personas during a conversation. Anthropic demonstrated that constraining movement along this axis could reduce harmful persona drift in experiments, including tests on Llama 3.3 70B.

Neuronpedia provides open infrastructure through which many of these findings can be explored. Operated by Decode Research, it hosts features, probes, circuit tracers and visualisation tools from several research groups. It is closer to a shared research laboratory than an enterprise compliance product, but lowers the technical barrier to interpretability experiments.

Commercial entrant Realm Labs is taking a more operational approach. The company says its Deep Neural Inspection technology extracts signals from inside models during inference and uses them for real-time monitoring and enforcement.

Realm positions this as an alternative to AI safety systems that inspect only prompts and responses. Its production-scale claims are potentially significant, although detailed independent evidence across different models and business environments is not yet publicly available.

 

Can companies stop an AI agent before it acts?

 

AI agents may provide the first compelling corporate use case for mechanistic interpretability.

An ordinary chatbot produces text. An AI agent can search databases, alter records, send messages, write code or initiate payments. Discovering the reason for a failure after the action may offer little comfort.

A 2026 preprint applied sparse autoencoders and linear probes to model states before an agent called a tool. The researchers tested whether those internal signals could indicate if the model recognised that a tool was required and how consequential its next action might be.

The experiments used open models and a function-calling dataset; they did not demonstrate a production system stopping real enterprise agents.

Now let's return to the questionable invoice. Permissions might prevent the agent from transferring more than a set amount. Logging would record which files it consulted. A human approval rule could hold the payment for review. Mechanistic interpretability might eventually add another signal: the model internally registered a conflict or suspicious instruction before deciding to proceed.

That signal would not prove fraud. It could, however, be enough to suspend the transaction and ask a human to look again.

For now, conventional controls remain more dependable. Companies still need restricted permissions, human approval for consequential actions, independent evaluations and detailed audit logs. Mechanistic interpretability can complement those safeguards, but it cannot certify that a model is safe or provide a definitive account of every decision.

Nor does an attractive visualisation automatically constitute regulatory evidence. Compliance officers need findings that are reproducible, relevant to the deployed model and understandable to auditors. Interpretability methods have yet to establish consistent standards for demonstrating all three.

Companies cannot finally see exactly what their AI is thinking. They are beginning to see fragments: features, pathways and internal warning signals that were previously inaccessible.

The first commercial success of mechanistic interpretability may not be explaining every thought inside a neural network. It may be recognising the moment when an AI agent is about to do something its employer will regret.

 

 

What is mechanistic interpretability?

Mechanistic interpretability is the study of how neural networks perform computations internally. Researchers examine activations, features and circuits to understand which internal mechanisms contribute to an AI model’s behaviour.

Can mechanistic interpretability explain every AI decision?

No. Current methods can reveal parts of a model’s internal processing, but they cannot yet provide a complete or consistently reliable explanation for every output or decision.

How could companies use AI interpretability?

Potential applications include debugging models, detecting hallucination-related signals, monitoring AI agents, analysing training data and identifying behavioural changes that conventional output monitoring may miss.

Is mechanistic interpretability an AI compliance tool?

Not by itself. It may eventually provide additional evidence for audits and risk assessments, but companies still need governance processes, evaluations, access controls, human oversight and production monitoring.

 

 

 

Liked this article? You can support our independent journalism via our page on Buy Me a Coffee. It helps keep MoveTheNeedle.news focused on depth, not clicks.

👉 https://buymeacoffee.com/movetheneedle.news