I've been spending time trying to understand AI safety. One area kept coming up: mechanistic interpretability. Mech interp, for short.
The premise is simple. We build these models, we train them, they do impressive things, and we have almost no idea how. We control what goes in and we can see what comes out. What happens in the middle is the black box. Mechanistic interpretability is the attempt to open it.
Here's what I learned from Week 4 of the Bluedot Impact Technical AI Safety course.
We build AI models, train them on enormous amounts of data, and they do impressive things. But ask most researchers what is actually happening inside one of these models when it produces an answer, and the honest response is, "we don't really know". We control what goes in. We can observe what comes out. What happens in between is largely a mystery. Mechanistic interpretability is the field trying to fix that.
The goal is to understand the internal reasoning processes of trained neural networks. Not just what a model outputs, but why. What is it actually doing when it reads a sentence, makes a prediction, or decides to refuse a request?
Features and Circuits
To talk about what is inside a neural network, researchers use two core concepts: features and circuits.
A feature is something the network has learned to recognise. Something like the model's vocabulary of concepts. A vision model trained on images might develop features for things like "curve," "fur," or "brown colour." A language model might develop features for "aggression," "formality," or "deception." Features tend to live within a single layer of the network.
Circuits are what connect features together. They are the computational relationships that allow a network to combine simple, low-level concepts into more complex ones across layers. A circuit might take the features for "curve," "fur," and "brown colour" and combine them into a higher-level feature, "dog ear." In a later layer, another circuit combines "dog ear," "dog eyes," and "snout" into "dog head." This process keeps happening as information moves deeper into the network, which is why deeper models can represent such abstract and sophisticated ideas.
A useful way to hold the distinction is features are what a model knows, and circuits are how it learns to know more complex things.
There is also a hypothesis worth knowing about called the universality hypothesis. It proposes that different neural networks, trained completely independently, tend to converge on similar features and circuits. Early evidence supports this. Researchers have found similar low-level features like curve detectors appearing across multiple vision models built with different architectures. If the hypothesis holds, it would be significant for the field because understanding one model would transfer to understanding others.
Why is Mech Interpretability Hard?
There are two problems that make this work genuinely difficult.
The first is that identifying what a neuron represents takes a lot of effort. You might show a neuron thousands of images and notice it fires most often for pictures of dogs. The natural conclusion is that it represents "dog" as a concept. But it might actually be responding to snouts, or a particular texture of fur, or some other pattern the dog images share. Ruling out the alternatives requires careful, repeated experimentation. And this is for a single neuron in a network that might have billions.
The second problem is deeper. A single neuron can sometimes represent multiple completely unrelated things. In a well-known example, a neuron in the vision model InceptionV1 responded to both cat faces and the fronts of cars. Two visually dissimilar concepts, sharing one neuron. This is called polysemanticity.
The leading explanation for why polysemanticity happens is superposition. The argument is that a network needs to represent far more useful features than it has neurons available, so it compresses by letting multiple features share a single neuron. This works because most features are sparse, meaning on any given input, only a small number of them are active at once. Two features that rarely appear together can share a neuron without causing too much interference.
The safety implications are significant. A neuron might be representing something harmless the vast majority of the time, and something dangerous in rare cases. Because they share the same neuron, disentangling one from the other is extremely hard.
How are Researchers Trying to Fix it?
Two main approaches have been tried.
The first is to train models not to use superposition from the start. This has been shown to work in small experimental models. The problem is that superposition is actually part of what makes large models capable. It allows a network to represent vastly more concepts than its neuron count would otherwise allow. Removing it would likely make frontier models significantly worse at their jobs, which no competitive lab is going to accept.
The second approach is sparse autoencoders. A sparse autoencoder is itself a neural network, but one designed specifically to interpret another model. It is trained on top of the model you are trying to understand, and it has more neurons than the original. The key constraint is that it is forced to keep most of its neurons inactive at any given time. This pushes individual neurons to specialise, so each one ends up representing a single clear concept rather than a mixture of several. Researchers call this monosemanticity.
Anthropic applied this technique to Claude 3 Sonnet and identified millions of distinct features, including ones relevant to safety like deception, discrimination, and misuse. They also demonstrated something called feature steering, where they directly amplified or weakened specific features to influence the model's behaviour. When they artificially increased the activation of the "Golden Gate Bridge" feature, the model began referring to itself as the bridge. It is an unusual demonstration, but it points at something important. Interpretability is not just about understanding a model. It is potentially about controlling it.
Why Does Mech Int Matter?
Accountability. AI is increasingly being used to make consequential decisions in areas like law, healthcare, and hiring. When a model rejects a loan application or screens out a job candidate, understanding why it made that decision matters enormously. Interpretability tools could expose cases where a model is discriminating based on race or gender, the kind of bias that would be invisible if you could only observe outputs.
Safety. A model might appear well-behaved while actually pursuing goals that are not aligned with its users. It might tell people what they want to hear, find shortcuts to game its training, or behave helpfully during evaluation and differently when deployed. If you can only observe outputs, detecting these tendencies is very hard. Interpretability offers a way to look for them directly inside the model.
Control. In the most optimistic case, interpretability does not just help us understand a model but intervene in it. If you can identify the parts of a network responsible for harmful outputs, you might be able to weaken or remove them. Feature steering is an early demonstration that this kind of targeted intervention is at least possible in principle.
The Honest Picture
Mechanistic interpretability has made real progress. But some of the people closest to the work have been candid about its limits.
Neel Nanda, who leads the mechanistic interpretability team at Google DeepMind and is one of the founding figures of the field, recently said that the most ambitious version of what he once hoped mech interp could achieve is probably not reachable. He does not see a path to deeply and reliably understanding what large AI systems are thinking, at least not in time to matter before competitive pressures push labs to deploy highly capable models. He now argues for what he calls a Swiss cheese model of AI safety: no single technique will solve alignment, so you layer multiple imperfect approaches on top of each other and hope the gaps do not line up.
Dan Hendrycks and Laura Hiscott take an even sharper position. Their argument is that mech interp rests on a flawed foundational assumption. Even when you successfully identify a feature, you have not necessarily gained reliable insight into how the model will behave. Knowing what a component does is not the same as understanding the system. The gap between reading individual parts and predicting overall behaviour is too large for mech interp to bridge on its own.
The field is young and the problems are genuinely hard. But the researchers being most honest about the limits are the ones who have spent the most time inside the work. That seems worth paying attention to.
I am not sure what to make of this all. On one hand, it is interesting that we can even begin to understand what is happening inside these models. But on the other hand, it is also terrifying that we may not be able to understand them well enough to control them.
I guess I'll just have to read more, get my hands dirty (and maybe try out some techniques for myself), and hope for the best. Hopefully, what I find out could be useful in neuroscience as well, as there are a lot of parallels to draw between the two fields.