No Surrogate Training—Directly Decompose Large Model Weights

Interpretability Takes a More Direct Route

Research on large language model interpretability has recently taken a noteworthy turn: instead of training an additional network to imitate the original model, researchers are directly decomposing the original model’s weights.

According to research recently published by the Zhizhi Research Institute, its new approach analyzes a model’s internal structure through weight decomposition, reducing data requirements to less than 1% of those of existing data-driven methods. This is not simply a matter of shrinking the training dataset by a factor of one hundred. It removes an expensive intermediate step that can also introduce errors: the surrogate network.

As of August 15, 2026, there is not yet enough public information to demonstrate that this approach has comprehensively replaced mainstream methods such as sparse autoencoders. But the direction is clear: in the past, researchers effectively attached an external EEG machine to a large model and trained an interpreter by collecting massive amounts of activation data. Now, they are trying to open the machine directly and examine how the gears themselves mesh.

For developers, the value goes beyond making research experiments cheaper. It could change the engineering workflows used for model auditing, fault localization, and security analysis.

The Problem with the Old Approach: Training Another Model to Understand the Model

Large language models are not composed of neurons with clearly defined responsibilities.

A single neuron may respond simultaneously to code indentation, personal names, geographic locations, and refusal language, while a single concept may be distributed across hundreds or even thousands of neurons. These phenomena are commonly known as polysemanticity and distributed representation. Observing individual neurons directly is like trying to understand an entire operating system through a single transistor—the level of granularity is clearly wrong.

Over the past few years, a mainstream approach in mechanistic interpretability has been to perform dictionary learning on model activations. Typical tools include sparse autoencoders, as well as approaches such as transcoders that attempt to reproduce the input-output relationship of an original module.

The basic process can be summarized as follows:

1. Prepare a large volume of text or task samples;
2. Run the target model end to end and record activations from selected layers;
3. Use those activations to train a sparse surrogate representation;
4. Decompose the high-dimensional, entangled activations into a set of features that are easier to label;
5. Verify whether each feature genuinely affects the output by ablating, amplifying, or suppressing it.

This approach works and has already demonstrated the ability to identify abstract features related to code, locations, entities, tone, and even safety within large models. But it comes with two unavoidable costs.

The first is the cost of data and computation. To cover the internal states a model may enter, researchers must collect enough activation data with sufficient diversity. The larger the model, the more layers it has, and the longer its context window, the higher the cost of data generation, storage, and interpreter training. Even without updating the original model, the process may consume a substantial number of GPU-hours.

The second is approximation error. Researchers ultimately want to explain the original model, but what they actually analyze is often a representation module trained afterward. If the surrogate module does not adequately reproduce the original module, the resulting structure may combine two different things: the original model’s genuine computational patterns and the interpreter’s own fitting biases.

In other words, to open one black box, researchers place a smaller black box around it.

What Exactly Is Being Decomposed When We Decompose Weights Directly?

The core premise behind the Zhizhi Research Institute’s approach is that the model has already encoded the structures it learned in its weights, so there is no need to relearn them from massive volumes of activation data every time.

Consider a linear transformation commonly found in Transformers. A weight matrix can be approximated as multiple low-rank components. Intuitively, the matrix W can be viewed as a superposition of several sets of directions:

• One set of directions identifies certain types of patterns in the input;
• Another set determines where the information is written after the pattern is identified;
• Each component’s contribution to the current token depends on how closely the input matches its corresponding direction.

If W is represented as the sum of multiple rank-one or low-rank components, a high-dimensional matrix multiplication that was previously difficult to interpret can be rewritten as many relatively independent signal channels. Researchers can then continue tracing how these channels are amplified, combined, or read by attention mechanisms across different layers.

The difference from sparse autoencoders is not merely that the object being decomposed has changed from activations to weights.

Sparse autoencoders usually begin by observing model behavior over a large number of samples and then learning a new feature dictionary. Weight decomposition, by contrast, prioritizes extracting structure directly from the parameters and minimizes the need to refit the model’s behavior. As a result, it does not require training a complete surrogate network and reduces interpretability errors caused by inadequate reconstruction in the surrogate module.

However, it is important to emphasize that direct weight decomposition does not mean that no data is needed at all.

A matrix may have many mathematically valid decompositions, but not every decomposition has human-readable semantics. Researchers may still need a small number of samples to:

• Determine which components are frequently active on real inputs;
• Match components with representative text and semantic labels;
• Filter out directions that are numerically significant but behaviorally unimportant;
• Test component stability across different domains and contexts;
• Verify through intervention whether causal relationships exist between components and model outputs.

Therefore, the claim of less than 1% is more accurately understood as follows: relative to baselines that depend on large-scale activation collection and surrogate-module training, data requirements are dramatically reduced. It does not mean that the total cost of the entire interpretability process falls by 99%, nor does it imply identical gains for every model, layer, and task.

Why This Approach Deserves Attention

1. It Reduces the Risk of “the Interpreter Explaining Itself”

The most important criterion in interpretability research is not whether the visualization looks compelling, but whether the explanation is faithful. No matter how intuitive a feature label may seem, if it comes from the surrogate network’s own representation rather than the computational path actually used by the original model, it is difficult to rely on it for auditing.

Working directly with the original weights is cleaner, at least in terms of the object being analyzed. It examines parameters that actually participate in model inference rather than an approximate module trained after the fact.

Of course, decomposition algorithms still introduce inductive biases. A preference for low-rank, sparse, or mutually independent components is itself an assumption about which structures are most worth explaining. Compared with retraining an entire mapping, however, these biases are generally easier to describe and validate explicitly.

2. Interpretability Could Move from Large Labs to Ordinary Teams

The cost of activation-based interpretability lies not only in training, but also in the data pipeline.

To analyze a large model with dozens of layers, a team must process enormous volumes of intermediate tensors, decide which tokens, layers, and positions to sample, and address storage throughput and GPU utilization. Many open-source model teams have enough resources to fine-tune models, but not to systematically train sparse autoencoders for every layer.

If weight decomposition requires less than 1% of the data used by baseline methods, model developers could perform internal checks more frequently:

• Compare changes in weight structure between base models and instruction-tuned models;
• Check whether safety fine-tuning merely suppresses outputs or actually changes internal representations;
• Identify why a continued-pretraining run damaged coding or multilingual capabilities;
• Analyze which directions were abnormally amplified after merging LoRA adapters;
• Conduct structured regression testing before releasing different checkpoints.

Interpretability can genuinely influence model engineering only when it becomes part of the iteration cycle rather than remaining an expensive experiment performed once before release.

3. It May Be Better Suited to Analyzing Differences Between Versions

A weight-based approach is naturally suited to answering an engineering question: what exactly changed between two model versions?

Traditional evaluations may tell you that a new model gained two points on a benchmark, but they struggle to explain whether the change came from added knowledge, altered reasoning paths, or an output style that better caters to the evaluator. Matching and comparing weight components may make it possible to identify internal directions that have appeared, disappeared, or changed in strength.

This is particularly useful for continued pretraining, domain fine-tuning, and model merging. Instead of focusing only on final accuracy, developers can observe which components contain most of the parameter updates and whether those changes align with the target task.

Do Not Think of It as a “Large-Model Decompiler” Just Yet

This work is highly significant, but there are also obvious ways in which its implications could be overstated.

Weight Structure Does Not Automatically Equal Semantics

A low-rank component may be mathematically separable without being labelable as “deception,” “Python,” or “Paris.” Moving from numerical directions to natural-language concepts still requires samples, automated labeling, and human judgment.

More problematically, the same component may perform different functions in different contexts. Computation in large models is dynamically composed: static weights indicate what could happen, while activations tell researchers what actually happened during a particular inference.

Weight decomposition is therefore more like obtaining a circuit diagram than receiving source code annotated line by line.

Nonlinear Layers Do Not Disappear Because a Matrix Has Been Decomposed

Transformers are not simply stacks of linear matrices. Activation functions, normalization, attention softmax, gated MLPs, and residual connections all affect how components interact.

In gated architectures such as SwiGLU, whether a weight direction actually writes information into the residual stream depends on the state of another gating branch for the current input. Decomposing a single matrix may reveal the parts while still leading researchers to misjudge the role those parts play in the machine as a whole.

A reliable approach must therefore continue to combine static weight analysis with dynamic activation tracing. Much less data may be needed, but researchers cannot look at a weight heatmap alone and declare that they have found the model’s thought circuits.

Parameter Accessibility Is a Hard Boundary

This approach is most directly applicable to open-weight models. External developers cannot access the weights of closed models that are available only through APIs, so they cannot independently perform the decomposition. Even if model vendors adopt similar techniques internally, third parties will still find it difficult to verify the results independently.

Quantization also introduces additional problems. Low-magnitude directions in INT4, FP8, or mixed-precision weights may be compressed or distorted, and the structures produced by decomposition may not match those in the original high-precision checkpoint. Serious audits should use high-precision weights from training or before release whenever possible and should document the methods used for quantization, tensor parallelism, and weight merging.

How Should Developers Determine Whether It Is Useful?

Engineering teams should not tear down their existing interpretability pipelines simply because they see the phrase “less than 1%.” A more sensible approach is to establish a set of comparative metrics.

Measure Five Things First

1. Reconstruction capability: Whether the decomposed components preserve the primary behavior of the original weights or module;
2. Sparsity: How many components are actually invoked for a single input, and whether the representation is more concentrated than the original neuron-level representation;
3. Semantic stability: Whether the same component expresses similar meanings across code, dialogue, multilingual, and other datasets;
4. Causal effectiveness: Whether suppressing or enhancing a component changes the target behavior as expected;
5. Cross-version comparability: Whether components can be matched reliably after model fine-tuning rather than being reshuffled every time.

The fourth item is the most important. Looking only at correlations makes it easy to mistake an accompanying phenomenon for a cause. For example, a component that activates frequently when the model produces a refusal is not necessarily responsible for triggering the refusal; it may simply be reading a safety state that has already formed.

Valuable validation requires controlled interventions on components, followed by checks of whether the target behavior, non-target capabilities, and perplexity change together.

Three Scenarios to Prioritize

First, locating fine-tuning regressions. If general capabilities suddenly decline after domain fine-tuning, teams can compare weight components before and after training to determine whether a small number of directions were updated excessively.

Second, auditing model merges. When multiple LoRA adapters or domain models are merged, weight conflicts often reveal problems earlier than final benchmark scores. Component-level analysis may help identify directions that cancel one another out or become abnormally amplified.

Third, validating safety mechanisms. Teams can examine whether safety training truly changes internal behavior or merely adds a layer of output suppression that is easy to bypass through jailbreaking. Such conclusions, however, must be supported by activation and intervention experiments rather than inferred from static weights alone.

The Real Change: Interpretability Costs Are Beginning to Keep Pace with Model Iteration

Large-model interpretability has historically faced an awkward problem: models are upgraded every few months, yet fully interpreting each version may require expensive data generation, activation storage, and additional training. Interpretability tools cannot keep up with model versions and ultimately become demonstration projects limited to a small number of laboratories.

The value of direct weight decomposition lies in narrowing this time gap.

If data costs below 1% can be replicated across different model sizes and architectures, researchers could first use weight decomposition for low-cost structural surveys and then devote activation analysis and causal interventions to a small number of critical components. Rather than eliminating sparse autoencoders, this approach may reorder the entire workflow:

1. Use weight decomposition to scan the structure quickly;
2. Use a small number of real samples to build behavioral profiles for components;
3. Collect dense activation data for high-risk or high-value components;
4. Confirm causal relationships through ablation, activation patching, and targeted interventions;
5. Incorporate the results into automated checks before model training and release.

This is more realistic than attempting to explain an entire large model all at once.

Assessment: Not the Destination, but Possibly a Better Starting Point

The most commendable aspect of the Zhizhi Research Institute’s approach is not that it creates more attractive feature visualizations, but that it challenges the cost structure of interpretability: if knowledge has already been stored in the weights, why should understanding it require training another approximate network?

Direct weight decomposition can reduce data dependence and surrogate error, while also making it better suited to model-version comparisons and frequent audits. This approach is particularly appealing to teams working with open-weight models.

But it has yet to solve three fundamental problems: how to translate mathematical components reliably into human semantics, how to handle nonlinearities and cross-layer composition, and how to use causal experiments to demonstrate that an interpretation is not merely coincidental.

A more accurate conclusion, therefore, is that the black box of large models has not been completely opened. Researchers have simply picked up a cheaper screwdriver that works closer to the original machine.

When interpretability data costs fall to one-tenth—or even below one percent—of their original level, the most exciting prospect is not the production of a few more neuron heatmaps. It is that interpretability may finally become part of the model development workflow: something run after every training, fine-tuning, and merging operation, just like unit tests, performance profiling, and security scans.

That step is far more practical than claiming, “We already know what the model is thinking.”

References

Due to the domain restrictions for links at the end of this article, only open-source resources accessible within China and directly related to the technical background discussed above are listed below. Links to public reports concerning the Zhizhi Research Institute are not repeated here.

• TransformerLens: A mechanistic interpretability toolkit for Transformers that supports activation caching, hooks, intermediate-state analysis, and causal interventions.
• SAELens: An open-source toolkit for training and analyzing sparse autoencoders for large language models. It can be used to understand the activation-based interpretability approach contrasted in this article.