Compile the Algorithm Directly into the Transformer

Generate Model Weights Directly—No Data, No Gradients

Recently, a developer open-sourced an experimental compiler called Torchwright: developers first describe a computation graph in ordinary Python, and the compiler then directly generates a set of Transformer weights. The output uses the standard Phi-3 architecture and can be loaded natively by Hugging Face Transformers, with no custom model code and no need to enable trust_remote_code.

More importantly, no training takes place anywhere in the process.

There is no training set, no loss function, no backpropagation, and no Adam or SGD. Rather than having a model “learn an algorithm from examples,” the compiler directly calculates which parameters should be placed in each layer so that the Transformer executes the specified program during its forward pass.

The project’s author draws a clear distinction: what he wants to investigate is not what Transformers can learn, but what Transformers can express. These two questions are often conflated, but their boundaries are entirely different.

The fact that a network can mathematically implement sorting, state transitions, or Boolean operations does not mean that gradient descent can reliably find those weights. Conversely, a failure to train does not prove that the architecture lacks the corresponding computational capability. Torchwright effectively bypasses the question of “how to learn it” and writes the target algorithm directly into the parameters.

It Compiles Computation Graphs, Not Arbitrary Python

“Compiling Python into a Transformer” is a phrase that is easy to misinterpret.

Torchwright is not intended to replace CPython, nor does it mean that you can feed it any Python project and get a model capable of running Django, NumPy, or operating system calls. What the author provides is a way to organize and express computation graphs using ordinary Python, then map the graph’s operations, data dependencies, and execution order onto Transformer layers, channels, and attention connections.

Conceptually, its usage is closer to the following:

# Conceptual illustration, not the project's exact API
@graph
def program(x, y):
    carry = logical_and(x, y)
    value = logical_xor(x, y)
    return value, carry

checkpoint = compile_graph(
    program,
    target_architecture=Phi3
)

What is actually being compiled here is logical_and, logical_xor, and their dependencies—not the full semantics of the Python interpreter. Capabilities such as loops, branches, dynamic memory, external I/O, and arbitrary-precision integers still depend on whether the compiler provides corresponding constructs and whether the sequence length, number of layers, and hidden dimensions can accommodate them.

This limitation does not diminish the project’s significance. Instead, it shows that Torchwright is more like a domain-specific compiler for Transformers: Python is the frontend description language, the computation graph is the intermediate representation, and the Phi-3 checkpoint is the final binary.

Here, the Transformer Is More Like a “Strange Chip”

When most people discuss Transformer weights, they assume those weights come from pretraining: parameters are randomly initialized and then gradually adjusted using massive amounts of data. Torchwright takes a different perspective—if we already know what we want the network to execute, why not solve for the corresponding weights directly?

A standard Transformer can be broken down into several kinds of computational resources:

• Attention layers read, select, and move information between tokens or positions;
• MLP layers perform local transformations on the representation at each position;
• The residual stream acts like a shared data bus and scratch space running through all layers;
• The number of layers can represent sequential stages of computation;
• Hidden dimensions and token positions can carry variables, intermediate results, and control signals.

In trained models, the purposes of these resources are jointly shaped by the data and optimizer. In compiled models, they are allocated in advance by the compiler. One attention head may be responsible for retrieving data from a designated position, a section of an MLP may implement a fixed mapping, and the next layer may continue processing the result.

What Torchwright does is therefore closer to hardware synthesis than model fine-tuning. Traditional compilers translate programs into CPU instructions, GPU kernels, or WebAssembly; Torchwright translates programs into batches of matrices, while the “virtual machine” executing those matrices happens to be Phi-3’s forward-pass code.

This is also the project’s most interesting aspect: the Transformer architecture can be viewed as a highly parallel computational target with an extremely awkward instruction set.

A “Standard Phi-3 Checkpoint” Is More Practical Than “No Training”

Manually constructing Transformer weights is not a new idea. RASP provides a programming language that can be mapped to Transformer sublayers, while DeepMind’s Tracr goes a step further by compiling RASP programs into actual weights. The project’s author explicitly acknowledges this prior work.

Torchwright differs in two main ways:

1. It lets developers describe computation graphs using familiar, ordinary Python rather than requiring them to learn a specialized language first;
2. Its output targets the existing Phi-3 architecture and can be loaded natively by Hugging Face Transformers.

The second point is particularly important.

Many research prototypes that “build computers inside neural networks” require a companion interpreter, custom operators, or a specialized model class for their demonstrations. They may prove that the construction works, but they are difficult to integrate into existing deployment pipelines. Torchwright generates standard-architecture checkpoints, meaning that model configurations, tensor names, and forward interfaces all fit within a mature ecosystem.

From the perspective of a deployment system, it is simply a Phi-3 model: the inference framework does not need to know whether the weights were trained, distilled, or calculated by a compiler.

Not requiring trust_remote_code also lowers the barrier to reproducing experiments. The loading side does not need to execute arbitrary Python code included in the repository, as long as the local version of Transformers already implements the corresponding architecture. Of course, this does not mean the checkpoint’s behavior is necessarily safe. A safe tensor format can prevent code execution during loading, but carefully constructed weights may still implement unexpected logic.

This raises a question worth considering in advance for the model supply chain: when auditing checkpoints in the future, it may no longer be enough to ask, “What data was this trained on?” We may also need to ask, “How exactly were these weights produced?”

It Is Not a Smarter Phi-3

A dose of cold water is needed here: although the compiled checkpoint uses the Phi-3 architecture, that does not mean it possesses Phi-3’s language capabilities.

The architecture is only a container; capabilities come primarily from the parameters. A randomly initialized Phi-3 cannot hold a conversation merely because its filename and configuration are correct. Likewise, a set of weights encoding an adder, state machine, or graph algorithm will not automatically retain the knowledge of the original pretrained model.

So this is not “building a large language model without training.” It is “making a large-language-model architecture execute a specific algorithm without training.” The two are very different.

Its relationship to conventional training can be summarized roughly as follows:

| Approach | How weights are produced | Advantages | Main issues | | --- | --- | --- | --- | | Supervised training or pretraining | Optimized from data and a loss function | Can handle ambiguous, open-ended, and high-dimensional tasks | Behavior is difficult to guarantee precisely; high cost | | Program distillation | Use a program to generate examples, then train a model to imitate it | Flexible interface; can approximate complex programs | Still introduces errors and still requires training | | Weight compilation | Construct parameters directly from an algorithm’s structure | No training; greater interpretability and determinism | Limited by the architecture, numerical precision, and compiler support | | Conventional program execution | Run code directly on CPUs or GPUs | Efficient, mature, and reliable | Cannot be embedded directly into a model’s unified forward-pass interface |

If the goal is merely to compute a deterministic algorithm, a CPU is usually faster, cheaper, and easier to verify. Putting addition or sorting into a Transformer does not magically provide a performance advantage. For now, the project is better viewed as a research tool and compiler experiment than as an attempt to replace LLVM, CUDA, or database execution engines.

The Real Value Lies in Drawing a Line Between “Learnable” and “Expressible”

Torchwright’s most direct value to developers and researchers is that it provides controlled experimental subjects.

Models produced through training often contain a tangled mixture of correlated features. When investigating whether a particular attention head truly implements “variable retrieval,” it is difficult to rule out the effects of the data distribution, redundant pathways, and accidental correlations. Compiled models are different: in principle, the construction process determines why every block of weights exists, what information it is expected to move, and where each intermediate variable should appear.

This enables several kinds of use cases:

1. Ground Truth for Mechanistic Interpretability

Researchers can first compile a known algorithm, then test whether probes, activation patching, path patching, and circuit-discovery tools can recover its true structure. It effectively provides interpretability research with an exam that comes with an answer key.

2. Verifying the Expressive Power of Transformers

Papers often prove that “there exists a set of parameters” with which a certain kind of network can implement a particular algorithm. But there is still a gap between an existence proof and a runnable checkpoint. A compiler can turn some theoretical constructions into actual tensors, allowing developers to inspect numerical errors, context boundaries, and real resource consumption within a standard inference stack.

3. Constructing Deterministic Neural Modules

In hybrid systems, some steps are well suited to learning, such as understanding natural language, while others require strict execution, such as protocol parsing, format conversion, or finite-state constraints. Compiling deterministic logic into the same architecture could, in theory, reduce interface switching between the model and external tools.

But the phrase “in theory” cannot be omitted. Combining compiled weights with existing pretrained weights without disrupting either side’s behavior is much harder than constructing an independent checkpoint. Directly overwriting parameters will usually erase the original model’s capabilities. Preserving subspaces, adding layers, or composing the result through adapters would require new compilation and verification methods.

4. Generating Stress Tests and Counterexamples

Because developers can precisely specify the program executed inside a Transformer, they can construct extreme routing patterns, long dependency chains, or unusual state transitions to test whether inference engines, quantizers, and model-analysis tools secretly rely on the statistical properties of “naturally trained weights.”

The Boundaries Are Equally Clear: Softmax Is Not a CPU Instruction

Compiling an algorithm into a Transformer does not mean every step has the precision of a traditional digital circuit.

First, Transformers typically operate on floating-point numbers. Attention-based selection may rely on large logit differences to approximate discrete addressing, while Softmax generally produces an approximately one-hot distribution rather than a true Boolean switch. Once a model is converted to FP16, BF16, INT8, or more aggressive quantization, whether the original numerical margins remain sufficient must be tested case by case.

Second, resource consumption may grow rapidly. A simple node in the computation graph does not necessarily correspond to just one neural-network layer, and variables may occupy hidden channels or token positions. As the algorithm scales, the checkpoint’s width, depth, and context length may all increase.

Third, dynamic control flow is difficult to obtain for free. Input-dependent loop counts, recursion, dynamic arrays, and random access must either be unrolled within a fixed-depth network or executed repeatedly through autoregressive steps. The former increases model size, while the latter increases inference latency.

Finally, there is the problem of verification. The compiler must prove or test:

• How large the supported input domain actually is;
• Whether floating-point errors accumulate across layers;
• Whether results remain consistent across different inference backends;
• Whether KV cache, Flash Attention, and quantization alter the semantics;
• Whether exceeding the designed length produces an explicit error or silently generates incorrect results.

These issues will determine whether the project ultimately remains an elegant demonstration or becomes a reliable toolchain.

Our Assessment: A Research Toy in the Short Term, but It May Change How Weights Are Produced in the Long Term

The easiest aspect of Torchwright to exaggerate is framing “executing algorithms without training” as an alternative to gradient descent. In reality, open-world knowledge, natural-language understanding, and fuzzy pattern recognition still depend on learning. Developers also have no reason to use Transformers to replace highly optimized general-purpose computers.

But it reveals a path worth taking seriously: training does not have to be the only way to modify neural-network weights.

Future model weights may come from a combination of processes: most knowledge acquired through pretraining, specific capabilities injected through fine-tuning, strict rules written by compilers, and safety boundaries enforced through formal constraints or runtime checks. At that point, a checkpoint would no longer be merely the product of a single training run, but more like an executable jointly generated from multiple linked units.

This would also force changes in model formats, evaluation, and security auditing. For a given set of parameters, recording only the dataset, training duration, and base model may no longer be enough. Toolchains may also need to record the compiler version, computation graph, numerical assumptions, and behavioral proofs.

At present, the Torchwright repository provides 12 runnable examples. For developers, the most appropriate way to approach it is not to deploy it in production, but to treat it as a transparent Transformer laboratory: first observe how a known program becomes attention and MLP operations, then use that insight to understand the circuits in trained models that appear to behave like programs.

It has not proven that Transformers are the best medium for general-purpose computation, but it demonstrates one point very clearly: model weights are not merely statistical results produced by training—they can also be compiler artifacts.

References

• Reddit: Project announcement and technical discussion by the author: The author explains the project’s motivation, training-free workflow, differences from RASP and Tracr, and its goal of producing standard Phi-3 checkpoints.
• GitHub: Torchwright project repository: Contains the source code, installation instructions, and 12 runnable examples for verifying the current scope of the implementation.
• Hugging Face: Phi-3 Mini model page: Provides information on the standard Phi-3 architecture targeted by Torchwright and how it is loaded within the Transformers ecosystem.