Compile Agent Workflows into Small Models, Cutting Costs by a Hundredfold

Last week, the University of Melbourne and the i14 Research Institute dropped a paper on arXiv with a straightforward title: Compiling Agentic Workflows into LLM Weights. The core idea is equally direct — the multi-step Agent workflow you painstakingly built using GPT‑4 can actually be compiled into a small 8B parameter model, with minimal performance loss but costs reduced by two orders of magnitude.

The paper sparked lively discussion in Reddit’s r/MachineLearning. One user, u/ThirdWaveCat, said their company is re‑evaluating the value of small language models due to token billing problems, and this paper provides a practical approach.

The problem: Agents are useful, but too expensive

Let’s start with the background.

Over the past year, AI Agents have gone from concept to reality. Whether it’s code generation, data analysis, or complex business process automation, developers are trying to make LLMs not just “chat” but actually “get work done”—call tools, plan steps, handle exceptions, and iterate for optimization.

A typical architecture for an Agentic Workflow uses a powerful frontier model (e.g., GPT‑4, Claude 3.5) as the “brain,” coupled with carefully designed prompts, tool invocation protocols, and state management logic. The results are indeed good—but the bill can be scary.

A medium-complexity Agent task may involve dozens of dialogue rounds and tens of thousands of tokens of context. At GPT‑4 pricing, each run can cost several cents to tens of cents. In a high-frequency production environment, the monthly bill can easily break five figures.

What’s worse, the “program logic” of these Agents doesn’t get solidified. Each time they run, the model has to re‑understand the task, parse tool documentation, and plan execution paths from scratch. A significant portion of what you pay is actually this repeated “understanding cost.”

And that’s where the “compile” term in the paper title comes from — programming languages need compilers to turn high‑level code into machine code. Could Agent workflows also be “compiled” into model weights?

The idea: Distill execution traces rather than prompt templates

The core method of the paper can be summed up in one sentence: Run Agent tasks using frontier models, collect complete execution traces, and then perform supervised fine‑tuning (SFT) on a small model using these traces.

This idea isn’t new — knowledge distillation is already a mature technology in machine learning. But the key is what to distill.

Traditional Agent prompt engineering essentially teaches the model “how to think.” You write a long system prompt telling it: your role, the tools available, how to handle certain situations. This knowledge exists as text in the context, and the model has to “read” and “understand” it every single time.

The paper’s approach is different. It doesn’t distill the prompts but the actual input–output pairs from the execution process:

• What question did the user ask
• What tool did the model decide to call
• What results did the tool return
• How the model interpreted the results and decided the next step
• What final answer it gave

The whole process is fully recorded, forming “execution traces.” These traces become the small model’s training data.

The benefits of this approach are obvious:

1. Implicit knowledge becomes explicit. Frontier models use a lot of hidden decision logic in Agent tasks—when to call search, when to answer directly, how to retry if a tool returns an error. This logic is hard to fully express in prompts but naturally appears in execution traces.

2. Remove redundant context. Original Agent systems need to send complete tool documentation and instruction sets each time. The fine‑tuned model “remembers” this knowledge, so at inference it only needs the user input.

3. Fewer reasoning steps. Original Agents may need multiple dialogue rounds to finish a task; distilled models often learn a more direct inference path.

Experiments: 8B model approaches GPT‑4 level

The paper ran experiments on multiple standard Agent benchmarks, using LLaMA 3.1 8B as the base.

Test tasks included:

• Tool use: API selection, parameter filling, error handling
• Multi‑step reasoning: complex tasks requiring planning and execution across steps
• Web navigation: completing specified actions in simulated web environments
• Code generation & execution: write code, run it, and fix it based on results

The results are interesting:

| Model configuration | Avg. accuracy | Single inference cost | |--------------------------------|---------------|-----------------------| | GPT‑4 + Agent framework | 78.3% | $0.12 | | Claude 3.5 + Agent framework | 76.1% | $0.09 | | LLaMA 8B original | 41.2% | $0.001 | | LLaMA 8B after compilation | 72.6% | $0.001 |

The compiled 8B parameter model reaches over 90% of frontier‑model performance in most tasks. And the cost? About 100× lower.

The cost difference comes from two factors:

1. Cheaper model itself — 8B inference costs a fraction of GPT‑4’s.
2. Fewer calls — Tasks needing multiple rounds in the original Agent framework can often be solved in one or two rounds post‑compilation.

Of course, the paper honestly notes limitations:

• Limited out‑of‑distribution generalization — compiled models excel on task types covered during training, but aren’t as flexible with entirely new types.
• Tool changes require recompilation — if tool interfaces change, you may need to recollect traces and fine‑tune again.
• Complex reasoning gap — small models still lag significantly on long‑chain, multi‑branch reasoning tasks.

Practice: How to use this in your own project

The idea is fairly reproducible. If you have a GPT‑4‑based Agent system and want to migrate to a cheaper small model, the steps are roughly:

Step 1: Collect execution traces

Add logging to your Agent system to record every round:

• User input (or system‑triggered input)
• Model output (including reasoning steps and tool calls)
• Tool return results
• Final answer

Data format can be simple JSON Lines:

{
  "task_id": "001",
  "turns": [
    {"role": "user", "content": "Check Beijing's weather for tomorrow"},
    {"role": "assistant", "content": "<tool_call>get_weather({\"city\": \"Beijing\", \"date\": \"tomorrow\"})</tool_call>"},
    {"role": "tool", "content": "{\"temperature\": \"28-35℃\", \"condition\": \"Sunny to cloudy\"}"},
    {"role": "assistant", "content": "Tomorrow in Beijing will be sunny to cloudy, with temperatures 28-35℃. Don't forget sun protection."}
  ]
}

How much data is enough? The paper used about 100K traces. But based on experience, for domain‑specific Agents, a few thousand to tens of thousands of high‑quality traces usually yield clear improvements.

Step 2: Clean and augment data

Raw traces will inevitably have “bad data”:

• Frontier models make mistakes — some final answers are wrong
• Some reasoning paths are overly long — not optimal
• Some edge scenarios aren’t covered

Cleaning strategies:

1. Result verification — filter out traces with wrong final answers for tasks with clear answers.
2. Path optimization — if multiple traces exist for a task, keep shorter/more efficient ones first.
3. Diversity sampling — ensure coverage of many task types and edge cases.

Augmentation strategies:

1. Negative samples — create flawed reasoning paths so the model learns to distinguish good from bad.
2. Tool failure scenarios — simulate failed tool calls to train error handling.
3. Instruction variants — generate multiple phrasings of the same task to boost generalization.

Step 3: Fine‑tune

Choose a suitable base model. The paper used LLaMA 3.1 8B, but you can weigh options based on needs:

• Cost‑effective: Qwen2.5‑7B, LLaMA 3.1 8B
• Extreme cost saving: Qwen2.5‑3B, Phi‑3‑mini
• Better performance: LLaMA 3.1 70B, Qwen2.5‑72B

Preferred frameworks: LLaMA‑Factory or Axolotl, supporting LoRA and full fine‑tuning with simple configs:

# LLaMA-Factory example config
model_name_or_path: meta-llama/Llama-3.1-8B-Instruct
stage: sft
finetuning_type: lora
lora_rank: 64
lora_alpha: 128
lora_target: all
dataset: agent_traces
template: llama3
output_dir: ./output/agent-compiled
per_device_train_batch_size: 4
gradient_accumulation_steps: 8
num_train_epochs: 3
learning_rate: 2e-4

Training time depends on dataset size and hardware. For 100K traces, single A100, LoRA fine‑tuning takes about 10–20 hours.

Step 4: Deploy and iterate

Deploy compiled models via vLLM, TGI, or Ollama. Small models and shorter contexts yield much lower inference latency than original Agent setups.

But don’t expect perfection from one fine‑tuning. In practice, you’ll find edge cases missing from training. Suggested approach:

1. Fallback mechanism — when small model confidence is below threshold, fall back to original frontier model.
2. Continuous data collection — gather new scenarios and mistakes from production, retrain periodically.
3. Monitor key metrics — task success rate, average reasoning steps, user satisfaction.

The bigger significance of this direction

This paper reminds me of programming language history.

Early programmers wrote in assembly, handling each instruction themselves. Later, high‑level languages and compilers let them just write logic while compilers optimized and translated. Then JIT compilation emerged, dynamically optimizing at runtime.

Agent workflows are now at the “assembly” stage—you have to explicitly tell the model each step via prompts, and it re‑interprets them each run. Compiling into small model weights essentially solidifies program logic into “machine code,” greatly boosting efficiency.

This idea can extend further:

• RAG systems — compile retrieval, filtering, and integration logic into the model to reduce retrieval calls.
• Code assistants — compile codebase knowledge and coding conventions into the model for code that better matches project style.
• Customer service bots — compile business processes and FAQ handling logic for faster, more consistent responses.

Of course, this isn’t magic. Compiled models trade flexibility for efficiency. For highly open, unpredictable tasks, large models + Agent frameworks remain superior.

But for many validated, relatively stable Agent applications, “compilation” is a route worth serious consideration. After all, the bill won’t shrink on its own.

In closing

This paper doesn’t propose a revolutionary new architecture, but it clearly explains and validates a practical idea.

In today’s AI world, where ROI matters more and more, “how to get similar results for less money” is a very real question. Knowledge distillation, model compression, inference optimization — these less flashy technologies may be the real keys to AI ubiquity.

If you’re struggling with high Agent costs, try this approach. Start with a small‑scale test — collect a few thousand traces, fine‑tune a small model, run it in a test environment. If results are good, gradually scale up data and application scope.

Technical debt can be paid slowly, but the bill arrives every month.

---

References

• Reddit r/MachineLearning discussion thread — community discussion of the paper, includes some practical experience sharing
• LLaMA‑Factory GitHub repository — recommended fine‑tuning framework from the paper