Stop Treating LLMs as a Runtime: Reflections from Production Environments

Recently, Unmeshed’s article “LLMs Are Not a Default Execution Engine” has been repeatedly shared on HN and X. Read together with another dev.to post, “Why LLMs Can Never Be ‘Execution Entities’,” they basically capture a growing consensus in the AI engineering world in the first half of 2026: treating large language models as script runtimes was the most expensive illusion of the past two years.

The reason this topic is resurfacing now is no coincidence. From the second half of 2025 through early 2026, a wave of so-called “AI Agent platforms” imploded one after another—not because the models were too weak, but because the architecture was flawed from the start. Developers placed LLMs where they never belonged: schedulers, state machines, and execution engines.

The conclusion first: LLMs are decision-makers, not executors

That may sound obvious, but when people actually start coding, most unconsciously blur the line between the two.

A typical anti-pattern looks like this: the user says, “Pull out customers whose sales exceeded 100,000 last month and send them a thank-you email.” You then build an Agent and hand the entire workflow—database queries, filtering, generating email copy, calling SMTP—to a single LLM, letting it “think” and “execute” at the same time. Run it three times, and maybe two work correctly. On the third run, it confidently tells you “emails have been sent,” when in reality not a single email went out.

This is not a hallucination problem. It’s that you assigned a probabilistic model the responsibilities of a deterministic system.

Why developers fall into this trap

Three reasons, ranked by severity:

First, the demos looked too good. The wave of AutoGPT and BabyAGI demos from 2023–2024 created the illusion that “as long as the prompt is good enough, the LLM can do the work itself.” But there’s an entire discipline of reliability engineering between demo scenarios and production systems.

Second, function calling was overly mythologized. When OpenAI introduced function calling, many people interpreted it as “LLMs can now execute code.” No—they simply know which function should be called. Actual execution, retries, idempotency, transactions, and rollbacks are none of the model’s responsibility.

Third, Agent framework APIs are misleadingly designed. Many frameworks wrap agent.run(task) to look like a normal function call, causing developers to forget that behind it is a non-deterministic inference process with risks of state drift.

What LLMs are fundamentally bad at

Here’s a list based on problems I and teams around me have personally encountered:

• Long workflow orchestration: Once workflows exceed five steps, success rates collapse when the LLM is responsible for both planning and execution. The longer the context, the easier it is for the model to “forget” the original objective.
• Operations requiring strict idempotency: Payments, emails, database writes. You cannot guarantee that running the same prompt twice won’t execute the operation twice.
• State management: LLMs are stateless. What looks like the model “remembering” the previous step is actually you stuffing the previous step into the context. Once the context gets too long, it starts fabricating details.
• Error recovery: In traditional systems, try/catch is explicit. An LLM encountering an error may simply “invent a success” and present it to you.
• Low-latency scenarios: A single inference takes 2–5 seconds. Stretch the chain long enough and you’re dealing with minute-level delays.

The arXiv paper “Large Language Models Do Not Always Need Readable Language” supports this point from another angle—the internal “reasoning” of LLMs doesn’t actually require human-readable intermediate representations. What you perceive as Chain-of-Thought may sometimes just be post-hoc rationalization. This means you cannot determine whether the model is truly executing correctly simply by inspecting its thought process.

The correct approach: layering

So where should LLMs actually sit? The answer is simple: the decision layer, not the execution layer.

A healthy AI application architecture typically divides responsibilities like this:

1. LLM layer: understand intent, decompose tasks, select tools

This layer translates natural language into structured instructions. The input is what the user says; the output is JSON or a DSL describing “what should happen next.” It only decides what to do—it does not execute it.

2. Orchestration layer: deterministic workflow engine

There’s a reason tools like Temporal, Airflow, Prefect, and Unmeshed have existed for decades. They provide:

• Persistent state
• Automatic retries and backoff
• Idempotency guarantees
• Observability and auditing
• Explicit error boundaries

The “plan” generated by the LLM gets executed here, where failures can be inspected, replayed, and compensated for.

3. Tool layer: the code that actually does the work

Database queries, API calls, file operations—these are just ordinary functions. You write unit tests, add monitoring, and run CI. None of this has anything to do with the LLM.

4. Feedback layer: feed execution results back into the LLM

After each step executes, feed the results (success/failure/data) back into the LLM as new context so it can decide the next action. The key point is that state between LLM calls is owned by the orchestration layer, not hidden inside the context window.

A concrete comparison

The wrong approach:

user_query -> LLM(with a bunch of tools)
              -> plans by itself
              -> calls tools by itself
              -> decides completion by itself
              -> retries by itself
              -> returns result

The correct approach:

user_query -> LLM(planner) -> outputs structured plan
                          -> Workflow Engine takes over
                             -> Step 1: execute -> persist result
                             -> Step 2: execute -> error -> automatic retry
                             -> Step 3: requires judgment -> call LLM(judge) again
                             -> ...
                          -> return aggregated result

What’s the difference? In the first model, the LLM is the main loop. In the second, the LLM is a repeatedly invoked pure function, while the main loop stays under your control.

On the abuse of the term “Agent”

Throughout 2025, “Agent” became a catch-all buzzword. But Agent systems that actually succeed in production are, at their core, simply traditional workflows with LLM-based decision nodes added in.

Anthropic’s paper last year, “Building Effective Agents,” already made this very clear: in most scenarios, what you need is not an Agent, but a Workflow. Only when tasks become truly open-ended and impossible to pre-orchestrate should the LLM control the main loop. And even then, you still need guardrails—step limits, cost limits, human review checkpoints.

The cost perspective: this isn’t just about reliability

Another often-overlooked point: treating an LLM as an execution engine will explode your bill.

For a 10-step task, if the LLM must plan, reflect, and decide the next action at every step, a single task may involve dozens of API calls. For the exact same task, if the LLM is only invoked at key decision points (for example: one planning call at the beginning, one branch decision in the middle, one summary at the end), costs can drop by an order of magnitude.

This is why more teams are now adopting model tiering: fast, cheap models handle routine judgments, while expensive models are reserved for the most critical decisions. This is where aggregation platforms like OpenAI Hub become valuable—you can use a single key to access GPT-5, Claude 4.5, Gemini 2.5, and DeepSeek V4 simultaneously, dynamically switching models in the orchestration layer based on node complexity, without separately integrating and maintaining authentication for each provider. Direct domestic connectivity also eliminates an entire layer of network troubleshooting overhead.

Three suggestions for developers building AI applications

1. Draw the architecture before writing your first prompt. Decide clearly which parts are deterministic and which require LLM judgment. Use code for the former and models for the latter.

2. Treat LLM calls like external APIs: they can fail, timeout, or return unexpected outputs. Timeouts, retries, fallbacks, and caching are all mandatory.

3. Be wary of the temptation to “let the LLM handle everything.” Every time you feel tempted, ask yourself: if this workflow fails, can I pinpoint exactly which step broke? If the answer is “no,” don’t implement it that way.

Final thoughts

It’s 2026. AI engineering is moving from the “show-off” phase into the “survival” phase. Projects that still use LLMs as the main loop, runtime, or universal glue are gradually being replaced by architectures that are more modest—but more reliable.

Models will continue getting stronger, but “stronger” does not mean “reliable,” much less “deterministic.” The concepts accumulated through decades of software engineering—state machines, transactions, idempotency, observability—are not becoming obsolete because of LLMs. If anything, they matter more now.

Use LLMs where they excel, and leave execution to systems designed for execution. That’s not conservatism. It’s common sense.

References

• WangRongsheng/awesome-LLM-resources - GitHub: A global collection of LLM resources, continuously updated with topics including Agents, reasoning, MCP, and more. Useful for systematically understanding the current ecosystem.