Agent Architecture Is Converging

Agent Infrastructure Is Finally Converging

Cloud application platform Render recently summarized infrastructure patterns for agentic applications in an engineering article. Viewed alongside the implementation experience recently shared by vendors such as AWS, one shift has become clear: production architectures for agent applications are beginning to converge.

Here, “convergence” does not mean that LangGraph, AutoGen, CrewAI, or any single cloud service will unify the market. Rather, after different teams have encountered problems such as timeouts, duplicate execution, context bloat, uncontrolled tools, and untraceable costs, they have gradually built similar systems:

1. Synchronous web services only accept tasks; they do not run the entire Agent;
2. Long-running tasks enter queues or durable workflows and are executed asynchronously by Workers;
3. Agent state is persisted in a database rather than kept only in process memory or Prompts;
4. Browsers, code interpreters, and high-risk tools run in isolated sandboxes;
5. Short-term context, long-term memory, and business facts are stored separately;
6. Every model inference, tool call, and state change is recorded in a unified Trace;
7. Human approval, retries, replay, and cancellation become fundamental runtime capabilities.

This looks a lot like a traditional distributed system. The difference is that an Agent delegates part of the control flow that was previously hard-coded to a nondeterministic model. Precisely because of this, traditional backend architecture has not become obsolete—it must instead be implemented more rigorously.

The model determines the ceiling of capability; the infrastructure determines whether it can show up for work reliably.

An Emerging Seven-Layer Architecture

If we condense common recent production practices into a single architecture diagram, it can be divided into seven layers.

Layer 1: Entry and Identity—Do Not Make the HTTP Request Wait for the Agent to Finish

A standard chat endpoint can complete inference within a single request, but real Agent tasks often take minutes or even hours. For example:

• Browse 30 supplier websites and generate a comparison table;
• Log in to an enterprise backend, export data, and then run an analysis script;
• Modify code, run tests, wait for CI, and continue fixing issues after failures;
• Gather information and wait for human confirmation at critical steps.

If this logic is placed directly inside a Web Handler, it will run into gateway timeouts, container restarts, dropped connections, and scaling-related migrations. A more sensible approach is for the entry layer to authenticate the user, validate parameters, and create the task, then immediately return a task_id. The client obtains progress updates through polling, SSE, or WebSocket.

The entry layer should also generate three key identifiers:

• task_id: a single user task;
• run_id: a specific execution of the task, which changes upon retry;
• tenant_id: the user or organization to which the task belongs, used for data and permission isolation.

Do not underestimate these three fields. Many teams only discover at the observability stage that their logs contain model request IDs but cannot answer, “Which execution of which user’s task did this tool call belong to?”

Layer 2: Queues and Durable Workflows—Turn Loops into Recoverable State Machines

The most common Agent logic is “think—act—observe—think again.” In a demo, a simple while loop is enough:

while not finished:
    decision = model(context)
    result = execute(decision.tool)
    context.append(result)

In production, this loop is nowhere near sufficient. The process may crash at step 12, the model may select the same tool twice in succession, an external API may have already charged the account before timing out, or human approval may not arrive until two days later.

The Agent execution loop should therefore be modeled as a durable state machine. Before each step begins, record its input and idempotency key; after it ends, record its output, duration, cost, and status. On failure, resume from the most recent checkpoint instead of feeding the entire context back to the model and taking another gamble.

A practical task state might look like this:

task:
  status: waiting_for_tool
  current_step: 7
  budget:
    max_steps: 30
    max_tokens: 120000
    max_cost_usd: 8
  retry_policy:
    model_call: 3
    read_only_tool: 5
    side_effect_tool: 0
  checkpoint:
    conversation_version: 14
    last_successful_action: search_documents

What matters most here is not whether you choose Redis Queue, Kafka, or a particular workflow framework, but whether you establish three properties: recoverability, cancellability, and replayability.

Ordinary message queues are suitable for short, relatively independent tasks. Durable workflows are generally more appropriate when tasks involve waiting for events, human approval, scheduled wake-ups, and multi-stage compensation. Do not try to reimplement half a workflow engine with a jobs table and endless polling.

Layer 3: The Model Gateway—Treat Models as Replaceable Inference Resources

An Agent’s dependency on models is more complex than a chatbot’s. Planning, code generation, visual recognition, and result summarization do not necessarily need to use the same model. Production systems need a unified gateway above the model layer that handles:

• Model routing and fallback;
• Timeouts, rate limiting, and concurrency control;
• Token and cost metering;
• Recording Prompt, tool-definition, and model versions;
• Redaction of sensitive information;
• Caching and provider failover.

One point must be emphasized: model fallback is not simply changing the name from A to B. Different models vary in how reliably they follow tool parameters, structured output requirements, and long context. Whether a task can continue after switching models must be validated through regression evaluations.

A more robust strategy is to create routing pools based on capabilities, such as “strong planning,” “low-cost classification,” and “visual understanding,” and then maintain a set of validated candidate models for each pool. Business code should depend on capability labels rather than a specific model name.

Layer 4: Tools and Sandboxes—MCP Solves Connectivity, Not Security

MCP is becoming a common interface through which Agents connect to data sources and tools, but calling MCP the “USB-C” of Agents tells only half the story. It solves interface discovery and invocation formats, but does not automatically provide:

• Permission isolation;
• Network egress control;
• File-system isolation;
• Invocation idempotency;
• Resource quotas;
• Auditing and human approval.

Browsers and code executors, in particular, should not run in the same container as the main Agent service. A tool that can execute Python, download files, and access an internal network is effectively close to a remote code execution environment. A production sandbox should at minimum restrict CPU, memory, disk, execution time, and network destinations, and it should be destroyed after the task ends.

Tools should also be classified by risk:

| Level | Examples | Recommended Policy | | --- | --- | --- | | Read-only | Search, read a knowledge base, query an order | Execute automatically and record everything | | Reversible write | Create a draft, modify a temporary file | Execute automatically or use sampled approval | | External side effect | Send an email, publish content, submit a ticket | Confirm before execution and set an idempotency key | | High-risk operation | Transfer funds, delete a database, modify production permissions | Require human approval and dual authorization |

A common mistake is to let the model both “determine whether an action is dangerous” and “decide whether to execute it.” That is equivalent to letting an applicant approve their own request. Risk rules should be enforced by a deterministic policy engine; the model may only provide recommendations.

Layer 5: State and Memory—A Vector Database Is Not a Universal Memory System

An Agent has at least three entirely different categories of data:

1. Runtime state: the current execution step, which tool it is waiting for, and how much budget remains;
2. Session context: the messages, tool results, and intermediate artifacts needed for the current task;
3. Long-term memory: user preferences, historical facts, and experience reusable across tasks.

Runtime state is best stored in a relational database or workflow store, where transactions and consistency are required. Large files, webpage snapshots, and code bundles are best stored in object storage. Long-term semantic retrieval is the use case suited to vector indexes, which usually need to work alongside structured databases.

“Chunking every chat record and placing it in a vector database” does not constitute a memory system. It creates three problems: conflicts between old and new facts, leakage of sensitive data across tenants, and an ever-growing number of retrieval results that become progressively less accurate.

A better approach is to extract candidate memories asynchronously and store the following for each memory: source, tenant, expiration, confidence, update time, and deletion status. Facts involving accounts, addresses, and business configurations are best stored in structured fields rather than retained only as natural-language summaries.

Short-term context cannot grow indefinitely either. If an Agent resends its complete history to the model at every step, costs will snowball. Production systems need to retain key decisions and references to original artifacts, summarize earlier conversations in layers, and allow the model to retrieve the original text on demand.

Layer 6: Observability and Evaluation—HTTP 200 Is Meaningless for an Agent

With traditional services, an HTTP 200 response generally means that the request succeeded. An Agent, however, may return a fluent passage without actually completing the task. Three groups of metrics must therefore be monitored simultaneously.

Infrastructure metrics:

• Queue wait times and Worker utilization;
• Sandbox startup latency, timeout rate, and abnormal exit rate;
• Error rates for databases, caches, and external services.

Agent execution metrics:

• Number of steps, model calls, and tool calls per task;
• Tokens, cost, and end-to-end latency;
• Tool parameter validation failure rate;
• Number of retries, rollbacks, human takeovers, and infinite loops.

Business outcome metrics:

• Task completion rate;
• Result accuracy;
• User modification or adoption rate;
• Average cost per successful task.

Every execution should generate a Trace spanning the entry point, models, tools, databases, and sandboxes. Logs only tell you that “an error occurred at a particular step.” A Trace can answer, “Why did the model choose this tool at step eight, and how did that step affect the final result?”

Evaluation should also be integrated into CI/CD. Code tests validate deterministic logic, trajectory evaluations verify whether the Agent took reasonable steps, and outcome evaluations determine whether the task was actually completed. Comparing only the final text can conceal dangerous paths: two Agents may both generate the correct report, while one of them may have accessed data it should not have.

Layer 7: Policies and Budgets—Give the Agent Brakes First

Agent failures do not always take the form of exceptions. An Agent may also “fail diligently”: searching endlessly, repeatedly calling the same tool, and continuously expanding its context until it exhausts its budget.

Every task should therefore have hard limits on:

• Maximum number of steps;
• Maximum execution time;
• Maximum model Tokens and cost;
• Number of calls per tool;
• Number of parallel branches;
• Number of consecutive steps without progress.

Termination conditions should not be left entirely to the model. Deterministic rules can detect repeated consecutive calls, unchanged output hashes, or prolonged lack of state changes, and then terminate the task or escalate it to a human.

A Practical Implementation Path for Most Teams

Having a complete architecture does not mean deploying a dozen services on day one. For most teams, the following progression is more realistic.

Stage 1: Start with a Monolith, but Define Clear Boundaries

The Web service, Agent Runner, and tool adapters can live in the same code repository and even be deployed in the same service initially. However, state must be written to a database, and task_id, run_id, and Trace identifiers must be generated consistently. Limits on steps, time, and cost should be established at this stage.

Stage 2: Move Long-Running Tasks Out of the Request Process

When tasks frequently take more than a few dozen seconds or require concurrent tool calls, introduce a queue and independent Workers. The entry point should only create tasks, while Workers scale according to task type. Browsers and code execution should be moved into isolated sandboxes.

Stage 3: Introduce Durable Orchestration and Human Approval

When tasks involve hours of waiting, external side effects, or multi-stage compensation, introduce durable workflows. Actions such as sending emails, publishing content, and modifying production data should be placed behind approval nodes.

Stage 4: Build Evaluation Sets and Multi-Model Routing

Build evaluation sets from actual failed tasks rather than relying only on idealized cases written by the team. Only then should you implement model routing and automatic fallback; otherwise, so-called cost reduction may merely hide the cost of failure.

What Has Not Yet Been Standardized

The emergence of architectural patterns does not mean the industry has completed standardization. At least four questions still have no unified answer:

• Should long-term memory be controlled by the application, the model platform, or the users themselves?
• How should identity propagation and fine-grained authorization for MCP tools interoperate across platforms?
• How much of an Agent’s trajectory should be recorded to balance debugging, privacy, and the protection of inference data?
• Does multi-Agent collaboration actually improve success rates, or does it merely amplify cost and uncertainty?

Multi-Agent systems are particularly worth scrutinizing. Many tasks can be completed with one planner and several deterministic Workers; there is no need to begin by assembling a “virtual company.” Every additional autonomous Agent increases the difficulty of context synchronization, permission assignment, failure attribution, and cost control. A multi-Agent architecture should be driven by unavoidable complexity, not treated as the default option on a product page.

Assessment: Agent Infrastructure Is Fundamentally About Putting Guardrails Around Uncertainty

Over the past two years, Agent development has focused primarily on Prompts, reasoning capabilities, and the number of available tools. The industry is now beginning to make up for its neglected infrastructure work—and that is a good thing.

The emerging general pattern can be summarized in one sentence: asynchronous execution handles long-running tasks, durable state withstands failures, sandboxes constrain the boundaries of action, layered memory manages context, and unified observability explains every decision.

This will not suddenly make models smarter, but it can turn an occasionally fallible model into a manageable production component.

For developers, the most worthwhile investment right now is not wrapping yet another “universal Agent class,” but answering a few basic questions first: Can a task continue after the process restarts? Will duplicate tool execution cause side effects? Can users cancel tasks? How much did a failed run cost? Whose data did the model read?

If these questions have no answers, the application does not have Agent infrastructure—it merely has a model call that runs for a relatively long time.

References

• Model Context Protocol Specification: The official MCP specification repository, useful for understanding how tools, resources, and context are connected.
• OpenTelemetry Collector: An open-source component for unified collection of Traces, Metrics, and Logs, which can be used to build end-to-end Agent observability.
• Temporal: A durable workflow engine suited to long-running tasks, retries, scheduled waits, and human approval scenarios.
• LangGraph: An open-source framework for stateful Agent orchestration, whose checkpointing, state graph, and human-in-the-loop designs can serve as useful references.
• gVisor: An application-kernel sandboxing project that can help explain isolation approaches for untrusted code execution environments.