The "Plumbing" of LLM Applications: How Developers Are Handling Memory, Context, and Multi-Model Routing

A Reddit Post That Hit Every AI Developer’s Pain Point

In early July, a post titled “Developers building with LLMs, how are you actually handling memory, context persistence, and multi-model routing?” blew up on r/MachineLearning. The author described spending “less than 30% of the past few months actually building the product, and the rest fighting context management, memory persistence, and multiple LLM providers.”

The thread received more than 800 replies, and nearly every developer who had built an LLM application agreed on one thing: everyone is independently rebuilding the exact same wheels.

This isn’t a new problem, but even in 2026, there’s still no universally accepted solution. GPT-5.1, Claude Opus 4.8, Gemini 3 Pro—the model-layer arms race is in full swing. But at the application layer, engineers are still hand-writing glue code for seemingly basic problems like “how do I make the Agent remember last week’s conversation,” “what do I do when context overflows,” and “which model should this task be routed to?”

We reviewed that discussion along with industry developments over the past six months, and one thing became clear: the “infrastructure layer” of LLM applications is rapidly taking shape, but it’s still in a very early Wild West phase.

Memory: A Paradigm Shift from “Stuff It Into the Prompt” to “Externalized State”

Let’s start with Memory. The term is now used so broadly that it needs to be broken down.

When developers talk about “memory,” they’re usually referring to at least three different things:

• Session memory: short-term context within a conversation session; essentially context window management
• Long-term memory: persistent memory across sessions, such as what the user said last week or a client’s preferences
• Semantic memory: structured knowledge bases, usually implemented through RAG

From 2024 through the first half of 2025, the mainstream approach was “just use RAG.” Developers treated vector databases as a cure-all: Pinecone, Weaviate, Chroma, Qdrant, rotating through them while retrieving top-k results and stuffing them into prompts. But after six months, the problems became obvious:

Retrieval accuracy falls apart in complex scenarios. Semantic similarity does not equal relevance. If a user asks, “How did we solve that bug last time?”, vector retrieval will likely return a pile of unrelated fragments containing words like “bug” and “solve.”

Context bloat spirals out of control. To be safe, teams shove in the top 20 results, leading to tens of thousands of tokens per call, exploding both cost and latency.

Nobody manages conflicts between memories. Three months ago the user said “I prefer concise responses,” and one month ago they said “Give me more detailed explanations.” Both get retrieved. Which one should the model follow?

What’s interesting is that over the past six months, a clear shift has emerged: from “retrieval-based memory” to “editorial memory.”

Some teams are starting to maintain memory like a “living document.” After each interaction, a lightweight background model (usually something cheap and fast like Haiku 4.5 or DeepSeek V3.2) updates a structured user profile and session summary, instead of blindly pushing new embeddings into a vector database. This idea is especially clear in the memory-bank-mcp-server project open-sourced in early July: context persistence is not about remembering everything, but maintaining a structured state that can later be “recalled.”

A Zhihu article titled “Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols, and Framework Engineering” makes a strong point: the way LLM Agents are built is shifting from “modifying model weights” to “reconstructing the runtime environment.” Memory is being explicitly externalized from the model itself. That’s an important directional shift worth remembering.

Context Persistence: What Claude Code and ChatGPT Agent Taught Us

If Memory is “what you remember,” then Context Persistence is “how you carry state through time.” The two concepts are often mixed together, but from an engineering perspective they’re completely different.

The core challenge of Context Persistence is this: an Agent task may run for minutes, hours, or even days. It may restart, switch devices, or get interrupted. How do you let it continue instead of starting over?

Anthropic’s Claude Code introduced Sub-agents and Session Resume earlier this year, and they’ve become a widely referenced example. The system serializes the complete session state to disk—including conversation history, tool call records, file modifications, and pending tasks—so it can be resumed at any time. Developers can quit the terminal, reboot their computer, and the next day run claude --continue to pick up exactly where the Agent left off.

It sounds simple, but there’s a lot of engineering nuance behind it:

• What state should be persisted and what shouldn’t? Should intermediate tool outputs be saved? If yes, storage usage explodes; if not, the Agent may need to rerun everything after resuming.
• How should state be compressed? Long conversations can easily reach hundreds of thousands of tokens. If you save everything directly, resuming requires loading it all back into context.
• How should concurrent sessions be handled? If a user opens one session on mobile and another on desktop, should memory synchronize between them?

Over the past few months, the industry has largely converged on a common approach: layered persistence + incremental snapshots. Hot data (currently active context) stays in memory; warm data (summaries of recent interactions) goes into KV stores like Redis; cold data (historical sessions and structured knowledge) goes into object storage or vector databases. Snapshots are incremental and only store changes.

In a whitepaper on Agentic AI infrastructure released in June, AWS directly compared this mechanism to “virtual machine snapshot recovery”—using state snapshots to restore Agents in sub-second time, making recovery 10–100x faster than restarting an Agent from scratch. The analogy fits well, because an Agent is fundamentally a long-running process. Operating systems solved process management decades ago; now LLM applications are reinventing it.

Multi-Model Routing: From “Choose One” to “Use Them Together”

Then there’s multi-model routing—the area with the most disagreement and arguably the most interesting developments.

In 2024, people were still debating “Should I use GPT or Claude?” By the second half of 2025, the question became “Which model should handle which task?” And by 2026, a new consensus started forming: any non-toy LLM application will inevitably be multi-model.

The reason is simple: no single model is best across every dimension.

• For complex reasoning and long contexts, Claude Opus 4.8 and GPT-5.1 are top-tier, but they’re expensive and slow
• For code generation, Claude still holds a noticeable lead in software engineering tasks
• In Chinese-language scenarios, DeepSeek V3.2 and Qwen3-Max dominate on cost-performance
• For high-concurrency lightweight classification and extraction tasks, Haiku 4.5 and Gemini 3 Flash are cheap enough that cost becomes negligible
• For multimodal tasks—especially video understanding—the Gemini family remains the only real option

In practice, developers’ routing strategies generally fall into several camps:

Rule-based camp: Write a pile of if-else statements and hardcode routing by task type. Simple and brute-force, but expensive to maintain. Every model update requires rewriting rules.

Cascade camp: Try the cheap model first, then escalate to a more expensive one if confidence is low or the user is dissatisfied. Saves money, but latency becomes unpredictable and user experience inconsistent.

Classifier camp: Train a lightweight model (or use an LLM with few-shot prompting) to make routing decisions. This is currently the most common production approach. Open-source projects like Martian and RouteLLM follow this path.

Parallel camp: Send the same request to multiple models simultaneously and use a judge model to pick the best response. High quality, but doubles the cost, so it’s only suitable for critical tasks.

But in real-world deployment, the biggest pain point usually isn’t the routing algorithm itself—it’s the infrastructure:

• Different providers use different API formats, requiring endless adapters
• Every provider has different rate limits, error codes, and retry strategies
• In China, direct access to OpenAI and Anthropic is restricted, requiring proxies, which creates stability and latency issues
• Billing and usage monitoring must be implemented manually, or the end-of-month bill becomes terrifying

That’s also why API aggregation services have gained traction among developers over the past six months. Platforms like OpenAI Hub aim to unify mainstream models (GPT-5.1, Claude Opus 4.8, Gemini 3 Pro, DeepSeek V3.2, etc.) under a single OpenAI-compatible interface. One API key can access every model, with domestic direct access and built-in handling for adapters, rate limits, and failover. For teams that don’t want to spend time on this layer, it’s genuinely convenient. But there’s a tradeoff: if your application depends heavily on fine-grained model control (such as specific system prompt formats or tool-use behavior), the aggregation layer may hide important details.

A Harsh Reality: This Infrastructure Layer Still Doesn’t Have Its Rails

At this point, one unavoidable question emerges: why, in 2026, is there still no de facto standard framework for LLM applications comparable to Rails, Django, or Spring?

Legacy players like LangChain and LlamaIndex have seen their reputations decline over the past year. The community complains that “there are too many abstraction layers, and when something breaks you have no idea where to fix it.” Newer frameworks like Vercel AI SDK, Mastra, and Inngest’s Agent Kit are more vertically focused, but none provide a complete solution spanning Memory + Context + Routing.

One viewpoint has been repeatedly cited in developer circles recently: LLM applications are still in the phase where every team is building its own runtime. It strongly resembles the period around 2005, when every web company wrote its own MVC framework. For a true Rails-level abstraction to emerge, the community first needs consensus on “what good Agent architecture looks like.” Right now, that consensus is nowhere close.

One encouraging sign is the rapid increase in MCP (Model Context Protocol) adoption over the past six months. Anthropic, OpenAI, and Google are all moving in this direction. If MCP truly becomes a cross-vendor “USB-C interface,” then standardized Memory and tool calling will finally have a foundation on which genuinely usable frameworks can emerge.

Practical Advice for Developers Building LLM Applications

Based on the Reddit discussion and recent industry developments, here are several recommendations that remain relatively uncontroversial:

1. Don’t rush into using a vector database. For 80% of Memory scenarios, a structured user profile document plus a session summary is enough. Before adopting a vector database, ask yourself: what exactly am I trying to retrieve?
2. Use incremental snapshots for context persistence. Storing complete conversation histories is the most expensive, slowest, and least scalable approach. Learn from Claude Code: layered storage with incremental updates.
3. Start multi-model routing with simple rules. Don’t train a classifier from day one. Use if-else rules to get the product working first, then consider ML-based routing once you have enough logs.
4. Abstract at the API aggregation layer. Whether self-built or using services like OpenAI Hub, don’t hardcode provider-specific APIs into business logic. Future model switching will become much cheaper.
5. Add TTLs to Memory. Human memory fades, and Agent memory should too. Setting different expiration policies for different types of memory matters more than most people expect.
6. Observability matters more than anything else. Log every prompt, response, latency, cost, and routing decision. Without observability, optimization becomes guesswork.

Closing Thoughts

This “plumbing layer” of LLM applications isn’t as glamorous as the model architectures beloved by ML engineers, nor as intuitive as the user experiences product managers like to discuss. But it determines whether your AI product survives its first 1,000 users, whether it avoids collapsing under infrastructure bills once ARR exceeds seven figures, and whether it can quickly switch to the next generation of models when they arrive.

At this stage in 2026, teams that can build robust Memory, Context Persistence, and multi-model routing systems are the ones positioned to compete for the long run in the AI application race. As for elegant frameworks and standards, they’ll come eventually.

References

• Developers building with LLMs, how are you actually handling memory, context persistence, and multi-model routing? - The original Reddit r/MachineLearning thread that sparked widespread discussion, with extensive real-world experience shared by developers in the comments
• Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols, and Framework Engineering - A Zhihu article reviewing externalization mechanisms in LLM Agents, offering a particularly clear theoretical perspective