A 4B Small Model Cuts Retrieval Costs to 1%

On August 6, Neon unveiled a post-training approach for agentic retrieval. Working with Castform, it used reinforcement learning to turn Qwen3.5-4B, a general-purpose open-source small model, into a document search specialist. In Neon’s published evaluation, the 4B model achieved an average score of 1.447, higher than GPT-5.6 Sol’s 1.369 and GPT-5.4’s 1.377, while costing only about 1/100 as much per inference as frontier models.

What truly deserves attention is not the easily misinterpreted headline that “a 4B model beat GPT-5.6,” but a more practical engineering conclusion: retrieval in agentic systems does not necessarily require the smartest model—only one that has been trained well enough to find information.

If these results can be reproduced across more datasets and production environments, they could directly reshape the cost structure of RAG and agents.

Agentic Retrieval Is Magnifying Search Costs

The core workflow of traditional RAG is not complicated: split documents into chunks, generate embeddings, use the user’s question to find semantically similar content in a vector database, and then have a large model answer based on that content.

It is more like a “semantic Ctrl+F.” One question typically corresponds to one or a small number of searches.

Agentic retrieval, however, works differently. When faced with a complex question such as “Compare the changes to the permissions system across the last three versions and determine which upgrade caused the compatibility issue,” the model usually needs to:

1. Break the original question into multiple subquestions;
2. Decide which keywords to search for first;
3. Read the first round of search results;
4. Rewrite its queries based on missing information;
5. Cross-check different documents, version histories, and code snippets;
6. Finally organize the evidence and generate an answer.

This is a form of multi-hop search. It more closely resembles how humans conduct investigations than a single vector retrieval does, and it usually performs better—but at the cost of rapidly increasing the number of model calls.

Neon provides a representative estimate: completing this kind of search with GPT-5.6 Sol takes more than 10 seconds and costs about $0.03. That may not seem like much for a single request, but if a customer service agent handles 100,000 requests per day and each request requires multiple rounds of retrieval, the expense is no longer negligible.

Latency is an even bigger problem. Every time the agent performs a search, it must wait for the model to reason, invoke a tool, read the results, and decide what to do next. The longer the chain, the more pronounced the tail latency. What users notice is not how many tokens the model generates per second, but how long the loading spinner remains on the screen.

Rather than continuing to push down the price of frontier models, Neon and Castform took a different approach: separate retrieval decisions from the general-purpose large model and hand them over to a specially trained small model.

It Learns Not Knowledge, but “What to Search for Next”

This post-training effort uses Qwen3.5-4B as its base model. A 4B-parameter model is not large by today’s standards, making single-GPU deployment, quantized inference, and local operation relatively straightforward. The problem is that a general-purpose small model without specialized training may not excel at multi-round retrieval: it may generate overly broad keywords or stop too early after finding a result that merely appears relevant.

Castform’s job was to have the model repeatedly perform retrieval tasks and then reward it based on the results.

The training loop can be roughly divided into three layers:

• Query layer: Can the model generate an effective next search query based on the current question and the evidence already collected?
• Evidence layer: Do the search results contain the correct documents, and do the cited passages actually support the conclusion?
• Answer layer: Is the final answer correct and complete, and is it grounded in the retrieved material?

The model first attempts to complete the task, after which the system scores the result. That score is then used for subsequent reinforcement learning. In other words, the model does not improve its accuracy by memorizing every enterprise document. Instead, it practices a more specific skill: when it encounters an information gap, determining what to search for, where to search, and when to stop.

This differs significantly from training a general-purpose chat model. A general model must balance writing, coding, reasoning, translation, and factual question answering. A retrieval model only needs to focus its limited capacity on tool selection, query rewriting, and evidence location. It has fewer parameters, but also a narrower task boundary.

It can be compared to a research assistant within a company. A frontier model is like a highly knowledgeable senior consultant who can handle almost any problem but charges by the hour. This 4B model is more like a specially trained information specialist who does not set corporate strategy, but instead quickly finds the right files and passages. For search tasks, the latter may be no worse—and is far cheaper.

PostgreSQL Is Not Just for Storing Documents; It Is Also the Retrieval Execution Layer

Neon provides the document storage and search infrastructure for this system. As a serverless PostgreSQL service, Neon separates compute from storage and supports automatic scaling, scale-to-zero, and Git-like database branching. These capabilities were originally designed primarily for cloud-native applications, but are now also being used for agent state, documents, and tool-call data.

Its retrieval system does not rely solely on vector similarity. Instead, it combines two approaches:

• BM25 full-text search excels at matching product names, error codes, function names, and specialized terminology;
• Vector search excels at finding content that is phrased differently but semantically similar;
• The results from both paths are then merged using a ranking-fusion method similar to RRF.

This matters. Pure vector search works well for natural-language paraphrases, but when dealing with ERR_CONNECTION_RESET, a specific API field, or an exact version number, traditional keyword search is often more reliable. Conversely, if a user asks “How do I undo an accidental operation?” while the document title says “Transaction Rollback,” vector search is more likely to connect the two.

Effective enterprise search is usually not a choice between keywords and vectors. Instead, it uses hybrid retrieval and then lets the model decide what additional information to seek in the next round. Neon is responsible for “finding information in the database,” while the model trained by Castform is responsible for “deciding what to find.” The division of labor is relatively clear.

How Should We Interpret 1.447 vs. 1.369?

According to data published by Neon, the post-trained 4B model achieved an average evaluation score of 1.447, compared with 1.369 for GPT-5.6 Sol and 1.377 for GPT-5.4. The small model received the highest score in this evaluation.

This result demonstrates the potential of specialized post-training, but it does not directly imply that “Qwen3.5-4B comprehensively outperforms GPT-5.6 Sol.” At least three points must be considered separately.

First, These Results Come From a Specific Retrieval Environment

The models operated against a given document collection, search tools, and task distribution. Both the training objectives and evaluation metrics focused on document location, passage citation, and final answers. The evaluation measured retrieval workflow performance—not general reasoning, code generation, or open-domain question-answering capabilities.

Second, the Evaluation Was Published by the Solution Providers

The information disclosed so far is sufficient to explain the training approach, but developers still need to see more complete details about the test set’s composition, sample size, variance, failure cases, prompt settings, and inference budget. In particular, any template-level or semantic overlap between the training and test data would significantly affect the strength of the conclusions.

Third, the Cost Comparison Depends on Deployment Conditions

Based on Neon’s claimed 100-fold difference, if one GPT-5.6 Sol request costs about $0.03, the corresponding cost for the small model would be roughly $0.0003. But that does not mean every team can directly achieve the same figure.

Actual costs will also be affected by factors including:

• GPU model, quantization precision, and batch size;
• Average number of search rounds per task;
• Context length and number of retrieved results;
• Whether the model is self-hosted, as well as GPU idle time;
• Costs associated with the search database, reranker, and network calls.

The small model is clearly cheaper. Whether it is exactly 100 times cheaper depends on the specific system.

The Real Value Lies in Dividing the Agent Across Different Tiers of Models

Many previous agent products used a single frontier model throughout the entire pipeline: understanding intent, planning tasks, generating search terms, invoking tools, reading results, and producing answers. This architecture is quick to develop, but expensive and wasteful of model capabilities.

A more reasonable approach is to use a tiered model architecture:

• Small models handle intent classification, routing, and query rewriting;
• Specialized retrieval models handle multi-hop search and evidence collection;
• Mid-sized models handle structured extraction and simple summarization;
• Frontier models are reserved for difficult reasoning, conflict resolution, and final presentation.

This is why the approach matters more than any single leaderboard score. It shows that “model routing” for agents is not merely about selecting a suitably priced model from several APIs. It can be refined to assign different models to different stages of the same task chain.

For an enterprise knowledge-base agent, 80% of the work may simply involve finding the right document. Involving the most expensive model at every step is like asking senior engineers to look up logs for everyone every day: they can certainly do it, but it is clearly not an economical way to organize the work.

Which Use Cases Are Worth Exploring?

These 4B retrieval models are particularly well suited to scenarios with stable data boundaries, high query volumes, and results that can be automatically verified:

• Internal enterprise knowledge bases and customer service SOPs;
• Search across API documentation, code repositories, and troubleshooting manuals;
• Locating clauses in legal contracts;
• Retrieving sales call records and customer information;
• Tracking differences across multiple versions of product documentation;
• Systems with strict data-localization and access-isolation requirements.

However, if the data changes dramatically every day, the task lacks a clear reward signal, or answers depend heavily on open-world knowledge, training a specialized model may not be worthwhile. A 4B model is also more likely to go off track during long reasoning chains, so production systems should retain safeguards such as maximum search-round limits, source allowlists, citation validation, and fallback mechanisms.

Permissions must not be overlooked either. A model that “knows how to search” does not necessarily “have the right to see everything.” The retrieval layer must apply tenant-, user-, and document-level permission filters before executing a query, rather than exposing content to the model first and expecting it to keep the information confidential.

Small Models Will Not Replace Frontier Models, but They Will Take Over a Large Share of Calls

Neon and Castform’s experiment reflects an increasingly clear direction: use large models to generate training tasks, define rewards, or handle difficult cases, then distill or reinforce high-frequency, clearly bounded capabilities into small models.

This will not make frontier models less valuable. On the contrary, frontier models will increasingly be used for task planning, difficult judgments, and final review. What will be compressed are the repetitive, high-frequency, verifiable intermediate steps that have long been handled by expensive models.

Therefore, the most accurate assessment of this release is not that “a 4B model defeated a flagship model,” but rather: in the narrow field of document retrieval, a small model trained with reinforcement learning may already be able to maintain performance while reducing costs by two orders of magnitude.

For developers, the next step should not be limited to downloading the model and running a demo. They should use their own documents, query logs, and permission systems to reevaluate four metrics: task success rate, P95 latency, end-to-end cost, and the percentage of failures that require fallback to a large model.

Only if all four metrics hold up can agentic retrieval truly evolve from an effective but expensive capability into infrastructure that can be deployed at scale.

References

• ITHome: Neon Open-Sources 4B Model With Document Search Capabilities Surpassing GPT-5.6 Sol — Summarizes the information published by Neon and Castform, including evaluation results, costs, and the training process.
• GitHub: Neon Serverless PostgreSQL Project — Neon’s open-source code repository, providing details on its compute-storage separation, compute nodes, and storage engine architecture.