Kapa in Practice: Trimming RAG Context to Keep Only the Truly Useful Parts

Kapa Cuts RAG Context Down to the Bone — and Gets Better Results

In early July, Kapa.ai published a practical postmortem on its official blog explaining how it turned context pruning in its RAG system from a “nice-to-have optimization” into a mandatory stage in the pipeline. For teams building RAG systems, this article is more worth reading than most academic papers — it’s not about “we tried a new idea,” but about “what broke in production and how we eventually fixed it.”

Here’s the conclusion first: Kapa’s approach reduced the number of tokens sent into the LLM by more than 50% on average, while also improving factual accuracy. This is not the kind of marginal “we improved things by 3%” optimization. It’s a change significant enough to directly impact both infrastructure cost and user experience.

The Starting Point: Why Reranking Still Isn’t Enough

Anyone who has built RAG systems knows the classic pipeline: vector retrieval recalls the Top 50 chunks, a reranker reorders them and keeps the Top 5–10, then everything gets stuffed into the prompt. This works fine in demos, but starts falling apart in production:

• The user asks “how do I configure webhook retry policies,” and each of the five retrieved documents is 2,000 tokens long, while only three sentences actually answer the question;
• The rest is background information, unrelated API docs, or even outdated content that conflicts with the current version;
• The model gets distracted by all this noise and either misses the key point or combines conflicting information into hallucinations.

Kapa’s observation is: reranking solves “which documents are more relevant,” but not “which sentences inside the documents are actually useful.” These are different levels of granularity. A highly relevant document can still contain 80% noise for the current question, and a high rerank score won’t help.

This is also why more and more teams are starting to treat pruning as a standalone component — it complements reranking rather than replacing it.

Kapa’s Approach: Sentence-Level Relevance Judgement

Kapa’s core idea is actually straightforward: split the reranked documents into sentences, then use a lightweight model to determine sentence by sentence whether each one contributes to answering the current question. Keep the useful ones and discard the rest.

But implementing this required several key engineering decisions:

First, they chose “sentence” as the granularity instead of “paragraph” or “token.” Paragraphs are too coarse because useful and useless information are often mixed together. Token-level approaches (such as LLMLingua-style compression) achieve high compression ratios but are difficult to interpret and debug. Sentences turned out to be the right balance between readability and precision, and Kapa explicitly says this choice came after comparing multiple approaches.

Second, they use a small model for classification instead of an LLM. Calling GPT-4 for every sentence would explode costs immediately. Kapa uses a specially fine-tuned lightweight classification model that takes “question + candidate sentence” as input and outputs a relevance score between 0 and 1. Latency stays within tens of milliseconds, and the cost is almost negligible.

Third, their training data construction is particularly interesting. They reverse-engineered training samples from existing QA pairs: for questions with known answers, the source sentences used in the answers are labeled “should keep,” while other sentences are labeled “should discard.” This self-supervised data generation process scales well and avoids manual annotation.

Fourth, they intentionally preserve some redundancy. They don’t use a hard threshold. Instead, they maintain a dynamic tolerance band — high-scoring sentences are always kept, low-scoring ones are always removed, and borderline cases are decided dynamically based on the current total context length. This is a very practical engineering trick that avoids edge cases where a sentence gets wrongly eliminated for being 0.01 points below the threshold.

Four Failure Modes of Long Contexts

Kapa’s blog also summarizes four common failure modes of LLMs in long-context scenarios. This section is worth highlighting on its own:

1. Lost in the Middle: This is a well-known issue. Models pay significantly less attention to information in the middle of long contexts compared to the beginning and end. The longer the context, the more information gets “buried.”
2. Distraction: Irrelevant content actively interferes with the model’s reasoning path. Even if the correct information is present, nearby semantically similar but conflicting noise can easily mislead the model.
3. Confusion from Contradictions: Documentation repositories often contain outdated content left behind from version updates. When old and new docs coexist, the model may merge information from multiple versions into a single response.
4. Cost Blowup: This is not a quality issue but a cost issue. A 32k-context model can cost several times more than a 4k-context model. If 80% of the tokens are noise, you’re essentially burning money for the cloud provider.

Pruning addresses all four problems simultaneously, which is why it’s not an “optional optimization” but a “required component.”

A Concrete Example

Kapa provides a real example in the article: a user asks, “How do I set custom headers in the Python SDK?” The retrieved documents include the SDK installation guide, authentication docs, request configuration docs, and error handling docs.

• After reranking, 3 documents remained, totaling around 6,000 tokens;
• After pruning, only 4 sentences remained, totaling around 200 tokens;
• Those 4 sentences precisely contained the code snippet showing how to pass the headers parameter during client initialization, plus a short explanation;
• The generated answer became shorter, more accurate, and no longer mixed in unrelated authentication workflows.

A 30× compression ratio is not an edge case — Kapa says this level of compression is extremely common in their production traffic because documentation inherently has low information density.

How Pruning Works Together with Reranking

It’s important to emphasize that Kapa does not replace reranking with pruning. Their pipeline roughly looks like this:

• Vector Retrieval: Recall the Top 50 candidate chunks
• Reranking: Use a cross-encoder reranker and keep the Top 10
• Sentence-Level Pruning: Retain only genuinely relevant sentences
• Prompt Assembly: Reconstruct the retained sentences in original order with necessary context markers
• LLM Generation

Each step progressively narrows the context, but at different levels of granularity. Reranking narrows at the document level, while pruning narrows at the sentence level. Combined, they produce the 30× compression effect.

Easy-to-Overlook Implementation Details

Preserving sentence order and context markers. Kapa specifically notes that after pruning, you can’t just throw isolated sentences into the prompt. Their relative order in the source text and document provenance markers must be preserved, otherwise the model loses the signal that “these sentences belong to the same topic.”

Coreference resolution issues. The biggest risk with sentence-level pruning is pronouns like “it” or “this” referring to earlier text that has been removed. Kapa handles this by checking for unresolved references during pruning. If a sentence contains unresolved pronouns, the preceding sentence is retained as well.

Cold-start fallback strategy. If the pruning model assigns low scores to all sentences — meaning retrieval itself likely failed to find relevant content — the system falls back to using only the reranked results, avoiding “empty context” situations that would otherwise encourage the model to fabricate answers.

Why This Direction Matters

Looking back from mid-2026, the optimization focus in RAG has clearly shifted from “how to retrieve more accurately” to “how to make the context cleaner.” This evolution was inevitable: vector databases, embedding models, and rerankers have all become highly optimized with diminishing returns, while the problem of noisy context remained largely unaddressed.

The value of Kapa’s system is that it has been running in production for a long time — it’s not just a paper concept. The metrics in the article (halved token counts, improved accuracy, reduced costs) come from real business traffic.

For developers, if your RAG system is already live and token usage remains high, pruning is currently one of the highest-ROI optimization directions available. The implementation is not especially complicated either: fine-tune a small model for sentence-level relevance classification, place it after reranking, and measure the results. Even training your own model isn’t difficult — you can generate training data backward from existing QA pairs.

And if you don’t want to build the entire pipeline yourself, using aggregation platforms like OpenAI Hub to compare rerank and embedding models from different vendors is also a convenient option — one API key for GPT, Claude, Gemini, DeepSeek, all compatible with the OpenAI format, making A/B testing easier without constantly switching SDKs.

RAG has already moved beyond the stage of “making it work.” The challenge now is “making it both cheap and accurate.” Pruning will very likely become a standard component in every serious RAG system over the next year.

References

• To Improve RAG Performance, Start with High-Quality Context Pruning - Zhihu Column — A systematic Chinese-language overview of Context Pruning, including ideas for constructing training data