REAP: Letting Production Environments Build Their Own Coding Agent Benchmarks

If you've been working on Coding Agents recently, you've probably run into the same problem: your SWE-bench scores look great, but the moment customers try the agent on real codebases, things fall apart.

A newly released arXiv paper, now also being discussed on r/MachineLearning, proposes a new solution: REAP (Relevance and Execution-Audited Pipeline), a pipeline that automatically extracts evaluation tasks directly from real developer-Agent interactions in production environments, without human annotation. Using it, the authors created a benchmark called Harvest, where every task comes from an actual developer prompt, and validation is performed using fail-to-pass tests pulled from production systems.

The significance here isn't just "another benchmark." What's important is that it pushes Coding Agent evaluation away from "gaming public datasets" and much closer to "continuously collecting production signals."

The Three Existing Evaluation Paths — And Their Problems

First, it's important to clarify what REAP is trying to solve. Today, the industry mainly relies on three approaches to evaluate Coding Agents, and the paper systematically criticizes all of them right at the beginning:

• Online A/B testing: Closest to real-world usage, but cycles can take weeks, and it requires using real users as test subjects. A single regression in Agent behavior can translate directly into a wave of issue tickets.
• Shadow deployment: Running the Agent alongside production without affecting users. The problem is that the resulting signals are not reproducible across runs — for the same prompt, the repository state changes, dependencies change, and outputs can no longer be meaningfully compared.
• Public benchmarks: SWE-bench, LiveCodeBench, and similar datasets suffer from the most obvious issue — their language distributions, prompt styles, and repository structures simply do not resemble real production environments. SWE-bench consists entirely of GitHub issues from open-source Python repositories, while your team's monorepo may be half TypeScript, half Go, plus a pile of internal frameworks.

All three approaches share the same core problem: they are either slow, noisy, or biased. And Coding Agents evolve extremely quickly — models change, prompts change, toolchains change. If evaluation can't keep up, optimization becomes guesswork.

What REAP Actually Does

The core idea behind REAP can be summarized in one sentence: automatically convert developer-Agent interactions that already happened in production into reproducible, automatically scorable evaluation samples.

Each sample looks like this:

• verbatim prompt: The exact text the developer originally entered, including all context, vague wording, and typos
• committed diff: The final code change merged as a result of the interaction (what industry teams usually call a patch or PR)
• fail-to-pass test suite: The tests that failed before the change and passed afterward

The third point is the key. Fail-to-pass tests provide an automated correctness signal, meaning evaluation doesn't require LLM-as-a-judge or humans reviewing diffs. You simply run the Agent-generated code against the tests: red-to-green means success. Brutal but effective.

The REAP pipeline handles several messy engineering tasks:

1. LLM-based task classification: Selecting the subset of conversations worth turning into evaluation tasks — trivial ones (like renaming a variable), poorly specified requests, or endless multi-turn discussions with no outcome are filtered out.
2. Test relevance verification: Not every test modified alongside a diff actually validates the change. REAP determines which tests truly turned from red to green because of the change and which were merely incidental edits.
3. Multi-run stability checks: Running the same task multiple times to verify result stability. Unstable tasks (for example, those relying on external services or flaky tests) are discarded outright.

Only what survives all of this becomes part of Harvest.

Monorepos Are the Real Hard Problem

One detail in the paper is especially worth developers' attention: in monorepo environments, benchmarks must be "continuously re-curated."

The authors describe monorepo build infrastructure as "ephemeral." Repositories change daily, dependencies constantly move, and tests that worked a month ago may fail today because some internal library was upgraded. This means:

You cannot freeze a single commit as a permanent evaluation baseline the way SWE-bench does.

REAP's answer is to turn curation itself into a continuously running pipeline synchronized with the current repository state. Conceptually, this represents an important shift: a benchmark is no longer a static dataset file, but a dynamic view synchronized with the production system.

From an engineering perspective, this resembles database views or materialized views in stream processing. The benchmark is "materialized" from production logs.

Why This Matters Now

Zooming out a bit, one clear trend over the past six months in the Coding Agent space is this: people are increasingly recognizing that simply scaling model capability is no longer enough.

At QCon Beijing last month, Baidu's Wenxin Kuaima team shared a similar view: the real challenges Coding Agents face in engineering environments are not "the model isn't capable enough," but rather "behavior is uncontrollable, effectiveness is difficult to quantify, and optimization depends heavily on a few experts." Their proposed solution was a flywheel built around "Feedback Loop + Benchmark + Agent Engineers" — collecting real usage signals, continuously evaluating against production-like benchmarks, and turning Agent evolution into an everyday engineering activity.

REAP is essentially an automated version of the same idea. When your Agent serves tens of thousands of developers and generates hundreds of thousands of interactions per day, manual benchmark curation becomes completely unrealistic. Pipelines are the only scalable solution.

That's why REAP's value goes far beyond "just another paper." What it really provides is a template for how Coding Agent teams should build their own evaluation infrastructure:

• Stop expecting public benchmarks to predict production performance
• Stop relying on engineers casually picking a few cases for regression testing
• Fully engineer the production, validation, and updating of evaluation datasets

Several Questions Worth Following Up On

That said, the currently public information about REAP still leaves several unresolved issues worth watching:

First, how are privacy and data compliance handled? Production prompts and diffs likely contain proprietary code, internal paths, or even credentials. Turning these into benchmarks — even for internal use — introduces substantial data governance costs. The paper does not go into detail about its anonymization strategy.

Second, what is the coverage ceiling of fail-to-pass tests? Not every code change comes with tests. UI changes, configuration modifications, and documentation generation are all common scenarios where F2P tests may capture only a limited portion of work. Harvest is therefore likely biased toward "teams with strong testing culture + backend-heavy distributions."

Third, could Agents overfit to their own history? If a team uses REAP-generated benchmarks to train its own Agent, it's easy to create a self-reinforcing loop — the Agent learns its previous solutions, which may not actually be optimal. Additional diversity mechanisms would be necessary.

Fourth, how transferable is this across languages and frameworks? The paper mentions that Harvest covers seven programming languages, but production distributions are uneven. Whether low-resource languages have enough samples for stable evaluation will require further data disclosure.

Practical Takeaways for Developers

Even if you don't plan to replicate REAP, several ideas from the paper are worth adopting:

• Use fail-to-pass tests as the primary evaluation signal. They're more stable than LLM judges and faster than manual review. Applying them to your own Coding Agent regression testing can eliminate a huge amount of debate.
• Use verbatim prompts instead of cleaned prompts. Real users do not write perfect prompts. Your Agent needs to operate in noisy environments. Preserving raw inputs during evaluation is the only way to measure real capability.
• Multi-run stability checks are the baseline for benchmark quality. Any task with high single-run success rates but large variance should be treated as an unreliable signal.

Looking at the bigger picture, the Coding Agent competition of 2026 is no longer about "whose model is stronger," but about "whose evaluation and feedback loop spins faster." REAP is just one milestone on that path, but it pushes something that previously relied on manual effort, intuition, and trial-and-error much closer to a fully engineerable system.

For teams building their own Coding Agents, instead of continuing to chase SWE-bench scores, it may be more worthwhile to seriously consider this question: how much high-quality interaction data generated every day in your production environment is currently being wasted?

---

References

• REAP paper discussion - r/MachineLearning：Original discussion thread for the paper, with many additional perspectives from industry practitioners in the comments
• wagner-group/reap-benchmark - GitHub：Note that this is a different project with the same name (an adversarial patch benchmark). The code for the REAP paper discussed in this article has not yet been released publicly; this keyword is worth following for future updates.