ScarfBench Released: The First AI Agent Benchmark for Enterprise Java Framework Migration

At the end of June 2026, IBM Research officially released and open-sourced ScarfBench — the industry's first AI Agent benchmark specifically designed for enterprise Java application cross-framework migration scenarios. The release of this benchmark fills a critical gap in current AI code migration evaluation: "behavior-preserving cross-framework refactoring." It provides a quantifiable and reproducible yardstick for measuring the capabilities of large language models and coding agents in real-world enterprise legacy system modernization tasks.

The benchmark has been simultaneously released on the Hugging Face platform with a technical blog, datasets, and accompanying evaluation scripts, while the full paper has been published on arXiv. Together with recent efforts such as AWS AI Labs' MigrationBench (focused on Java version upgrades), ScarfBench marks the entry of AI code migration evaluation into a new era of "cross-framework, cross-paradigm" assessment.

1. Why Is ScarfBench Needed?

Over the past two years, AI Coding Agents have steadily improved their performance on benchmarks such as SWE-Bench, HumanEval, and LiveCodeBench. Some frontier models have even approached senior engineer-level performance on certain subsets. However, in real enterprise production environments, one of the biggest developer pain points is not "writing an algorithm problem solution," but rather the modernization of legacy systems — especially the deeply entangled framework migration challenges within the Java ecosystem.

Typical scenarios include:

• Upgrading from Spring Boot 2.x to Spring Boot 3.x (accompanied by the global namespace transition from javax.* to jakarta.* under Jakarta EE);
• Migrating from legacy web frameworks such as Struts / Apache Wicket to Spring MVC or Quarkus;
• Migrating from Java EE (JBoss / WebLogic) application stacks to cloud-native lightweight frameworks such as Micronaut or Quarkus;
• Switching the ORM layer from older versions of Hibernate to standard JPA implementations or jOOQ;
• Upgrading the testing stack from JUnit 4 to JUnit 5, along with coupled migrations involving Mockito, AssertJ, and related dependencies.

These tasks share several characteristics: large-scale code modifications, complex cross-file dependencies, and strict runtime behavior preservation requirements. Traditional code completion benchmarks are completely incapable of capturing the difficulty of this kind of "system-level refactoring." Although MigrationBench, previously released by AWS AI Labs, provides 5,102 Java 8→17/21 upgrade projects, it fundamentally focuses on language-level evolution within the same framework and does not address the higher-order challenge of "framework paradigm switching."

The IBM research team explicitly states in the paper that ScarfBench aims precisely to fill this gap — putting AI Agents under the real pressure of "rewriting half an application."

2. Core Design of ScarfBench

1. Dataset Composition

ScarfBench carefully selects a set of real, complete, compilable, and testable enterprise-grade Java projects from the open-source ecosystem, covering multiple typical migration paths. Each sample includes:

• A complete source framework code repository (including Maven/Gradle build scripts);
• A reference implementation for the target framework (used as ground truth);
• The original test suite (for behavior preservation verification);
• Migration specification documents (describing which observable behaviors must be preserved);
• Build and runtime environment images (Dockerized to ensure reproducible evaluation).

This design follows the same "PR-based evaluation" philosophy as SWE-Bench, but goes further — ScarfBench requires agents to complete not just a single patch, but an entire migration project.

2. Task Type Hierarchy

ScarfBench divides migration tasks into multiple difficulty levels:

| Level | Typical Task | Complexity Characteristics | |------|---------|-----------| | L1 | API renaming (e.g., javax→jakarta) | Mechanical and regex-friendly | | L2 | Annotation semantic migration (e.g., Spring @Autowired ↔ CDI @Inject) | Requires semantic understanding | | L3 | Configuration paradigm migration (XML → Java Config → YAML) | Cross-file dependencies | | L4 | Core framework API rewrites (e.g., Struts Action → Spring Controller) | Architectural-level transformation | | L5 | Multi-module coordinated migration (Web layer + persistence layer + testing layer) | Full-stack consistency |

3. Evaluation Metrics: Behavior Preservation Is Central

Unlike many benchmarks that focus only on "code similarity," ScarfBench treats behavioral equivalence as the primary metric. Specifically, evaluation dimensions include:

• Build Pass Rate: Whether the migrated project can be successfully built;
• Test Pass Rate: Whether the original test suite passes completely under the new framework;
• Runtime Equivalence: Verifying semantic consistency by observing HTTP responses, database state changes, log events, and other side effects under identical inputs;
• Code Quality Metrics: Cyclomatic complexity, duplication rate, SpotBugs static analysis, etc.;
• Migration Completeness: Whether legacy framework API calls still remain.

4. Agent Evaluation Protocol

ScarfBench provides agents with a standardized working environment:

• A readable and writable project workspace;
• A complete shell toolchain (Maven, Gradle, JDK, Git);
• File system access;
• Optional network access (for consulting documentation).

Agents must autonomously complete the full loop of exploration → planning → rewriting → verification → fixing within constrained token and time budgets.

3. Differentiation from Existing Benchmarks

ScarfBench does not emerge in isolation, but instead forms a clear complementary relationship with several recent important efforts:

• MigrationBench (AWS AI Labs, 2025): Focuses on Java 8→17/21 upgrades across 5,102 projects, but does not involve framework paradigm switching;
• SWE-Bench / SWE-Bench Verified: Centers on single-PR bug-fixing tasks across multiple languages, but with smaller task granularity;
• xbench (Sequoia China, 2025): A dual-track evaluation system emphasizing evergreen evaluation and vertical-domain practical value, positioned for tracking general agent capabilities;
• AgentBench (Tsinghua et al., 2023): An early systematic agent benchmark covering multiple scenarios but with limited depth.

ScarfBench's unique value lies in formally bringing the "framework modernization" problem — one of the biggest headaches for CTOs in enterprise development — into the scope of AI capability evaluation. IBM Research candidly stated in its blog that this is a category of tasks "that traditional benchmarks cannot capture, yet enterprises are precisely the most willing to pay for."

4. Initial Evaluation: How Far Are Current Agents from Production Readiness?

According to preliminary evaluation results disclosed by IBM Research in its Hugging Face blog (specific numbers subject to the official leaderboard), mainstream frontier models and coding agent frameworks show a clear three-stage performance cliff on ScarfBench:

• On L1-L2 tasks, frontier models already achieve fairly strong pass rates. Mechanical namespace replacements and annotation migrations can largely be automated;
• Once entering L3 configuration paradigm migrations, pass rates drop significantly. Models struggle with holistic refactoring across files and formats (XML/YAML/Java Config);
• At the L4-L5 architectural and full-stack migration levels, most agents still cannot complete end-to-end tasks without breaking existing tests. Behavioral drift, compilation failures, and repetitive error-fixing loops are the three most common issues.

These results align closely with enterprise developers' practical experience — AI can already help eliminate repetitive work, but fully taking over a repository-wide Spring Boot 2→3 migration still requires human engineers to supervise.

5. What Does This Mean for Chinese Developers?

Java remains the dominant technology stack in Chinese enterprises (finance, telecommunications, government, e-commerce). Tasks such as Spring Boot 2→3 upgrades, Struts decommissioning, and version alignment between Dubbo and Spring Cloud are real engineering challenges hanging over many teams. The release of ScarfBench means:

1. A Quantifiable Capability Map: Teams now have a benchmark more closely aligned with real migration scenarios when selecting AI coding assistants;
2. Reusable Evaluation Infrastructure: Teams developing in-house coding agents can draw on ScarfBench's Dockerized sandbox and behavioral equivalence validation methods to build private evaluations;
3. Clear Signals of Capability Weaknesses: Cross-file and cross-paradigm refactoring above L3 remains an AI weak spot, making it a high-value direction for differentiated competition among domestic agent teams;
4. Opportunities for Collaborative Dataset Building: IBM Research also calls on the community to contribute more real migration cases. Chinese open-source communities could contribute migration samples involving local frameworks such as Dubbo, Sofa, and TongWeb, influencing the benchmark's evolution in return.

6. Broader Reflections in the Tech Industry

The emergence of ScarfBench also prompts renewed thinking about the evolution of AI Coding Agent evaluation paradigms. From early single-function completion tasks, to SWE-Bench's single-PR fixes, and now to repository-wide framework migration, the granularity of evaluation units is expanding from "lines" to "modules" and then to entire "systems." The underlying logic is clear: as model capabilities improve, the truly scarce skill is no longer "whether it can write code," but "whether it can manage the long-tail constraints of complex systems."

The "Profession Aligned Eval" philosophy repeatedly emphasized by Sequoia China during xbench's launch closely aligns with ScarfBench's direction — bringing evaluation closer to real business workflows and making benchmark scores correspond to monetizable work output. ScarfBench can be viewed as a concrete implementation of this philosophy within the massive enterprise Java vertical.

7. Access and Next Steps

Developers can get started quickly through the following methods:

• Read the official IBM Research blog on Hugging Face (see reference links below);
• Pull the datasets and evaluation scripts from Hugging Face Hub;
• Explore the Dockerized evaluation environment in the accompanying repository and submit agent results to the leaderboard.

IBM Research stated that ScarfBench will be maintained as a continuously evolving benchmark — the covered framework matrix will expand alongside the evolution of Spring, Quarkus, Micronaut, and others, while migration paths will gradually extend beyond Java into JVM ecosystems such as Kotlin and Scala. In the next phase, the team also plans to introduce evaluation protocols for multi-agent collaborative migration, simulating the division of labor in real development teams involving "architects + backend developers + testers."

Conclusion

The release of ScarfBench is a critical piece in the journey of AI Coding Agents toward "enterprise-grade productivity." It brings migration tasks — hidden deep inside enterprise repositories, painful yet commercially valuable — into the spotlight. For model teams, it represents a new list of hard problems; for enterprise teams, it provides a realistic capability assessment; and for the entire industry, it clearly points to the true battleground of the next phase of competition — not on LeetCode, but inside Spring Boot applications that have been running for ten years.

References

• ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration (Official IBM Research Blog) — The official introduction to ScarfBench, detailing the benchmark's design motivation, dataset composition, evaluation protocol, and initial results.
• IBM Research Homepage on Hugging Face Hub — ScarfBench datasets, evaluation scripts, and other models and benchmark resources released by IBM Research can be found here.