MirrorCode: How Large a Project Can AI Build Independently?

AI Coding Evaluations Are Finally Looking Beyond a Single Issue

As of August 4, Epoch AI had recently released the MirrorCode evaluation, seeking to answer a question that is more relevant to real-world procurement decisions than “Can the model fix bugs?”: Without continuous human intervention, how large a software project can an AI coding agent complete independently?

At first glance, this may seem like merely scaling up evaluations such as SWE-bench. In reality, it changes the nature of the test.

Over the past two years, most coding benchmarks have been designed around localized tasks: give an agent an existing codebase and an issue, then ask it to add a feature, fix tests, or modify several files. Such evaluations are valuable, but they measure something closer to “Can AI take on the next ticket?” rather than “Can AI deliver an entire project?”

MirrorCode expands the scope of observation from patches to complete software. The agent must understand requirements, plan modules, establish the project structure, implement functionality, run tests, and manage dependencies between earlier and later steps. Architectural choices and errors from one stage directly affect subsequent work.

That is precisely where its greatest value lies: A software project is not a stack of independent coding problems, but a chain of decisions that can contaminate one another.

MirrorCode Does Not Measure “How Many Lines of Code Can It Write?”

As the name suggests, MirrorCode’s core idea is to have an agent reimplement existing software according to project requirements and external behavioral constraints, then use executable tests to determine whether this “mirror version” provides the correct functionality.

This differs from directly comparing generated source code. Two implementations may not share a single identical line and still both be valid as long as their external behavior is equivalent. Conversely, even if an agent generates a large amount of elegantly structured, thoroughly commented code, the project cannot be considered complete if its critical paths do not work.

Therefore, the “largest project” cannot simply be defined by lines of code. At least four dimensions must be considered simultaneously:

• Functional surface area: How many user-visible behaviors, commands, or interfaces must be implemented;
• Dependency density: Whether numerous implicit constraints exist between modules;
• State complexity: Whether the project involves persistence, concurrency, caching, permissions, or failure recovery;
• Verification difficulty: Whether the agent can use test feedback to identify the true root cause rather than merely patching surface-level errors.

A 3,000-line service with a concurrent state machine may be more difficult than a 10,000-line static transformation tool. Reducing “project size” to LOC alone is like judging the difficulty of a mathematics textbook by its page count: easy to quantify, but limited in explanatory power.

A more reasonable way to interpret MirrorCode is to observe how an agent’s success rate declines as engineering complexity increases. It does not provide a sharp dividing line—projects below a certain size always succeed, while those above it always fail—but rather a probability curve that gradually becomes steeper.

The Conclusion Is Not Romantic: Handling Large Projects Does Not Mean Delivering Them Reliably

Based on the direction shown by MirrorCode, frontier coding agents can already complete some nontrivial projects independently. They are no longer limited to completing functions; they can also create directories, select dependencies, connect multiple modules, and repeatedly run tests over longer trajectories.

But as project size continues to grow, success rates decline noticeably. This is usually not because the model suddenly forgets how to write a particular syntax, but because the engineering process begins to spiral out of control.

Here, two metrics that are often conflated must be distinguished:

1. Largest successful case: The largest project that an agent has completed in a particular run;
2. Reliable delivery boundary: The project size at which repeated runs on similar projects still achieve a success rate acceptable to the team.

The former is suitable for demonstrations; the latter is what budgets should be based on.

If an agent succeeds once in ten attempts on a large task, that success certainly demonstrates the upper limit of its capabilities—but a company cannot dismiss its engineering team on that basis. Real-world development also requires reproducible results, auditable changes, and recoverable failures, without shifting the cost of unlimited retries onto the compute bill.

Thus, MirrorCode’s answer to “How large a project can AI complete independently?” is not a dazzling line count. It is this: The occasional upper limit of the strongest agents has already reached the scale of complete small software projects, but their reliable autonomy boundary remains significantly below their largest showcased successes. The closer a project is to a real production system, the wider this gap becomes.

Why Agents Are More Likely to Collapse Suddenly as Codebases Grow

The challenge of a long-running task is not simply repeating a short task one hundred times. Long tasks introduce new failure mechanisms.

1. Early Architectural Errors Compound

When human engineers discover that a domain model is wrong, they usually stop to discuss whether to refactor it. Agents are more likely to keep adding code in the same direction: if an interface is poorly designed, they add another adaptation layer; if state definitions are confused, they add more conditional branches; if tests fail, they apply local patches.

In the short term, every step appears to be “solving the problem.” In the long term, the agent is paying compound interest on the first mistake.

This is also why long-running agents often exhibit a counterintuitive pattern: they keep editing, testing, and fixing while token consumption continues to rise, yet they get no closer to a deliverable state. High activity is sometimes not a sign of progress, but a sign that the agent is lost.

2. A Large Context Window Does Not Guarantee Reliable Project Memory

Even with a very large context window, a model cannot keep every source file, test output, design decision, and failed attempt on the same plane of attention at all times. Agents typically rely on summaries, retrieval, work logs, and context compression.

The problem is that compression loses information. Causal information—such as “why this design was chosen at the time”—is especially likely to be compressed into “the current module uses approach A.” A later agent sees only the conclusion, not the limits within which approach A applies, and therefore makes changes that conflict with earlier constraints.

In large-scale engineering, remembering the code is not enough; the agent must also remember the decisions.

3. Passing Tests Does Not Mean the Engineering Work Is Complete

Executable tests are currently the most effective reward signal for agents, but they also create a boundary around what the agent can see. Agents tend to optimize for visible assertions, while real software includes many requirements not covered by tests: exceptional paths, resource cleanup, security boundaries, observability, upgrade compatibility, and performance degradation.

If an evaluation environment asks only whether the output is correct, an agent may deliver an implementation that is functionally correct but impossible to maintain in practice. MirrorCode can cover more behaviors than a single-patch benchmark, but it still cannot be equated with production acceptance.

4. The Toolchain Amplifies Differences Between Models

An agent is not just a bare model. At minimum, it includes the model, prompts, file retrieval, terminal tools, test feedback, context compression, retry strategies, and budget controls.

The same model placed in different harnesses may behave like two entirely different products. Whether the agent is allowed to create a plan, whether it can preserve a decision log, what information it receives after a test failure, and when rollbacks are triggered can all affect the maximum project size it can complete.

MirrorCode therefore evaluates something closer to the combination of a model and an agent system, rather than producing a standalone model scorecard. For enterprise procurement, comparing only the names of foundation models is becoming increasingly meaningless.

MirrorCode, SWE-bench, and EvoClaw Measure Three Different Things

Looking at several types of evaluations together makes the boundaries of AI coding capabilities clearer.

| Evaluation Type | Typical Task | Primary Capabilities Assessed | Problems Easily Missed | | --- | --- | --- | --- | | SWE-bench-style tasks | Resolve an issue in an existing repository | Code localization, local modifications, test fixes | Long-term architecture and accumulated technical debt | | MirrorCode | Implement a complete project from requirements | Planning, setup, cross-module integration, end-to-end delivery | Long-term operations and multi-person collaboration | | EvoClaw-style continuous evolution | Complete multiple milestones sequentially in the same repository | Maintaining consistency, absorbing historical errors, continuous iteration | Ability to build a project from scratch |

MirrorCode is more like asking AI to build a house independently; SWE-bench asks it to repair a leaking window; continuous-evolution evaluations require it to maintain the house for years while residents continually make new requests.

None of the three replaces the others. An agent may excel at generating well-structured small projects from scratch but struggle to locate problems in a million-line legacy repository. Alternatively, it may be an excellent repair agent but quickly lose control once asked to make architectural decisions autonomously.

For development teams, this means that “coding capability” can no longer be compressed into a single score. Before procuring an agent, teams must at least determine whether their primary use case is greenfield development, legacy-system maintenance, or continuous feature evolution.

What Teams Should Change Now

MirrorCode’s most important impact is not proving that AI can write a few thousand more lines of code. It is reminding teams that task decomposition, engineering environments, and acceptance mechanisms are becoming the primary variables limiting agent output.

Rewrite Requirements as Executable Contracts

“Build an internal knowledge base” may provide direction for a human team, but for an agent it is almost equivalent to providing no requirements at all. More effective inputs should include:

• Clearly defined user behaviors and non-goals;
• APIs, data structures, and error semantics;
• Automatically executable acceptance tests;
• Performance, security, and dependency constraints;
• Which decisions the agent may make autonomously and which require human confirmation.

In the future, part of a high-quality PRD will be translated directly into tests and policy files. Whether requirements can be verified by machines will affect whether agents can truly close the loop.

Do Not Assume “One Long Run” Is Better Than Multiple Milestones

The longer a project runs, the more severely errors accumulate. Rather than handing an entire set of requirements to an agent and letting it run continuously for hours, a more reliable approach is to establish staged checkpoints: separately validate the scaffolding, core model, critical paths, exception handling, and integration tests.

This does not reduce the agent to code completion; it controls the blast radius of errors. An error discovered in the second phase may take only tens of minutes to fix. Once it has spread across ten modules, even the strongest model may be inclined to keep applying patches.

Tie Budgets to Uncertainty

Long-running tasks cannot be managed with only a total token limit. Teams need to monitor the test pass rate, the number of repeatedly edited files, the proportion of ineffective commands, the number of rollbacks, and the cost per unit of progress.

A simple framework can help determine whether it is still worthwhile to let the agent continue:

Expected delivery cost = Cost per run × Expected number of attempts
                         + Human review cost
                         + Failure rework cost

If the agent has repeatedly modified the same group of files across several rounds without passing any additional tests, continuing to add context and tokens is often not patience—it is a sunk cost.

Route Models by Stage, Not by Employee Preference

Project planning, bulk implementation, difficult debugging, and final review require different model capabilities. Strong reasoning models are well suited to architecture and root-cause analysis, while faster and less expensive models can handle mechanical implementation, test completion, and formatting.

A truly mature agent workflow will not “use only the strongest model from start to finish.” Instead, like a compiler pipeline, it will invoke different capabilities at different stages while maintaining unified logs. Good model routing may produce greater benefits than a minor model-version upgrade.

Shift Reviews from Code Style to Systemic Risk

When 80% or even 90% of the code is generated by agents, inspecting every variable name line by line is both expensive and ineffective at identifying the main risks. Human review should prioritize:

1. Whether module boundaries are reasonable;
2. Whether data or permissions cross intended boundaries;
3. Whether failure modes are recoverable;
4. Whether tests cover real business invariants;
5. Whether the agent has “solved the test” through hard-coding or workarounds.

The value of engineers is shifting from “personally writing every line” to “determining which errors must never enter the system.”

Our View: Autonomous Software Engineering Is Not Here Yet, but the Boundary of Delegation Is Expanding Rapidly

MirrorCode can easily be interpreted as a competition over “when AI will replace programmers,” but that reading is too simplistic.

What it actually shows is that coding agents have moved beyond localized productivity tools and entered the stage of engineering delegation. Small CLIs, data transformation tools, internal administration pages, prototype services, and well-bounded standalone components are increasingly suitable for end-to-end delivery by agents. Humans provide the contracts, environments, and acceptance criteria without having to supervise every line.

However, core systems involving payments, permissions, complex distributed state, long-term compatibility, and maintenance by multiple people are still not suitable for a fully autonomous “submit requirements and wait for delivery” model. The problem is not merely that models make mistakes, but that they still lack the ability to reliably preserve design intent, recognize implicit organizational constraints, and take responsibility for long-term consequences.

In other words, the largest project that AI can complete is growing; the largest project that enterprises can confidently delegate is not growing as quickly.

MirrorCode’s significance is that it encourages the industry to discuss AI coding in terms of project success rates rather than the spectacle of impressive demos. The next stage of competition will not occur only at the level of model parameters; it will center on harnesses, tests, logs, budgets, rollbacks, and human-AI handoff mechanisms.

Whoever turns these elements into a reliable engineering system will truly have an “AI software team.” Simply having an agent generate more code in one go does not count.

References and Further Reading

• SWE-bench GitHub Repository: A classic coding benchmark based on real GitHub issues, useful for understanding the difference between localized repair tasks and complete-project evaluations.
• OpenHands GitHub Repository: An open-source software development agent platform demonstrating how models, terminals, file editing, and execution environments can be combined into a complete harness.
• Aider GitHub Repository: An open-source AI coding tool for real-world codebases. Its evaluations and workflows provide an additional perspective on model routing, code editing, and closed-loop testing.