AI Research Evaluation Is No Longer Just About Leaderboards

The evaluation of AI research agents is shifting from “who scores highest on the leaderboard” to “who can deliver results in real research workflows.”

Recently, a research-agent evaluation study led by a Chinese team attracted attention. The study not only ranked highly on traditional capability and practical-task leaderboards, but, more importantly, placed real research tasks, expert acceptance criteria, and final deliverables within a single evaluation framework. The signal is clear: competition among research agents is no longer merely a competition between model capabilities. It is a systemic competition involving task definition, tool use, experiment execution, and result acceptance.

This is also one of the most noteworthy changes in AI research recently. In the past, whether a model “understood science” was often determined by several sets of question-answering tasks, multiple-choice questions, or paper-generation assignments. Now, evaluations are beginning to ask a more practical question: Can it start from a paper, propose a testable hypothesis, write executable experimental code, analyze experimental results, and deliver a research report that an expert would be willing to sign off on?

If the answer still comes only from a single leaderboard, it is clearly insufficient.

From “Answering Questions Correctly” to “Completing Research Tasks”

Over the past few years, large-model evaluation has broadly gone through four stages.

The first stage was model-capability evaluation. Benchmarks such as MMLU, GLUE, and SuperGLUE used static questions to measure knowledge, language understanding, and reasoning. The core assumption at this stage was simple: if a model answered more questions correctly, it was stronger.

The second stage was interactive-agent evaluation. Benchmarks such as SWE-bench, WebArena, and AgentBench no longer required models merely to output a single answer. Instead, they placed them in code repositories, browsers, or simulated environments to complete multistep operations. Models had to plan, call tools, handle errors, and maintain task state along the way.

The third stage was domain-simulation evaluation. Projects such as TheAgentCompany, SWE-Lancer, and τ-bench attempted to simulate real work, asking agents to handle complex tasks in customer service, software engineering, office collaboration, and other domains. Compared with static questions, these evaluations were closer to actual business operations, but the tasks were still designed by researchers, leaving a “translation layer” between them and real customer needs.

The fourth stage is now emerging: production-grade evaluation. It extracts tasks directly from real workflows, evaluates results according to customer acceptance criteria, and further estimates how much labor was saved, how much time was reduced, or how much practical value was created after the task was completed.

AlphaEval provides a representative example. According to publicly available materials, its evaluation set contains 94 real deliverables from seven companies. The key point is not how large the number 94 is, but that the complete task chain is preserved:

• Query: The customer’s original need in a real-world environment;
• Rubric: The acceptance criteria actually used by the customer;
• Answer: The result generated and delivered by the agent;
• Value: The labor or business value represented by the task in the real workflow.

This is much closer to the real world than “Write a sorting function.” Business personnel typically do not formulate their needs as structured prompts, nor do researchers organize problems into exam questions with clear boundaries and unique answers. They are more likely to say: “Help me figure out why this experiment is unstable,” “Summarize the relevant work from the past three years,” or “Find an experimental design that can test this hypothesis.”

These requests may sound simple, but they involve a long chain of work, including problem decomposition, information retrieval, evidence assessment, experimental-design planning, code implementation, result interpretation, and risk communication.

The Challenge for Research Agents Is Not Generating Text That Looks Like a Paper

Today’s large models are already quite strong at answering questions about scientific knowledge, but that does not mean they have acquired stable research capabilities.

The scientific-intelligence evaluation system “Sikuinan,” launched by the Shanghai Artificial Intelligence Laboratory, breaks the research process down into literature retrieval, hypothesis generation, experiment execution, and result analysis, and examines model performance through a “general capabilities–professional capabilities” coordinate system. Relevant results show that the growth of mainstream models’ scientific capabilities lags significantly behind their general capabilities: knowledge comprehension is near the passing line, while hypothesis generation and code generation remain weak points. Multimodal models can often “see,” but may not be able to organize charts, experimental phenomena, and theoretical logic into a testable chain of reasoning.

This is precisely where research agents diverge from ordinary chatbots.

A model may accurately answer “What is protein folding?” but not necessarily propose a novel, workable experimental hypothesis based on a target protein’s structure and existing literature. It may generate a seemingly reasonable piece of Python code, but not necessarily handle outliers, unit errors, and experimental batch effects in real data. After an experiment fails, it may continue explaining the results through an elegant narrative instead of acknowledging that the original hypothesis needs to be rejected.

The evaluation of research tasks should cover at least four categories of capability:

1. Information acquisition: Can the agent find relevant literature, identify the scope of a paper’s applicability, and distinguish primary evidence, secondary interpretations, and unverified conclusions?
2. Scientific reasoning: Can it propose hypotheses with novelty, falsifiability, and a theoretical basis, rather than simply concatenating existing concepts?
3. Experimental execution: Can it generate executable code, configure an experimental environment, access data or tools, and record intermediate steps?
4. Result analysis: Can it explain results, uncertainty, and reasons for failure, while avoiding presenting correlation as causation?

If only the first category is measured, models are easily overestimated. They look more like interfaces to knowledge bases than collaborators capable of independently advancing research.

“First on Both Leaderboards” Is Not the End Goal; A Unified Benchmark Is

The Chinese team’s leading performance on both leaderboards is certainly significant in terms of visibility. But what really matters is that the evaluation is beginning to address three long-standing problems with leaderboards.

First, Task Sources Must Be Closer to Real Research

Traditional benchmarks are generally designed by researchers. Their task boundaries are clear, instructions are complete, and acceptance criteria are explicit. This is useful for controlling variables, but it cannot reflect the ambiguity of research.

Real research often begins with an incomplete question. A researcher may have only an observation, a set of anomalous data, or a conjecture that has not yet been sufficiently validated. An agent must not only answer the question, but also determine whether the question is worth investigating, what evidence should be collected next, and which paths are not worth further investment.

Therefore, a new evaluation system needs to preserve the uncertainty of real tasks while anonymizing, structuring, and versioning them to prevent customer data from being directly exposed to models.

Second, Scoring Cannot Rely Solely on Automated Metrics

Whether code passes tests and whether an answer matches a reference text are still useful metrics, but they are suitable for evaluating only some tasks. Research reports, experimental designs, and theoretical explanations often have no single correct answer, and quality is not equivalent to textual similarity.

A more reasonable approach is multidimensional evaluation. For example:

• Factual accuracy and citation traceability;
• The novelty and falsifiability of hypotheses;
• The feasibility of experimental designs;
• Code correctness, reproducibility, and resource efficiency;
• Whether result interpretation conforms to statistical and domain knowledge;
• Whether uncertainty, failure paths, and potential risks are clearly identified.

Some of these aspects can be checked automatically, while others still require domain experts. Model-based evaluation can reduce costs, but it cannot inherently replace experts—particularly when “another language model says it looks good” becomes the sole acceptance standard.

Third, Leaderboards Must Support Continuous Updating

Public benchmarks face an unavoidable problem: they can be “consumed” by model training data. After questions are published, they may enter pretraining corpora, prompt libraries, and developer debugging workflows. As models become increasingly familiar with the questions, scores gradually shift from measuring generalization to measuring memorization.

Even if the data have not been contaminated, benchmarks can become invalid because their tasks grow outdated. Scientific tools change, the number of papers changes, experimental workflows change, and the ways models use tools also change. A static test set cannot represent frontier capabilities forever.

Future research-agent benchmarks will likely need to be maintained continuously, much like software testing and security evaluation: regularly introducing new tasks, retaining hidden test sets, tracking task leakage, recording model versions, and reporting confidence intervals and failure cases instead of displaying only a total score rounded to two decimal places.

What Exactly Should a Unified Benchmark Unify?

“Unified standards” can easily be misunderstood as requiring every model to answer the same set of questions and be ranked by the same score. In practice, a more valuable form of unification is to standardize task semantics, input/output interfaces, and execution mechanisms.

A research agent may be a text-only model, a multimodal model, or a composite system capable of running code, accessing databases, and calling search tools. Directly comparing the total score of a model that can only read text with that of an agent capable of operating in an experimental environment is inherently unfair.

A unified benchmark should contain at least the following layers:

| Layer | What Needs to Be Unified | Primary Purpose | |---|---|---| | Task layer | Problem background, objectives, constraints, and available resources | Clarify exactly what the model is being asked to complete | | Interface layer | Formats for text, images, tables, code, and tool calls | Support different models connecting to the same evaluation process | | Execution layer | Sandbox environments, dependency versions, permissions, and runtime limits | Ensure that experiments can be reproduced | | Evaluation layer | Automated metrics, expert rubrics, and failure classifications | Prevent a single score from concealing capability weaknesses | | Value layer | Time, labor, cost, and delivery quality | Determine whether the model is genuinely useful |

This means that future leaderboards may no longer have a single overall ranking. Instead, they may present a set of capability coordinates: retrieval reliability, hypothesis quality, code-execution success rate, tool-use stability, experimental reproducibility rate, expert acceptance rate, and cost per task.

For developers, this change may actually be more practical. When choosing a model, the most important question will shift from “Which one has the highest total score?” to “Which capability range does my task fall into?” For literature reviews, retrieval and citation are the priorities; for materials simulation, code execution and numerical reasoning matter most; for medical research, safety boundaries, evidence levels, and hallucination rates also require special attention.

This Will Change How Model Companies Compete

In the past, model companies could improve their scores by optimizing for public question types, adding tool-use prompts, or conducting targeted tuning for specific leaderboards. Production-grade evaluation will bring the competition back to areas that are more difficult to disguise: Can the system complete long-chain tasks? Can it recover after failure? Can experts accept the results?

This places several higher demands on models.

First is process observability. Research tasks cannot be evaluated solely by their final reports. Retrieval records, tool calls, code versions, experiment logs, and intermediate conclusions must also be preserved. Only then can users determine whether the model arrived at its result through effective reasoning or happened to generate a plausible-looking answer.

Second is explainable failure. Failure is acceptable in research, but disguising failure as success is not. A mature research agent should be able to indicate which conclusions come from the literature and which are its own inferences; at what step an experiment failed; whether the results are sufficient to support the original hypothesis; and what data are needed next.

Third is cost control. If an agent needs to call a model hundreds of times and run code for hours to complete an ordinary literature analysis, it may not be suitable for large-scale use even if its accuracy is high. Production-grade evaluation must account for tokens, tool calls, runtime, and the cost of human review.

Fourth is clear human–AI collaboration boundaries. At present, the more realistic direction is not for AI to independently replace scientists, but for it to take on frequent, time-consuming, and verifiable work: organizing literature, cleaning data, generating experimental scripts, checking code, and identifying anomalies. Experts will still need to make final decisions in areas involving research direction, ethical judgment, and major conclusions.

For Developers, the Evaluation Framework Should Come Before Model Selection

If a research-agent project conducts a leaderboard test only at the very end, it is often already too late. A more reliable approach is to establish an internal evaluation set and acceptance criteria at the outset of system design.

You can begin at three levels:

• Offline replay set: Use historical papers, experiment records, and resolved problems to construct fixed tasks for rapidly comparing changes to prompts, models, and toolchains;
• Expert blind-evaluation set: Have domain experts score hypotheses, experimental designs, and reports without knowing which model produced them;
• Online shadow evaluation: Without affecting real workflows, have a new version of the research agent process some tasks in parallel, comparing completion rates, the number of revisions, and final acceptance results.

At the same time, avoid saving only successful cases. The samples that truly differentiate models are often failures: citing nonexistent papers, misreading charts, producing code that cannot run, ignoring experimental boundary conditions, or mistaking correlation for causation. Classifying these failures and continuously feeding them back into the system generally drives iteration more effectively than focusing on the total score.

From this perspective, a new unified benchmark is not intended to create a more authoritative “ultimate leaderboard,” but to enable different teams to discuss capabilities, reliability, and value in a shared language. It may not eliminate leaderboards, but it will reduce the power of any single leaderboard to monopolize conclusions.

Conclusion: Research Agents Have Not Yet Become “Scientists”

Past model evaluations answered the question “How much does the model know?” Interactive-agent evaluations answered “Can the model operate tools?” Production-grade research evaluation must answer “Can the model complete a real research task under constraints?”

These are not the same question.

Current large models have become nearly “super students” in general knowledge and text generation, but they remain clearly far from being “scientists” capable of independently exploring the unknown and assuming responsibility for research. Introducing real-task data will not immediately solve problems such as hallucinations, insufficient originality, and irreproducible experiments. It will, however, turn these problems from abstract debates into engineering metrics that can be measured, diagnosed, and improved.

That is also where the significance of leading on both leaderboards lies: it does not prove that a particular model has solved AI research. Rather, it shows that the rules for evaluating research agents are changing. In the future, the truly competitive systems may not be champions of a static leaderboard, but systems that can reliably complete tasks in real research workflows, honestly report failures, and deliver results at an acceptable cost.

That is closer to the real battlefield of AI research than earning a few more points.

Sources

1. Large-Model Evaluation: A Paradigm Shift from Static Knowledge to Dynamic Agents: Reviews the evolution of model evaluation from static benchmarks and interactive agents to production-grade evaluation, and discusses benchmark deficiencies, data contamination, and evaluation degradation.
2. Relevant public reports and evaluation materials: Cover the latest developments in Chinese teams incorporating real-world scientific research data into agent evaluation, achieving leading results on multiple leaderboards, and establishing unified evaluation frameworks for research agents. Because the domains of the original reports are not included in the list of permitted domestically accessible sources for this article, their external links are not provided. egl, code, API, AI, etc.