OpenAI Launches New Benchmark GeneBench-Pro to Tackle AI Biology “Pseudo-Intelligence”

OpenAI unveiled something new on June 30: GeneBench-Pro. It’s a benchmark specifically designed to evaluate AI models’ ability to conduct biological research, featuring 129 questions, each of which would take a human expert 20 to 40 hours to complete. Greg Brockman summed it up directly on X: GPT-5.6 Sol represents a major leap forward on this benchmark.

This move is not happening in isolation. Just days earlier, OpenAI released its first life sciences–focused reasoning model, GPT-Rosalind, in collaboration with pharmaceutical and research organizations including Amgen, Moderna, and the Allen Institute. Viewed together, these two developments make OpenAI’s intentions in the life sciences space very clear: build the models first, then build the ruler, and finally shape the ecosystem around it.

What’s wrong with traditional benchmarks?

To understand the value of GeneBench-Pro, it’s necessary to first explain the problems with existing biology AI benchmarks.

Current mainstream approaches generally fall into two categories. One is knowledge-based question banks that test models with multiple-choice questions, essentially measuring whether the model “memorized the material.” This type of evaluation was already saturated in the GPT-4 era, with frontier models all hovering above 90%, making distinctions between them nearly meaningless.

The other category is more advanced, such as BixBench and LAB-Bench, which give models real bioinformatics tasks involving code execution and database queries. But there’s a hidden trap here: when benchmarks are built from historical real-world data, the same dataset often supports multiple reasonable analysis paths. Even if a model chooses the wrong method, it may still arrive at the correct answer due to coincidences in the data itself. Evaluators see the correct answer and award the point, even though the model’s reasoning process was flawed.

This is the so-called “shortcut” problem. In long-horizon tasks, this kind of bias becomes magnified indefinitely—you can’t tell whether the model genuinely understands the task or simply got lucky.

Synthetic data is both the solution and a shift in design philosophy

OpenAI’s solution is to build all benchmark questions using synthetic data. This choice is crucial.

Synthetic data means OpenAI has complete control over the underlying causal structure and data generation process. They know not only what the answer is, but also what the correct path to the answer looks like. When a model produces a conclusion, evaluators can verify whether the model followed the correct reasoning path, rather than only checking the final result.

To use an analogy, traditional evaluations are like math word problems where only the final answer matters; GeneBench-Pro is more like checking both the answer and the solution process, while also requiring the solver to handle deliberately injected misleading conditions during the calculation. The model must genuinely understand “what this dataset is telling me,” rather than applying a template.

An even tougher aspect is that the questions are intentionally designed to be “ambiguous, incomplete, and noisy.” This closely resembles real scientific research environments—data coming from sequencing instruments is never perfectly clean, experimental records are never fully complete, and researchers must determine for themselves which data is trustworthy, what methods should be used, and whether the results are strong enough to support the next decision.

129 questions covering 10 major domains

The structure of GeneBench-Pro is worth examining in detail.

• Total: 129 questions
• Categories: 10 major domains, 21 subdomains
• Coverage areas: genomics, quantitative biology, translational medicine
• Example subdomains: statistical genetics, population genetics, functional genomics, proteomics

Each question follows a similar structure: a dataset resembling a real research environment, a brief experimental background description, and a goal-oriented question tied to downstream decision-making. The model must independently explore the data, select methods, revise its strategy along the way, and ultimately produce an answer that can support decision-making.

The phrase “related to downstream decision-making” is important. This is not about simply calculating a p-value and stopping there. The model must answer open-ended questions such as, “Based on this data, what experiment should we conduct next?” This is the most fundamental difference between GeneBench-Pro and previous benchmarks—it evaluates not just data analysis capability, but research judgment.

What does 20 to 40 hours of expert work really mean?

Brockman’s statement that “each question takes a human expert 20 to 40 hours” is striking.

For comparison, a problem on SWE-bench might take a human engineer anywhere from tens of minutes to several hours; even the PhD-level questions in GPQA Diamond are generally solvable within tens of minutes. What does 20 to 40 hours represent? It’s essentially equivalent to one to two weeks of work for a PhD student, or the time a junior postdoc would spend running a complete analysis pipeline.

The duration itself sends a signal: OpenAI is no longer satisfied with measuring “intelligence” alone—it is beginning to measure “endurance” and “engineering coordination ability.” Long-duration workflows, heavy tool usage, and multi-step decision-making are where OpenAI is placing its next bets. This also explains why GPT-Rosalind specifically emphasizes tool-calling capabilities and integration with more than 50 scientific databases.

How scoring works

Avoiding evaluation bias is something OpenAI repeatedly emphasizes. Beyond synthetic data as the foundation, the company also implemented several additional measures:

1. Verifiability first: because the correct reasoning path is known, the scoring system can inspect intermediate steps rather than just the final answer
2. Robustness checks: every question undergoes agent testing and expert review to ensure the benchmark itself has no loopholes
3. Third-party evaluation: OpenAI will release 50 of the 129 questions to Artificial Analysis for independent evaluation, creating a neutral public leaderboard across different models

The third point is particularly clever. Artificial Analysis is widely recognized as an independent benchmarking organization. By partially handing over evaluation authority, OpenAI not only avoids criticism about “writing its own test and grading itself highly,” but also effectively positions GeneBench-Pro as a cross-vendor industry standard.

How did GPT-5.6 Sol perform?

Brockman described GPT-5.6 Sol as “a big step forward,” but did not provide specific scores. However, data disclosed on the official GPT-Rosalind page suggests that OpenAI’s models are consistently leading in biological tasks:

• In BixBench (real-world bioinformatics and data analysis), GPT-Rosalind ranked among the top models with publicly available scores
• Across 11 tasks in LAB-Bench 2, GPT-Rosalind outperformed GPT-5.4 in 6 tasks, with the most significant improvement in CloningQA (end-to-end molecular cloning design)
• In RNA sequence function prediction tests conducted with Dyno Therapeutics, the model’s best performance across ten submissions surpassed 95% of human experts

Viewed together, GeneBench-Pro is essentially a new and more demanding ruler designed for this generation of life sciences models. The old rulers can no longer meaningfully measure the differences.

What has already been released

OpenAI has currently open-sourced 10 representative benchmark questions on Hugging Face, complete with an interactive web interface for external researchers to explore. The remaining questions will gradually be opened to Artificial Analysis for third-party evaluation.

For developers, several points are worth paying attention to:

• These 10 public questions can be directly used to test whether the models you use are actually capable in computational biology scenarios
• The datasets themselves are valuable material for researching computational biology agents, even outside the context of benchmarking
• Future Artificial Analysis leaderboards will become an important reference point for comparing cross-vendor model capabilities

OpenAI Hub currently supports the full GPT model lineup, allowing users to switch between models using the same API key to test performance on the public GeneBench-Pro questions. Direct domestic access is available in China without requiring self-hosted proxy infrastructure, reducing deployment overhead for teams building AI applications in biotech and pharmaceuticals.

A broader perspective

From GPT-Rosalind to GeneBench-Pro, OpenAI’s strategy in life sciences is becoming visible: build capabilities first, define standards second, and consolidate the ecosystem last. This approach closely resembles how OpenAI established itself in general-purpose large models—first GPT, then benchmarks such as MMLU and HumanEval, and finally pulling the industry into its evaluation framework.

Life sciences is a unique battlefield. It involves massive datasets, compute-intensive workflows, long decision chains, extremely high barriers to entry, and commercial value measured in tens of billions of dollars. Developing a new drug can take 10 years and cost $2 billion, so any tool capable of shortening that cycle will attract intense interest from pharmaceutical companies. This is why OpenAI has directly brought in giants such as Amgen, Moderna, and Thermo Fisher Scientific as joint customers.

In the short term, GeneBench-Pro may not become as widely discussed among developers as SWE-bench, since not everyone works in biology. But for every company trying to push large models toward the level of “actually conducting research,” this benchmark will become an unavoidable reference point.

Who is genuinely doing research, and who is only pretending to do research—you can find out by running through 129 questions.

References

• OpenAI launches the GeneBench-Pro benchmark for evaluating AI models’ biological computation capabilities - IT Home: Chinese-language report published by IT Home on July 1, covering the core design and question structure of GeneBench-Pro
• GeneBench-Pro public question dataset - Hugging Face: OpenAI’s officially open-sourced set of 10 representative benchmark questions, including an interactive web interface