A 27B Small Model Beats GPT-5.5? Faraday Moves the Battleground to Scientific Reproduction

On August 23, London-based AI lab Inherent announced the launch of Faraday, a scientific research agent. The company claims that Faraday outperformed OpenAI’s GPT-5.5 and Anthropic’s Claude Opus 4.8 on a task requiring it to “autonomously reproduce research results from papers.”

More notably, Faraday’s core model is not one of these frontier models with far more parameters and compute behind them, but Qwen 3.6, with just 27 billion parameters. In Inherent’s design, Faraday is more like a “PhD advisor” responsible for formulating research plans and determining experimental directions than a programmer who personally handles all the coding.

This is not an ordinary update to a model leaderboard. The real question it seeks to answer is this: In long-horizon tasks such as scientific research, software engineering, and data analysis, what determines an agent’s performance ceiling—the size of the model, or its ability to decompose tasks correctly, use the right tools, and adjust course promptly after failures?

Faraday Does Not Test “Can It Answer?” but “Can It Reproduce the Results?”

Traditional large-model evaluations typically place the question, context, and answer within the same window, and the model’s job is to generate a result that is as accurate as possible. Scientific reproduction does not work that way.

Faraday is given published scientific papers, without the key results being directly provided. The agent must understand the research hypothesis from the paper, read the methodology and experimental setup, build the coding environment, prepare the data, run the experiments, and then compare the final results with the paper’s conclusions.

In other words, it is not given the instruction, “Please calculate the answer,” but rather, “Here is a paper. Recreate the experiment yourself.”

Tasks of this kind present several inherent challenges:

• Papers do not tell you about every failed path. Papers generally present only the final experimental approach that worked. Data cleaning, dependency versions, hyperparameter trials, and intermediate failures are rarely documented in full.
• Code is only the execution layer, not the entire job. The same research objective may have multiple possible implementations, and the agent must first determine which paths are most likely to succeed.
• Results are not simply right or wrong. Some reproduced results may differ numerically from the original paper while still matching its overall trends, conclusions, and statistical significance.
• The task cycle is long. A single experiment may require multiple rounds of searching, programming, execution, and correction. A mistaken judgment at any stage can continuously amplify downstream costs.

Therefore, although Faraday is competing against GPT-5.5 and Claude Opus 4.8, the comparison is not about one-shot answering ability, but about completion rates for long-horizon scientific workflows.

The Small Model Does Not Write All the Code—It “Directs”

According to currently available information, Faraday’s core idea is to separate “scientific judgment” from “engineering execution.”

Faraday uses the 27-billion-parameter Qwen 3.6 as its primary decision-making model. It does not need to generate every line of code itself. Instead, it is responsible for understanding the paper, proposing experimental hypotheses, selecting the next action, checking current results, and deciding whether to switch to another approach when an attempt fails.

The actual coding and execution can be delegated to more powerful external models and tools. Public discussions indicate that Faraday can invoke coding agents such as GPT-5.5 Codex to write and modify code. In other words, it is not forcing a 27B model to shoulder the entire scientific workflow on its own. Instead, it assigns the small model the role of “research lead” and treats large models as an engineering team it can call upon.

This architecture closely resembles the agent systems many developers are already experimenting with:

1. Planner: Understands the objective, decomposes the task, and proposes an experimental plan.
2. Executor: Invokes coding models, terminals, browsers, data-processing tools, and training tools.
3. Observer: Reads runtime logs, metrics, error messages, and intermediate results.
4. Evaluator: Determines whether the current results are sufficiently close to the target and whether another iteration is worthwhile.
5. Memory and backtracking module: Stores previously attempted approaches to avoid repeating mistakes during long workflows.

The difference is that ordinary agents are often more concerned with “getting the task done,” while Faraday goes a step further by attempting to train the model to determine “whether this experiment is worth doing.” This is what Inherent calls “research taste.”

“Research Taste” Is Harder to Train Than Accuracy

The hardest problem for scientific research agents is not getting the model to run the code associated with a paper, but enabling it to choose the right experimental direction when resources are limited.

For example, suppose a machine learning paper proposes a new training method. The agent could try dozens of things: switching optimizers, adjusting the learning rate, reprocessing the data, increasing the number of training epochs, modifying the model architecture, or directly investigating potential implementation differences in the paper. Without a way to prioritize these options, it can easily waste vast amounts of compute on low-value experiments.

Human researchers typically make trade-offs based on experience: first testing the variables most likely to affect the conclusion, then gradually narrowing the problem; if a particular direction repeatedly fails, they cut their losses promptly. This kind of judgment is difficult to express as a fixed answer and is also difficult to reward directly through conventional reinforcement learning.

Inherent’s approach is to automatically generate a more detailed set of evaluation criteria for each task—a per-task rubric—and then have an evaluator score the experimental process and final results. For scientific reproduction, the criteria may include:

• Whether the research question in the paper was correctly understood;
• Whether the key algorithm or experimental procedure was implemented;
• Whether reasonable data and parameter settings were used;
• Whether the major trends reported in the paper were reproduced;
• Whether discrepancies between the reproduced results and the original paper can be explained;
• Whether informative next steps were taken after an experiment failed.

The core value of this approach is that it breaks the vague question of “Does this behave like a good researcher?” into multiple behavioral indicators that can be assessed.

Of course, automatically generating a rubric does not mean the problem has been solved. The evaluator itself may make mistakes or even be exploited by the agent: if the model learns to cater to the scoring criteria, it may generate an experimental report that appears complete but has little actual scientific value. Novelty, reproducibility, and long-term value in scientific research are also difficult to measure fully with a single automated score.

For now, therefore, Faraday is better understood as an exploration of methods for training and evaluating scientific research agents, rather than an “AI scientist” already capable of independently publishing original scientific findings.

Before Declaring Victory Over Frontier Models, Examine the Boundaries of the Test

Inherent’s announcement that Faraday surpassed GPT-5.5 and Claude Opus 4.8 is certainly noteworthy, but it should not be interpreted simply as “a 27B model comprehensively outperforming frontier large models.”

At least three boundaries need to be clarified.

First, this is a comparison on a specific scientific reproduction task, not a ranking of general capabilities. A model may be better at planning and debugging for paper reproduction, but that does not mean it is also superior across mathematics, knowledge-based question answering, code generation, visual understanding, or open-ended writing.

Second, the comparison is between complete agent systems, not just the underlying models. Faraday’s performance may come from the model itself, but it may also come from its prompt design, toolchain, task decomposition, search strategy, evaluator, and multi-round sampling mechanisms. Directly comparing the Faraday system with a standalone model inherently means the results will be influenced by system design.

Third, the execution model and compute budget affect the results. If Faraday can invoke GPT-5.5 Codex to handle coding, then what is really being tested is a collaborative architecture in which “a small model makes decisions and a large model executes,” rather than Qwen 3.6 completing the entire task by itself.

This does not diminish Faraday’s significance. Quite the opposite: in real-world production environments, developers are never concerned solely with a model’s raw benchmark score under isolated conditions. What matters is whether the overall system can complete the task at an acceptable cost. A smaller, cheaper model that can reliably orchestrate more powerful tools may offer greater commercial value than simply scaling up parameter counts.

What This Means for Developers: Model Invocation Is Becoming a Systems-Design Problem

Faraday sends a clear signal: competition among agents is shifting from “Who has the larger model?” to “Who can organize models, tools, and feedback mechanisms more effectively?”

For developers, this offers at least three lessons.

1. Do Not Make the Most Expensive Model Handle Every Step

Within a complex workflow, task comprehension, state tracking, tool selection, and code execution do not present the same level of difficulty. Small models can handle frequent, low-cost routing and planning, while more powerful models can be reserved for critical reasoning, complex code modifications, or final review.

The benefits go beyond cost savings. Small models are easier to deploy locally or in private environments and are also better suited to handling large numbers of concurrent tasks. Large models, meanwhile, can concentrate their compute on the genuinely difficult stages.

2. Tool Interfaces Matter More Than Prompts

If a model can only output text, it will struggle to complete long workflows such as scientific reproduction. What often determines performance is whether tools are observable, whether rollbacks are supported, whether logs are structured, and whether the model can accurately read execution results.

An agent designed for scientific or engineering work should expose at least the following information to the decision-making model:

• The current environment and dependency versions;
• Commands already executed and a history of modifications;
• Changes in training or experimental metrics;
• Error types and where they occurred;
• Remaining compute, time, and budget;
• The gap between the current results and the target results.

Without this state information, even the strongest model can easily fall into a pattern of “trial and error by intuition.”

3. Solve Verifiable Tasks Before Pursuing General-Purpose Agents

Scientific reproduction, machine learning engineering, data analysis, and software testing all share one characteristic: although their processes are complex, their outcomes can be partially verified through code, metrics, or test results.

These tasks are better suited to agent training because the system can receive relatively clear feedback. By contrast, there is currently no stable, low-cost, automated way to evaluate whether an agent has “proposed a genuinely valuable new theory.” Faraday’s decision to begin with paper reproduction is therefore a more pragmatic path: first train the model to become a reliable experimental executor, then gradually explore original research.

Inherent’s Ambition: From Reproduction to Discovery

Inherent was founded by former Google DeepMind employees and is headquartered in London. After recently raising $50 million in seed funding, the company has gradually emerged from stealth and set its sights on building AI systems capable of discovering new scientific knowledge.

Viewed in light of that goal, paper reproduction is only the starting point. It trains a relatively foundational but indispensable set of capabilities: reading existing knowledge, proposing testable hypotheses, designing experiments, handling failures, analyzing results, and comparing conclusions with prior research.

Human scientific training follows a broadly similar path. PhD students generally do not begin by proposing theories that transform an entire field. They first reproduce the work of others, then look for differences and opportunities for improvement. Inherent co-founder and Chief Scientist Edward Hughes has also emphasized that paper reproduction is a standard part of scientific training.

However, expectations should be tempered: successful reproduction is not the same as scientific discovery. In reproduction tasks, the objective already exists, and evaluation criteria can be extracted from the original paper. Original research, by contrast, requires posing new questions in unexplored territory and proving that the results are reliable, important, and reproducible. Whether Faraday can cross this divide will require validation through more public tests and real-world scientific cases.

Conclusion: The Opportunity for Small Models Is to Become the “Master Orchestrator”

The most valuable aspect of Faraday is not the attention-grabbing headline that “a 27B model beat GPT-5.5,” but the possible form of AI system it demonstrates: instead of trying to do everything itself, a small model handles judgment, delegation, and orchestration, combining larger models, coding tools, and computing resources.

For model vendors, this means parameter count is no longer the only dimension of competition. For agent developers, the real barriers to entry may lie in task definition, tool orchestration, failure recovery, and automated evaluation. For scientific users, the usefulness of AI should not be judged solely by whether it can generate a seemingly plausible piece of code, but by whether it can continue advancing experiments within budget constraints and explain why it is taking each step.

Faraday’s reported results still come primarily from Inherent’s own corporate statements, and publicly available information provides limited details about the size of the test set, task difficulty, success rates, runtime costs, and specific comparison conditions. It is therefore not yet sufficient to prove that small models have comprehensively surpassed GPT-5.5 or Claude Opus 4.8.

But it has at least brought a crucial question to the forefront: in long-horizon tasks, the strongest model may not be the one with the greatest standalone capabilities, but rather the system that is best at organizing other models and tools. This is a direction developers should continue to watch.

References

• ITHome: Inherent Launches Small-Parameter AI Agent Faraday, Surpassing GPT-5.5 and Claude Opus 4.8 in Scientific Reproduction — Introduces the Inherent team’s background, the launch of Faraday, its 27-billion-parameter model, and its performance on scientific reproduction tasks.