Claude Takes Over Constraint Optimization: LLMs Are Designing Experiments for Humans

Last week, Elliot Smith published a blog post about how he integrated Claude into a constrained optimization workflow. At first glance, it looked like yet another AI coding writeup. But halfway through, you realize something is different — he wasn’t asking Claude to help write code. He was letting Claude decide what experiments to run, which parameters to tune, and which metrics to inspect. Humans only looked at the conclusions in the end.

Two years ago, this would have sounded like marketing hype. But here in mid-2026, from Karpathy personally building autoresearch, to multiple reproductions across Claude Code, Codex, and even AutoResearchClaw, one signal has become unmistakable: LLMs participating in mathematical search and experimental design are no longer demos — they are becoming an engineered methodology.

The Tipping Point: From “Writing Code” to “Doing Research”

Start with Elliot’s example, because it’s probably the cleanest illustration of this shift.

The problem he wanted to solve was classic: a constrained combinatorial optimization problem with a parameter space too large for brute force search. Traditional solvers were either too slow or failed to converge. The old workflow looked like this: build a heuristic algorithm, manually tune parameters for hours, let experiments run overnight, then inspect the results the next day.

What Elliot did instead was formalize the problem description, constraints, and evaluation metrics into a markdown spec, hand it to Claude Code, and let it iterate on its own. Claude would read the spec, write solver code, run tests, inspect outputs, modify strategies, and launch another round. Elliot mostly just watched, occasionally stepping in when the direction was clearly drifting off course.

The interesting part isn’t that “Claude can write a solver” — that capability has existed for a while. The real shift is the transfer of decision-making authority: previously, humans figured out exactly what to do and AI executed it. Now humans provide objectives and constraints, while AI decides the path.

That distinction sounds subtle, but its engineering implications are enormous. It means LLMs are no longer just smarter autocomplete systems. They can now take over the most time-consuming part of research workflows: trial and error.

Karpathy’s autoresearch: Encoding Research Organization into Markdown

The person who pushed this idea furthest is Karpathy. His open-source autoresearch project, released earlier this year, is essentially a research pipeline programmed in markdown.

The GitHub repo is so minimal that it almost feels incomplete:

• program.md: human-written behavioral instructions defining how the agent should experiment, evaluate, and decide whether to keep or roll back changes
• train.py: the model code, which the agent modifies itself
• Everything else is left to Claude Code or Codex to execute autonomously

A line from Karpathy’s project has already been quoted endlessly on Twitter: “Cutting-edge AI research used to be done by biological brains squeezing research between meals and sleep... that era is long gone.” The statement carries Karpathy’s usual flair for drama, but the underlying judgment is serious — once agents can self-improve in environments with clear metrics, much research work can be outsourced to compute.

The key phrase is “clear metrics.” ML training naturally provides them: loss curves, validation accuracy, inference latency — all measurable numbers. The agent modifies train.py, runs training, inspects metrics, keeps improvements, rolls back regressions. Once that feedback loop closes, humans no longer need to supervise continuously.

The Reproduction Wave: Claude, Codex, and “Lobsters” Expanding in Parallel

After autoresearch took off, reproductions and extensions arrived quickly. Several directions are worth watching.

Codex Autoresearch (leo-lilinxiao/codex-autoresearch) focuses on generalization. Karpathy targeted ML training specifically, while the Codex version extends the “autonomous research driven by measurable metrics” logic to all software engineering scenarios — performance optimization, test coverage improvement, bug-fix rates, anything quantifiable.

Claude Autoresearch (uditgoenka/autoresearch) is currently the most complete implementation on the Claude Code platform, packaged as a skill plugin with nine commands. Installation is straightforward:

git clone https://github.com/uditgoenka/autoresearch.git
cp -r autoresearch/claude-plugin/skills/autoresearch .claude/skills/autoresearch
cp -r autoresearch/claude-plugin/commands/autoresearch .claude/commands/autoresearch

The usage pattern looks like this — you provide a goal and a verification method:

Goal: Increase test coverage from 72% to 90%.
Verify: npm test -- --coverage | grep "All files"

Claude handles the rest. It reads the codebase, identifies weak coverage areas, writes tests, runs them, evaluates results, and iterates. You just check how far it got when you wake up.

AutoResearchClaw (aiming-lab/AutoResearchClaw) is the most ambitious of the bunch. It implements a fully automated pipeline with 23 stages and 8 phases, covering everything from “research scope definition” to “experiment execution” to “paper draft generation.” The final artifact is a paper_draft.md file containing the introduction, related work, methodology, experiments, and conclusions. It sounds almost science-fictional, but the project is open source on GitHub and the pipeline definitions are publicly visible.

The core differences between these projects are actually quite clear:

| Project | Agent Platform | Scope | Automation Level | |------|-----------|---------|-----------| | Karpathy autoresearch | Any | ML training | Medium | | Codex Autoresearch | OpenAI Codex | All software engineering | Medium-High | | Claude Autoresearch | Claude Code | General development scenarios | High | | AutoResearchClaw | Multi-platform ACP | Academic research | Extremely High |

Why Constraint Optimization Is Such a Good Entry Point

Back to Elliot’s blog post. I don’t think he picked constrained optimization randomly. This domain is unusually well-suited for LLMs, for three reasons.

First, the problem description can be fully formalized. Constraints, objective functions, feasible regions — everything can be expressed precisely in mathematical language. The LLM doesn’t need to guess your intentions; it understands the specification directly.

Second, the feedback signals are clean. Every solver run outputs a feasible solution and an objective value. Success or failure is obvious. This is fundamentally easier than asking an LLM to generate frontend code and then judge whether “the UI looks good.”

Third, the search space is too large for humans to handle effectively anyway. Traditionally, combinatorial optimization relied on carefully crafted heuristics designed by mathematicians. But designing heuristics is itself a search problem — searching through algorithm space. LLMs’ broad prior knowledge allows them to perform heuristic search within algorithm space much faster than human intuition.

Weng Li’s blog post on Harness engineering earlier this year described a similar idea. Their AFlow framework represents agent workflows as graphs, with nodes as LLM calls and edges as code logic, then optimizes the graph using Monte Carlo Tree Search. It’s still the same underlying concept: using LLMs for structural search.

Real Results: One Number

One example circulating on Instagram is worth noting. In March, a developer handed a training optimization task to an autoresearch tool and reduced training time to 1 hour and 48 minutes within two days. The original baseline duration wasn’t clearly stated, but the compression ratio itself was real.

That’s why this feels like a tipping point. You can debate whether LLMs “truly understand” optimization, but when they can complete in two days what would take a skilled engineer a week, the question of “understanding” becomes mostly philosophical. From an engineering perspective, usefulness is what matters.

Don’t Get Carried Away: The Boundaries of This Approach

After all that praise, it’s worth cooling things down a bit.

First, metric dependency. The entire autoresearch paradigm assumes there are “clear, automatically measurable metrics.” ML training has loss values, software engineering has tests, but many real-world scenarios are messier. Product usability, code maintainability, the long-term impact of architecture decisions — agents cannot reliably evaluate these themselves. Force them into the framework, and you may get solutions that technically optimize the metric while being completely wrong for the business.

Second, cost is still a problem. Letting an agent run overnight burns through a substantial number of tokens. If you use Claude Opus or GPT-5 Pro-class models for the entire loop, even medium-scale experiments can generate significant bills. That’s why many current autoresearch workflows use hierarchical setups — cheaper models handle execution and data organization, while flagship models are only called for decision points. The ARIS project (Auto-claude-code-research-in-sleep) is built around this exact idea: cross-model collaboration, where Claude Code executes while external LLMs provide strategic guidance.

Third, humans still need to hold the steering wheel. Karpathy himself notes in the README that autoresearch isn’t about replacing humans entirely. It shifts humans from executors into guardians of direction. Without oversight, agents can drift deeper and deeper into local optima, eventually producing solutions that are technically perfect but strategically useless.

What This Means for Developers

If you write code — especially work involving repeated parameter tuning and experimentation — it’s probably worth spending a weekend trying these systems out. The barrier to entry is lower than it looks. Install Claude Code or the Codex CLI, clone one of the repositories above, read through the program.md templates, adapt them to your own problem, and start running experiments.

In practice, I’d suggest starting with two categories of tasks:

• Hyperparameter search and ablation studies: the ideal autoresearch scenario, with clean metrics and closed feedback loops that can run for dozens of iterations unattended.
• Code performance optimization: provide a benchmark and let the agent explore optimization directions itself. It’s less tedious than manual tuning, and it will often try approaches you wouldn’t think of.

As for integrating new models, comparing Claude, GPT, and Gemini within the same workflow is becoming increasingly common, since different models often have different intuitions about different problems. Aggregation platforms like OpenAI Hub can simplify API key management and routing, especially for users in China who want direct access without networking complications. Feeding the same spec into three models and comparing the resulting solutions would have been purely theoretical discussion two years ago. Now it’s a practical experiment people actually run.

One Final Thought

The most interesting thing about 2026 isn’t that another new model came out. It’s that our imagination of what AI can do is finally catching up to the models’ actual capabilities. The real value of the autoresearch wave isn’t any single repository — it’s that the idea itself has been fully developed: LLMs aren’t just completing your next line of code; they can take over the entire “trial-error-learning-iteration” loop.

The remaining question is simply this: which loops are suitable to outsource, and which still require human hands on the controls? That’s the question every engineer will need to answer for themselves over the next year.

References

• karpathy/autoresearch (GitHub) — Karpathy’s original project, an early prototype of markdown-programmed research organization
• uditgoenka/autoresearch (GitHub) — The most complete autoresearch plugin implementation on the Claude Code platform
• leo-lilinxiao/codex-autoresearch (GitHub) — A Codex version that generalizes autoresearch across all software engineering scenarios
• aiming-lab/AutoResearchClaw (GitHub) — A fully automated academic research pipeline covering 23 stages
• wanshuiyin/Auto-claude-code-research-in-sleep (GitHub) — An autonomous ML research workflow built around cross-model collaboration
• Karpathy’s AutoResearch Guide: Automating Machine Learning with AI Agents (Zhihu) — A detailed Chinese explanation of the core autoresearch concepts
• Weng Li’s New Blog Post: Achieving AI Self-Improvement Through Harness Engineering (Zhihu) — Discussion of AFlow and agent workflow search techniques