AI2 moves the internal evaluation platform to Hugging Face: olmo-eval aims to take over the model development loop

The Hugging Face blog today features the AI2 team’s latest work, olmo-eval, positioned as a "evaluation workbench in the model development loop". This is not just another benchmarking leaderboard, nor a leaderboard submission tool, but the same internal evaluation pipeline AI2 used when training the OLMo series — now cleaned up, with configurations, data, and metric calculations packaged together and released on Hugging Face as open source for everyone to use.

For teams working on foundational model training, this is a tool worth a close look. The reasons will be explained later — but first, let’s clarify what it is.

One-sentence positioning: Evaluation is not the endpoint, it’s the feedback loop for training

In the past two years, the most widely used evaluation framework in the open-source community has been EleutherAI’s lm-evaluation-harness, which has practically become a de facto standard. Anyone who has trained a model a few times knows that harness solves the “I have a checkpoint, give me a score” problem. It’s clean, general, and well maintained by the community, but it doesn’t care where that checkpoint came from, how it differs from the last one, or how to trace tomorrow’s new run.

olmo-eval aims to solve exactly these latter issues. AI2 calls it an evaluation workbench for the model development loop — the keyword here is “loop.”

This positioning sounds abstract, but it becomes concrete on the engineering level:

• A single configuration can continuously evaluate a series of checkpoints during training, automatically skipping parts already computed
• A model’s results across multiple task sets can be aggregated into tables to facilitate horizontal comparison of training strategies
• Intermediate evaluation outputs (model outputs, prompt templates, tokenization details) are all recorded — not just a final score
• Configurations are written in jsonnet, allowing free combination of model sets × task sets × metric sets

In short, AI2 has turned the actual internal toolchain used for OLMo 2 and OLMo 3 into an open-source version. This is critical — when AI2 released OLMo 3 recently, they made the entire training pipeline “fully open,” including checkpoints, data, and dependencies at every stage. To achieve this level of openness, you need an accompanying evaluation tool to trace every checkpoint’s metrics. olmo-eval is that tool.

How does it differ from lm-eval-harness?

This is what developers care most about. Let’s get practical.

Task coverage: harness wins on ecosystem — the community has contributed hundreds of tasks, so you can find almost anything you want to run. olmo-eval still lags in breadth, but its “core evaluation sets within training” are more refined — AI2 has long been fine-tuning for scenarios like PALOMA (perplexity suite), MMLU variants, and math reasoning.

Reproducibility: This is olmo-eval’s main differentiator. It uses tango as a workflow engine that caches inputs and outputs for each step. In other words, if you’ve run an evaluation today and tomorrow you add a new model, the old ones won’t rerun — only the new parts are computed. This feature is almost a necessity for long-term training projects. Harness can do this but requires you to build your own caching layer.

Configuration language: jsonnet. Opinions differ — lovers find it more expressive than YAML, cleaner than Python configs, and particularly handy for model-task matrix cross-products; detractors may feel learning another DSL is a burden.

Coupling with training frameworks: olmo-eval is natively compatible with OLMo training code, able to directly read intermediate training artifacts. If your training stack differs significantly from AI2’s, integration costs may be higher than with harness.

My judgment: In the short term, olmo-eval won’t replace harness as a universal baseline tool, but it will become a standard complement for teams serious about foundational model training. The two tools solve problems at different layers — harness is the “evaluation SDK,” olmo-eval is the “evaluation platform.”

What a typical usage looks like

The repo’s getting-started command is simple:

tango --settings tango.yml run configs/example_config.jsonnet \
  --workspace my-eval-workspace

This reads the config → pulls the model → runs specified tasks → computes metrics → writes all intermediate results into the my-eval-workspace Tango workspace. Next run:

tango --settings tango.yml run configs/eval_table.jsonnet \
  --workspace my-eval-workspace

If you’ve added new models or task sets in the config, only the new parts are computed; old parts reuse the cache. Anyone who has run full-cycle OLMo evaluations will immediately see how much this saves — a single full evaluation can involve dozens of checkpoints × dozens of task sets, and recomputation costs are extremely high.

At the configuration level, jsonnet allows you to organize like this:

{
  models: ['olmo-2-7b-step10000', 'olmo-2-7b-step20000', 'qwen3-7b'],
  task_sets: ['mmlu_core', 'gsm8k', 'paloma_subset'],
  metrics: ['acc', 'perplexity', 'bits_per_byte'],
}

This m × t × k Cartesian product is the most natural way to express training process evaluations. If you’ve written similar sweep scripts yourself, this will feel familiar.

Why release it now

The timing is interesting. AI2 released OLMo 3 in November last year with a “fully open” stance — not just weights, but all pre-training, mid-training, and long-context expansion stage data, code, and checkpoints. OLMo 3.1 Think 32B reached top performance in math, reasoning, and coding among fully open LLMs, using only one-sixth the training tokens of peers.

Openness at this level demands that evaluations be fully open too. If you only release the model without the evaluation tools, the research community can’t reproduce the “true capability curve for each checkpoint.” By officially introducing olmo-eval on Hugging Face’s blog, AI2 has effectively filled the last gap in the “fully open” chain — from training data, to model weights, to evaluation pipeline, all fully traceable and reproducible.

Worth noting: AI2 archived the old OLMo-Eval repo (now called OLMo-Eval-Legacy) in January this year. This means olmo-eval is a reengineered next-generation version, not just a reskin. Judging from configuration organization, workflow engine, and Hugging Face Hub integration, its engineering level is clearly a step up from the legacy version.

Who should use it, who shouldn’t

Straight to the point:

Ideal users:

• Training or fine-tuning their own foundational model and needing systematic evaluation over a series of checkpoints
• Running data ablation experiments where “swap data → train model → evaluate” must be a repeatable pipeline
• Writing papers that require evaluation results to be reproducible for reviewers and readers
• Teams with MLOps roles willing to invest some learning time for long-term efficiency

Not necessary for:

• Just want to run MMLU to see how a fine-tuned model scores — use harness
• Doing prompt engineering or downstream application evaluation — this tool doesn’t focus there
• Temporary, one-off evaluation needs — overkill

A broader note: the evaluation toolchain is layering

In the past, evaluation was seen as a single thing. Now it’s increasingly clear that evaluation is layered:

• Low-level executors: lm-eval-harness, lighteval, focusing on “correctly run a task’s metrics on a given model”
• Training-loop evaluation: olmo-eval, focusing on “producing evaluation results continuously, incrementally, and traceably during training”
• Leaderboards & comparisons: Open LLM Leaderboard, various private evaluation tracks (e.g., Aupeng recently partnered with Hugging Face on a speech-recognition private track), focusing on “horizontal model comparison under uniform conditions”
• Downstream task evaluation: application-level evals focusing on real-world business performance

Each layer has its focus, and tools will become increasingly specialized. olmo-eval occupies the second layer, which previously lacked mature open-source solutions — AI2’s release contributes its internal tooling here.

For developers in China, if you’re doing model training, it’s worth spending at least an afternoon running olmo-eval’s example — not necessarily to replace your existing tools, but to learn how evaluation can be engineered. Half of AI2’s value here is in the code, and half in the methodology it embodies.

If during evaluation you need to pull some closed models for comparison (e.g., GPT-4o, Claude Sonnet 4.5, or Gemini as judge models), you can use an aggregation API like OpenAI Hub — one key to call all mainstream models, avoiding the hassle of configuring multiple SDKs, and no network workarounds needed in China.

References

• olmo-eval: An evaluation workbench for the model development loop (Hugging Face Blog) — Official AI2 blog introducing olmo-eval
• allenai/OLMo-Eval-Legacy (GitHub) — Archived previous evaluation repo, shows basic Tango workflow usage
• allenai/OLMo-2-1124-7B (Hugging Face) — OLMo 2 7B model card, to understand the training stage breakdown of AI2’s model family
• allenai/OLMo-7B (Hugging Face) — First-generation OLMo 7B model, olmo-eval was originally created for its evaluation
• Top 10 Papers of the Week (Hugging Face Blog) — Contains the full openness details of OLMo 3, an important reference for understanding the background of olmo-eval