Hugging Face Teams Up with NVIDIA to Supplement Agent Training Data

Where Are Agents Getting Stuck? Not the Model — the Data

Last night, Hugging Face and NVIDIA quietly announced a joint initiative called Data for Agents on the official Hugging Face blog. The name sounds plain enough, but after reading through it, I think this is one of the most important developments developers should seriously study in the past six months. Instead of releasing yet another new model, the two companies are trying to solve a more fundamental problem: what data should actually be used to train Agents?

Over the past two years, the industry's interest curve around Agents has been pretty obvious: in 2024 everyone competed on reasoning, and starting in the second half of 2025, the second wave of Agents after AutoGPT truly took off. Products like Claude Code, Cursor Agent, Devin, and Manus showed engineering teams real production value. But there’s an open secret everyone understands — almost every high-performing Agent system today relies on post-training to brute-force agentic capabilities into the model, usually in one of two ways: synthetic data + RLHF/RLAIF, or quietly recycling user logs from proprietary products.

The problem with the former is narrow distribution coverage. The problem with the latter is compliance — and smaller teams can’t access that data anyway. The entire open-source ecosystem has effectively been operating under a famine-level shortage of Agent training data.

What Exactly Is Data for Agents?

In simple terms, Data for Agents is an open-source, large-scale Agent trajectory dataset and tooling stack designed around real-world tasks. Hugging Face provides the platform and community, while NVIDIA contributes compute infrastructure and synthetic data pipelines (an extension of the Nemotron stack). Together, they aim to do three things:

1. Release a large set of high-quality Agent trajectories, covering coding, browser operation, desktop GUI interaction, multi-tool collaboration, and more. According to the blog, the scale is “millions of trajectories,” with each sample including the full observation-reasoning-action-result chain.
2. Provide a complete synthetic data generation and filtering pipeline, enabling the community to generate additional trainable data for their own use cases instead of relying solely on the seed dataset.
3. Establish evaluation baselines, fully decoupling training data from evaluation sets to avoid the old SWE-bench-style problem where test data accidentally leaks into training.

For the open-source community, the most immediate value of the first point is obvious: finally, models like Llama, Qwen, DeepSeek, and GLM have proper agentic post-training data available for SFT/RLHF. Before this, your options were either early conversational datasets like ShareGPT (which are fundamentally mismatched with Agent scenarios), or building your own simulation environments for data collection — an extremely expensive process.

Why These Two Companies?

The pairing is actually pretty interesting. On the surface, Hugging Face is a community platform and NVIDIA is a hardware company — together it looks like “one handles distribution, the other supplies electricity.” But if you follow NVIDIA’s moves over the past year, you’ll notice they’ve been laying groundwork around data for a while:

• The Nemotron-4 340B model itself was designed as a synthetic data generator
• After acquiring Run:ai, NVIDIA significantly strengthened its control over training pipelines
• Earlier this year, NVIDIA launched the Cosmos data pipeline specifically for embodied AI trajectory synthesis

In other words, NVIDIA is no longer satisfied with just selling GPUs. They want to move down into the data layer and turn “training an Agent” into a full-stack business spanning data, compute, and models. Hugging Face, meanwhile, lacks large-scale data generation capabilities — they have the Datasets Hub, but not the infrastructure to mass-produce high-quality new data.

So this partnership is essentially: “NVIDIA provides the data factory, Hugging Face provides the distribution channel.” It looks complementary, but it also signals something important: control over Agent training data is shifting from model companies toward infrastructure companies.

What Does the Data Look Like?

The blog included several structural examples of the dataset. Simplified, it looks roughly like this:

{
  "task_id": "web-nav-00123",
  "instruction": "Find papers on MoE routing stability published after June 2025 on arXiv and export the BibTeX",
  "environment": "chromium-browser",
  "trajectory": [
    {
      "step": 1,
      "observation": "<html>...arxiv search page...</html>",
      "thought": "I need to construct search keywords first, then filter by date",
      "action": {"type": "type", "selector": "#search", "text": "MoE routing stability"},
      "result": "success"
    },
    {
      "step": 2,
      "observation": "...",
      "thought": "There are too many results, I should use advanced search to constrain the date",
      "action": {"type": "click", "selector": ".advanced-search"},
      "result": "success"
    }
  ],
  "final_answer": "[BibTeX entries]",
  "reward": 1.0,
  "metadata": {
    "length": 14,
    "tools_used": ["browser"],
    "difficulty": "medium"
  }
}

A few details are especially notable:

• Every step includes a thought, meaning an explicit chain of reasoning. This is critical for SFT training on reasoning processes, and has become a major focus after o1/R1.
• There’s a reward signal, meaning the data can be used directly for RL training, not just SFT.
• The dataset includes failure trajectories (mentioned in the blog though not shown in the sample). This is extremely important — models trained only on successful trajectories become overly optimistic and often give up once errors occur mid-task.

From a dataset design perspective, the people behind this clearly understand Agent training. This isn’t a superficial “run GPT-4o through a bunch of tasks and dump the logs” approach.

Scenario Coverage: No Longer Just Code

Most open-source Agent datasets until now have focused on two areas: code (because GitHub data exists) and web browsing (because simulators like WebArena exist). Data for Agents is clearly expanding beyond that:

• Software engineering: multi-file repository modifications, dependency analysis, test-driven development
• Desktop GUI: operations in real software like Word, Excel, and Photoshop (NVIDIA clearly used substantial synthetic environments here)
• Data analysis: full pipelines from raw tables to conclusions, including SQL, Pandas, and visualization
• Multi-Agent collaboration: scenarios where a “manager Agent” coordinates multiple “worker Agents”
• Long-horizon planning: median task lengths reportedly exceed 30 steps — an order of magnitude longer than existing public datasets

The last point is actually the hardest part. Open-source models currently have a common Agent problem: short tasks (under 5 steps) work reasonably well, but performance starts drifting badly beyond 10 steps. The core reason is simply that training data contains too few long-task examples. The model has barely seen them. This dataset directly prioritizes long-horizon tasks as a target scenario.

What This Means for Open-Source Models

I tend to believe that Data for Agents will significantly raise the lower bound of open-source Agent capabilities within the next 3–6 months. There are three reasons:

First, the data quality itself is strong. NVIDIA’s synthetic data capabilities have already been validated through the Nemotron series, and combined with human filtering (the blog mentions a verifier pipeline), the signal-to-noise ratio should be far better than community-collected datasets.

Second, the tooling stack is open. That means teams behind Qwen, DeepSeek, GLM, Llama, and others can directly integrate this pipeline into their own training workflows without rebuilding everything from scratch. The gap between major companies often comes not from model architecture, but from the maturity of their data pipelines — and once that layer is flattened, catch-up speed increases dramatically.

Third, the separation between training and evaluation is handled cleanly. The open-source community has long criticized benchmarks like SWE-bench for training contamination, which inflates leaderboard scores. Data for Agents explicitly separates training subsets from held-out evaluation sets, which is healthier for the ecosystem in the long run.

That said, there’s no reason to be overly optimistic. Synthetic data has an inherent ceiling — it can cover tasks you can imagine, but not the long tail you can’t. In real production environments, Agent failures often happen in bizarre edge cases no one explicitly designed for. The only way to cover that is through real user logs, and the open-source community will probably continue struggling there.

A Slightly Colder Observation

There’s something the blog never says explicitly, but it’s worth thinking about: neither OpenAI nor Anthropic is involved in this collaboration.

Over the past two years, Hugging Face has maintained a somewhat distant relationship with many proprietary model vendors. Companies may publish weights on HF (if they’re open source), but deep data collaboration has been almost nonexistent. This direct alliance with NVIDIA is effectively an explicit bet on the open-source camp. And NVIDIA, traditionally positioned as the “arms dealer,” is no longer acting neutral either.

This reflects a broader trend: proprietary model companies are becoming increasingly closed around their data moats, forcing the open-source ecosystem to collaborate on infrastructure. Over the next year, we’ll likely see more “infrastructure alliances” emerge — around data, evaluation, safety alignment, and more — all sharing one common trait: bypassing OpenAI/Anthropic standards and building alternative ecosystems.

How Developers Can Use It

For independent developers and small-to-medium teams, the most practical short-term uses include:

1. Fine-tuning domain-specific Agents. For example, if you’re building an Agent focused on financial data analysis, you can select relevant trajectories from the dataset’s analytics subset for SFT. The results will be dramatically better than zero-shot prompting.
2. Building evaluation baselines. Use the held-out evaluation set to benchmark your own Agent system instead of relying on closed internal testing.
3. Learning trajectory design. Even if you’re not training models and only building Agent applications, studying how these trajectories structure thought-action-result flows can improve your prompts and tool schemas.

The dataset itself will be released gradually on Hugging Face Hub, with the first batch reportedly arriving this week. The tooling repositories will also be open-sourced on GitHub simultaneously.

Closing Thoughts: Agent Training Enters the “Data Competition” Era

2023 was the competition for model scale. 2024 was the competition for reasoning ability. Starting in the second half of 2025, the main battlefield for Agent capabilities has quietly shifted — from “whose model foundation is better” to “whose training data better matches real-world tasks.”

Data for Agents is the first genuinely substantial open-source answer to that trend. It may not immediately reshape the landscape, but it marks an important beginning: Agent training data should not remain the private property of a handful of proprietary companies.

By the way, if you’re developing Agent applications and need to compare models like GPT, Claude, Gemini, and DeepSeek on specific tasks, OpenAI Hub already integrates all of these mainstream models. You can switch between them with a single API key, with direct China connectivity and OpenAI-format compatibility, which makes Agent comparison testing quite convenient. Once fine-tuned open-source models based on Data for Agents start rolling out, we’ll integrate them immediately as well.

References

• Official Hugging Face Blog: Data for Agents - Official announcement from Hugging Face and NVIDIA introducing Data for Agents, including dataset design philosophy, scenario coverage, and pipeline details