HF Teams Up with NVIDIA: No More Building Your Own Scaffolding for Multimodal Fine-Tuning

Diffusers Can Finally "Run Big"

On July 17, 2026, Hugging Face and NVIDIA jointly released something new — officially integrating NeMo Automodel into 🤗 Diffusers, providing an out-of-the-box scalable fine-tuning solution for video and image generation models.

At first glance, this might look like just another toolchain integration, but for multimodal teams, its significance runs much deeper. Over the past two years, image/video generation models have quietly grown from a few billion parameters to tens of billions. When open-source video models such as Wan, CogVideoX, HunyuanVideo, and Mochi were released, everyone’s first reaction was: inference works, LoRA runs fine — but full fine-tuning? You’d better get that 8×H100 distributed training script running first. Diffusers was designed for research-friendliness and single-GPU usability; once you move up to multi-node, multi-GPU training, users must stitch together Accelerate, DeepSpeed, and FSDP, or even modify Megatron code. Few teams have really mastered this path.

NeMo Automodel’s purpose is to flatten that road.

What Problems Does This Collaboration Actually Solve

Let’s clarify the positioning first. NeMo Automodel is a PyTorch DTensor-native training library that NVIDIA gradually rolled out between late 2025 and early 2026. It was initially designed to provide "Day-0" acceleration support for Hugging Face Transformers — when a new model is released, you don’t have to wait for the Megatron-Core team to finish weight conversion; you can directly from nemo_automodel import ... and start distributed training. It already worked with LLMs and VLMs, and now it has officially expanded to the Diffusers ecosystem.

In other words, this update has one key phrase: scalable fine-tuning for diffusion models is now officially supported.

Previously, if you wanted to fine-tune a 14B video generation model, you’d typically:

• Modify Diffusers’ training scripts to run LoRA or DreamBooth
• Or manually wrestle with Accelerate + DeepSpeed ZeRO-3
• Want FSDP2? Write your own wrap policy and handle sharding for VAE, Text Encoder, and DiT
• When the sequence grows longer (video resolutions blow up fast), VRAM explodes

Now, you just take a ready-made NeMo Automodel recipe, specify the model ID, pick a parallel strategy, and launch with torchrun.

The Technical Highlights

Developers care most about underlying implementation details, so here are a few key ones.

1. FSDP2 Is Default, Not Optional

NeMo Automodel is built directly on top of PyTorch DTensor, and FSDP2 is its native sharding method — not a compatibility layer like many training frameworks that still rely on FSDP1. This matters a lot for video models — FSDP2’s per-parameter sharding saves more VRAM than FSDP1’s flat parameter groups, and it’s easier to combine with tensor parallelism and context parallelism.

For video generation models, the memory bottleneck isn’t just the weights but the activations. A 5-second 720p video easily exceeds 100k tokens, and the attention matrix grows quadratically. Context Parallelism is almost a must-have — splitting the sequence dimension so multiple GPUs share the attention computation. The Diffusers ecosystem previously lacked a clean way to do this; every team had to improvise.

2. Four Parallel Strategies You Can Mix Freely

NeMo Automodel supports flexible combinations of FSDP2, Tensor Parallel, Pipeline Parallel, Expert Parallel, and Context Parallel. For diffusion models, the most common combinations are:

• FSDP2 + Context Parallel: The sweet spot for mid-size (1B–14B) video models
• FSDP2 + TP + CP: For large DiT models like HunyuanVideo (13B), add TP to reduce per-GPU weight load
• Add EP: For those building MoE diffusion models (a direction gaining traction this year)

Worth noting: NeMo Automodel’s results on LLM MoE fine-tuning are impressive — for 30B MoE fine-tuning on multi-GPU setups, throughput improved 3.4–3.7× over native Transformers v5, with peak VRAM usage down 29–32%. It stably supports full-parameter fine-tuning of models with up to 550B parameters across 128 GPUs. These capabilities now extend to diffusion models.

3. DeepEP and TransformerEngine Integration

At the operator level, NeMo Automodel integrates DeepSeek’s open-source DeepEP (for MoE expert routing communication optimization) and NVIDIA’s TransformerEngine (providing FP8, fused attention, fused layernorm, and other kernels). For diffusion models, this means:

• FP8 training works out of the box on Hopper/Blackwell GPUs — no code modification needed. DiT structures actually tolerate FP8 better than LLMs, since many layers use cross-attention + MLP with controllable dynamic range.
• Fused Attention eliminates most of the compute overhead from the heaviest part of video models.

4. Consistent API with Diffusers

This is the second major point: users don’t need to learn a new API. As with Automodel integration on the Transformers side, you load pipelines exactly as before; only the training loop layer switches to NeMo Automodel’s recipe. The output weights remain in standard Diffusers format, so after training you can directly use vLLM, SGLang, or push it back to the Hub.

Migration cost is almost zero, which is even more valuable than raw performance numbers.

How It Stacks Up Against Existing Solutions

Let’s compare with current mainstream approaches:

| Solution | Ease of Use | Scale Support | Video Model Friendliness | | --- | --- | --- | --- | | Diffusers + Accelerate | Low | ≤ Single-node 8 GPUs | Average, relies on LoRA | | DeepSpeed ZeRO-3 | Medium | Multi-node, but clear VRAM limits | Moderate, struggles with long sequences | | Custom Megatron modifications | Very high | Large | Good, but only feasible with big teams | | NeMo Automodel + Diffusers | Low | Hundreds to thousands of GPUs | Excellent, native CP |

The real value isn’t speed on one metric, but turning "full-parameter fine-tuning" from a luxury only large enterprise research groups could afford into something mid-sized teams can run over a weekend. The second phase of open-source video generation will be about who can most efficiently adapt foundation models into domain-specific ones — ad generation, e-commerce videos, and film previsualization are all waiting for this.

A Sober Assessment

Let’s start with the positives. The most commendable aspect of this collaboration is "not reinventing the wheel." NVIDIA could easily have built its own replacement for Diffusers, like it did with NeMo Megatron on the LLM side. Instead, it chose to integrate upstream — Automodel is explicitly an acceleration layer within the Hugging Face ecosystem, not a competitor. That’s good news for the open-source community — no need to pick sides between disjointed tech stacks.

Now, the caveats.

First, hardware binding. NeMo Automodel deeply depends on TransformerEngine and DeepEP, both of which either can’t run or perform poorly on non-NVIDIA GPUs. For teams using AMD or domestic accelerators, this update won’t change much.

Second, the "Day-0 support" promise sounds great, but in the past year, some Transformer models still required extra lead time for optimization. Video model architectures are far more diverse (DiT, UNet, MM-DiT, Latte, etc.), so how well "Day-0" coverage actually works remains to be seen in upcoming releases.

Third, fine-tuning isn’t the same as training. While NeMo Automodel officially supports both pretraining and fine-tuning, its real core use case is fine-tuning. If you want to train a SOTA video model from scratch, Automodel alone isn’t enough — you’ll still need data pipelines, evaluation systems, and post-training recipes (like DPO or reward models). Whether NVIDIA will deliver these next is worth watching.

What Developers Can Do Now

If you’re in any of the following situations, this update brings immediate benefits:

• Fine-tuning open-source video models like Wan, CogVideoX, HunyuanVideo — no more makeshift scripts
• Full-parameter image model fine-tuning (SDXL, SD3, Flux series) — higher ceilings than LoRA
• Utilizing idle multi-node GPU clusters — previously too large for large models, now fully usable
• Research groups working on MoE diffusion models — the EP + FSDP2 combo is unmatched

Getting started is simple: Hugging Face’s blog provides a complete recipe, and you can run training with torchrun. The docs include single-node and multi-node examples, covering SFT, LoRA, and Full FT modes.

Finally, if you want to quickly evaluate results in an application setting after training, OpenAI Hub supports all major multimodal inference APIs (GPT, Claude, Gemini, DeepSeek, etc.), allowing standardized cross-model evaluation with a single key — no need for separate accounts per provider.

This update has slashed open-source multimodal engineering barriers significantly. Now we’ll see what the community can build upon it.

References

• Fine-tune video and image models at scale with NVIDIA NeMo Automodel and Diffusers — Hugging Face official blog, first-hand launch post with recipes and benchmark data
• NeMo Automodel · Hugging Face Docs — Integration guide in the Transformers docs, covering FSDP2, TP, PP, EP, and CP strategies
• Instantly Run Hugging Face Models with NVIDIA NeMo Framework’s Day-0 Support — A Zhihu article explaining Automodel integration on the LLM side, useful background reading for this collaboration