HF teams up with NVIDIA to open large-scale multimodal fine-tuning

Video diffusion model fine-tuning: finally no need to build distributed training yourself

Yesterday, the official Hugging Face blog published an article that may not look eye-catching but means a lot to multimodal generation teams: NVIDIA’s NeMo Automodel is now deeply integrated with 🤗 Diffusers. The large-scale fine-tuning capabilities for video and image diffusion models are now “out-of-the-box” open to the open-source community.

Simply put, before this, if you wanted to fine-tune a video diffusion model on the scale of Wan, CogVideoX, or HunyuanVideo, you had to build FSDP2 yourself, write context parallel logic, tune activation checkpointing — even senior engineers at big companies would find it exhausting. Now NeMo Automodel takes care of that for you, still using the familiar Diffusers interface, DiffusionPipeline.

From the perspective of mid-2026, the timing is excellent — OpenAI’s Sora 2 has been commercialized for half a year, and Runway, Keling, and Jimeng are competing in video generation. But the open-source side has long lacked a decent training framework that small to mid-size teams can use for domain-specific fine-tuning. This release is basically NVIDIA packaging its engineering stack and delivering it to the community’s doorstep.

More than “just another training library”

NeMo Automodel first appeared publicly last November during the NeMo framework launch day. Its initial positioning was day-0 fine-tuning support for LLMs and VLMs—you could pull a freshly released model from the HF Hub, change a single import line, and immediately run distributed training. The weights still export in standard safetensors format, directly usable for inference in vLLM.

From its GitHub changelog, the model coverage expansion has been rapid: in February, it added Qwen3.5 MoE 397B-A17B; in March, smaller Qwen3.5 models; in May, Baidu’s ERNIE 4.5 and Xiaomi’s MiMo-V2-Flash; in June, it even included Zhipu’s freshly released GLM-5.2 with a 32K context CP recipe. This “train-new-models-on-release” cadence is rare in open-source training frameworks — most community libraries lag behind months in supporting MoE and long-context models.

This update expands that capability from language/VLM models to diffusion models. In engineering terms, training a video diffusion model is quite different from training an LLM:

• Massive activations: For a 5-second 720p video clip, intermediate latent activations can occupy more VRAM than the parameters themselves.
• Highly irregular sequence lengths: An image is a 1024×1024 patch grid; videos multiply that by frame count, often reaching hundreds of thousands of tokens.
• Fragmented checkpoint formats: In Diffusers, UNet, DiT, VAE, and text encoders load separately, making tensor parallelism easy to break.

Automodel’s solution is to embed FSDP2, Tensor Parallelism (TP), and Context Parallelism (CP) directly into the Diffusers pipeline abstraction. Users only declare the parallelism strategy in their training scripts, and the framework handles model partitioning, communication, and gradient synchronization.

Performance data: not stunning, but very practical

For context, let’s recall some public data from the LLM side. On multi-GPU single-node setups fine-tuning a 30B-scale MoE model, NeMo Automodel achieved 3.4–3.7× throughput over native Transformers v5, while reducing peak memory usage by 29–32%. More importantly, it made full-parameter fine-tuning of a 550B model possible with 16 nodes and 128 GPUs — previously nearly impossible in native HF setups.

For diffusion models, the official blog uses Wan2.2 and CogVideoX as LoRA/full-finetuning examples. Although it doesn’t state explicit speedup multipliers vs. diffusers-native training, the parallelism config gives clues:

# Example configuration (not full code)
parallelism:
  fsdp2:
    sharding_strategy: hybrid_shard
  tensor_parallel_size: 2
  context_parallel_size: 4
  activation_checkpointing: selective

Context parallelism is key here. The attention layers in video diffusion models are bottlenecks for both VRAM and compute. CP splits the sequence dimension across GPUs, theoretically pushing long-video fine-tuning from “needs 8×H100” down to “works on 4×A100”.

Practically, it also preserves the native checkpoint format. This means that after training on a 128-GPU cluster, you can feed the resulting safetensors directly into DiffusionPipeline.from_pretrained for inference — no extra conversion scripts needed. For teams deploying pipelines, this detail is crucial.

Ecosystem position: filling a middling gap

As of mid-2026, the open-source landscape for training diffusion models is fragmented. DeepSpeed excels at ZeRO and offloading but doesn’t understand diffusion model pipelines well. xDiT performs nicely on inference parallelism but has limited training support. Diffusers’ own train_text_to_image.py is just a beginner-level script — anything larger needs a rewrite.

NeMo Automodel’s approach is clever: instead of replacing Diffusers, it embeds itself as a “backend accelerator.” The API layer remains community-familiar; users don’t have to learn a new DSL. The backend is swapped for a DTensor-native SPMD implementation using TransformerEngine operators and DeepEP communication kernels.

This “compatible on top, accelerated underneath” pattern has been validated many times—like vLLM vs. Transformers and Unsloth vs. PEFT—and is the easiest adoption form for the community.

Who will use it—and who won’t

Some judgment calls. This tool suits two main categories:

1. Companies with clusters doing industry SFT: Film, advertising, game CG—these often need to fine-tune open video/image models on private datasets. Previously limited by training infrastructure, they now have a standardized option.
2. Labs doing base model post-training: Teams working on RLHF, DPO, or reward model training on top of base models will save a lot of boilerplate with Automodel’s recipe system.

Less suitable for:

• Single-GPU users doing small LoRAs: Honestly, diffusers + peft + accelerate is enough. Automodel’s complexity is more burden than benefit for individual developers.
• Teams doing deep pipeline customizations: Automodel’s abstraction over the training loop is heavy. If you need to modify EMA logic or add a custom sampler, hand-writing may still be faster.

There’s also a hidden cost: environment setup. NeMo Automodel achieves best performance using NVIDIA container images (NGC provides a nemo-automodel image). If you’re using a self-built CUDA + PyTorch environment, compiling TransformerEngine and DeepEP can be tricky. A common issue with NVIDIA products, not unique to this update.

A small observation

Since late 2024, Hugging Face and NVIDIA have clearly deepened their collaboration—from early NIM integrations, to Transformers v5’s deep TE integration, and now Automodel bridging to Diffusers. Behind this is an under-discussed signal: the open-source model training stack is moving from “academic scripts” to “industrial frameworks.” In the past, training code was disposable; now, recipes, configs, checkpoint formats, and parallel abstractions are being standardized.

For developers, that’s good news: it lowers the marginal cost of model tuning. For startups, it’s even better: no need to maintain a dedicated distributed training team.

By the way, the inference APIs for open-source video models like Wan, CogVideoX, and HunyuanVideo are already available on OpenAI Hub. If you only need to validate business use cases and aren’t ready to train, you can call them directly. For self fine-tuning, following the NeMo Automodel approach described here is currently the smoothest path.

References

• NVIDIA-NeMo/Automodel · GitHub — Official repository with recipes and model coverage list, updated frequently
• NeMo Automodel · Hugging Face Docs — Official Transformers integration documentation describing DTensor-native SPMD parallelism strategies
• Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers — Official blog post for this update, with example configurations for Wan and CogVideoX
• Instantly run Hugging Face models on launch day with support from NVIDIA NeMo · Zhihu — Chinese background interpretation of NeMo Automodel’s project origins, helpful for understanding its design goals