HuggingFace in Practice: Fine-Tuning NVIDIA Cosmos 2.5 with LoRA

A Fine-Tuning Guide for Video Models — For Robot Developers

Teams working on embodied intelligence all face the same problem: general-purpose video generation models excel at making movies, but when asked to generate footage of a robotic arm grasping blocks, the physics fall apart. HuggingFace and NVIDIA recently published a blog post directly addressing this issue — they broke down and explained the LoRA/DoRA fine-tuning process for Cosmos Predict 2.5, specifically targeting robotic video generation.

Cosmos 2.5 is part of NVIDIA’s World Foundation Model series, updated in late March. Its main focus is physical consistency, and compared with the previous Cosmos 1.0, it unifies text, image, and video inputs under a single architecture, ranging from 2B to 14B parameters. The HuggingFace tutorial essentially signals to the community that this model is no longer a closed artifact running only on NVIDIA’s NeMo framework — it can already be customized with the familiar diffusers + peft workflow.

Why LoRA and DoRA, Not Full Parameter Fine-Tuning

Let’s start with the numbers. The 14B version of Cosmos Predict 2.5 requires at least 8× H100 GPUs for full fine-tuning, consuming over 70 GB of memory even with ZeRO‑3 enabled. For most robotics labs, that cost is prohibitive.

The tutorial provides two options:

• LoRA: Injects low-rank matrices into attention and some MLP layers, reducing trainable parameters to about 0.5% of the base model. A single A100 80G can fine-tune the 2B model.
• DoRA (Weight-Decomposed Low-Rank Adaptation): Decomposes weights into direction and magnitude — direction updated by LoRA, magnitude trained independently. It costs about 10% more time but approaches the performance of a full fine-tune.

One detail worth noting: the HuggingFace team explicitly shared that for temporally modeled video diffusion tasks, DoRA performs significantly better than LoRA on small datasets (< 500 videos), showing a 3–5 FVD point gap. But once the dataset reaches thousands of videos, the difference shrinks to under 1 point — that’s where LoRA’s speed advantage kicks in.

In short: if you only have a couple hundred robot demonstration videos, pick DoRA blindly; if your dataset is at RT‑X scale, LoRA will do.

Handling the Data Pipeline

Data preprocessing, not the model, is where video fine-tuning often fails. The tutorial dedicates an entire section to it, with several practical insights.

Resolution and frame rate trade‑offs. Cosmos Predict 2.5 natively supports 720p @ 24 fps, but robot camera data often comes as 480p @ 30 fps. The tutorial advises not to upsample to 720p; instead, adjust the model’s VAE encoder configuration to match the input resolution. Otherwise the tuned model will show color shifts and loss of detail during inference.

Video clip length. Cosmos 2.5 generates 121 frames (≈ 5 seconds) by default, but training with 121 frames at once will exhaust GPU memory. The tutorial recommends training with 49 frames, then expanding to 121 at inference — relying on the model’s temporal generalization to fill in gaps, similar to Open‑Sora’s training strategy.

Captioning pitfalls. Robot video captions tend to be overly abstract, e.g., “robot grasps object.” The tutorial stresses writing action‑level captions: “two‑finger gripper approaches the red cube from the right, closes, then moves 30 degrees up‑left.” Such fine‑grained descriptions teach the model the text‑to‑motion mapping, instead of collapsing all robot videos into one semantic cluster.

Key Hyperparameters: Hidden Traps in the Tutorial

This section is the most practical part — HuggingFace engineers listed all the pitfalls they hit.

Choosing LoRA Rank

A common misconception: higher rank = better performance. Ablation data from the tutorial:

• rank = 8: fast training, but poor motion precision; arm trembles on grasp.
• rank = 16: sweet spot, good for most robot tasks.
• rank = 32: diminishing returns — 40% longer training, only +0.8 FVD.
• rank = 64: overfits small datasets (~200 videos).

Start with rank = 16; increase to 32 for complex scenes (like dual‑arm coordination).

Learning Rate

A counter‑intuitive finding: LoRA fine‑tuning for video diffusion models needs a learning rate an order of magnitude lower than for image models. Recommended:

• LoRA: 1e‑4 → 5e‑5
• DoRA: 5e‑5 → 2e‑5

Video models’ temporal attention layers are highly sensitive to LR — 5e‑4 (common for SD fine‑tuning) will send loss exploding in Cosmos.

Which Layers to Inject

The tutorial clearly advises not injecting LoRA into the VAE or text encoder, only into the DiT backbone’s self‑attention and cross‑attention layers.
If the training data introduces many new concepts (e.g., special‑shaped workpieces), you may additionally fine‑tune the last two text‑encoder layers with a much smaller LR (~1e‑6).

Engineering Details of the Training Workflow

The tutorial uses diffusers 0.32 and peft 0.13. Core configuration:

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["to_q", "to_k", "to_v", "to_out.0"],
    lora_dropout=0.0,
    bias="none",
    use_dora=True,  # switch to DoRA
)

model = get_peft_model(transformer, lora_config)
model.print_trainable_parameters()
# trainable params: 47,185,920 || all params: 2,047,185,920 || trainable%: 2.30

Notable engineering practices:

• Set gradient accumulation to 4, with batch size = 1 (effective batch = 4). For 49 frame clips, that’s the limit for a single GPU.
• Use bf16, not fp16 — some Cosmos attention layers overflow under fp16.
• Enable gradient checkpointing — saves ≈30% memory.
• Keep EMA (exponential moving average) with decay = 0.9999; use EMA weights for inference.

Training duration: on a 2B model with 500 videos, single A100, rank = 16 LoRA, 3 000 steps ≈ 18 hours, enough to show clear domain adaptation.

Evaluating — Beyond FVD

Traditional video generation relies on FVD (Fréchet Video Distance). The tutorial points out FVD can be misleading in robotics — robot arm motions are relatively repetitive, inflating scores.

HuggingFace recommends a combined evaluation:

1. FVD for overall distribution similarity
2. Physical consistency metrics: replay generated trajectories in a physics engine to check rigid‑body constraints
3. Task success rate: use the generated video to drive a VLA model (e.g., OpenVLA) and measure downstream completion

This combined approach is far more reliable, especially when using video generation models as world models for robots.

Notable Broader Context

The timing of this tutorial is interesting. Last month, Physical Intelligence released π 0.5, boosting VLA physical interaction capabilities; earlier this month, Figure showcased Helix deployments in warehouse scenarios. Across embodied‑AI, the scarcest resource is high‑quality robot‑interaction data — and video generation models as synthetic data sources are being rapidly reevaluated.

The Cosmos Predict 2.5 + LoRA/DoRA combination matters because it drops training costs from “rent an 8‑GPU cluster” to “run 2 days on one A100” — the critical threshold for spinning the data flywheel.

Developer suggestions:

• Just experimenting? Use the RoboNet subset provided in the tutorial to avoid manual preprocessing.
• For production, focus investment on caption quality and granularity — it yields bigger gains than hyperparameter tweaks.
• Though DoRA is new, PEFT support is mature — no need to avoid it out of “immaturity” concerns.

Side note: OpenAI Hub is already integrating Cosmos models. Teams wanting quick capability tests without fine‑tuning can call the base API directly. But the tutorial’s true value lies in teaching how to turn general models into domain‑specific tools — that eventually requires local training.

Final Thoughts

Tutorials that lay out engineering details are more valuable than any benchmark score. Over the past two years, HuggingFace has clearly embraced its role as an open‑source toolchain evangelist — from Diffusers to PEFT to TRL, each time NVIDIA or Meta releases a new model, a fine‑tuning guide appears within weeks. That tempo benefits the entire open‑source ecosystem.

By 2026, robotic video generation will likely become a mainstream synthetic‑data method, and efficient fine‑tuning technologies like LoRA/DoRA will be the ticket allowing smaller teams to join. The tutorial’s code and configs are plug‑and‑play — developers should spend a weekend running through it; it’s more useful than reading ten papers.

References

• Fine-Tuning NVIDIA Cosmos Predict 2.5 with LoRA/DoRA for Robot Video Generation – HuggingFace Blog — Original joint Cosmos 2.5 fine‑tuning tutorial from HuggingFace + NVIDIA, with complete code and dataset links