NVIDIA has released Cosmos 3, the world’s first fully open-source, fully modal physical AI foundational model. Its hybrid Transformer architecture integrates visual reasoning, world generation, and action prediction. NVIDIA has also formed the Cosmos Alliance together with Runway, Black Forest Labs, and others, reducing the robot training cycle from several months to just a few days.

NVIDIA Open-Sources Cosmos 3: Has the “GPT Moment” for Physical AI Arrived?

On June 1, NVIDIA officially released Cosmos 3, claiming it to be the "world’s first fully open-source, full-modality foundation model for physical AI." It’s a massive model that combines three capabilities—visual reasoning, world generation, and motion prediction—into a single system, with one ultimate goal: to enable robots, autonomous vehicles, and visual agents to truly see and act in the real world.

At the launch, Jensen Huang repeated a line he’s said no fewer than ten times over the past six months: “The ChatGPT moment for robotics has arrived.” This time, though, he paired the statement with a concrete embodiment.

Jensen Huang releasing Cosmos 3, with a live demo of robots performing tasks in a virtual environment.

One Model, Three Abilities in One

To understand the ambition of Cosmos 3, you have to first understand the awkward state of physical AI over the past few years.

Anyone working on robotics knows that the entire tech stack is fragmented: perception uses one VLM, world simulation relies on Isaac Sim or a custom engine, and motion control is handled by yet another VLA model. Data formats don’t align, training objectives differ, and benchmarks are isolated — meaning that developing a new robotic skill from simulation to deployment typically takes months at minimum.

Cosmos 3 offers a unified architectural solution: a hybrid Transformer that fuses reasoning and generative Transformers to work collaboratively. The model first parses object interactions, motion patterns, and spatiotemporal relationships—essentially “understanding what’s happening in the world”—and then hands it off to the generative module to predict future video frames and motion trajectories.

Officially, NVIDIA defines three modes of use that developers can invoke as needed:

Multimodal vision-language model (VLM): Cross-modal understanding and reasoning—a “physics-aware” VLM.
World model / video foundation model: Simulates physical environments and forecasts scene dynamics, potentially replacing or enhancing existing simulators.
World-action backbone network: Serves as the backbone for robot skill training, with task-specific heads layered downstream.

This “one model, three uses” design is critical for teams building real robotic products—it means a single representation can run through data generation, policy training, and evaluation, instead of shuttling data among three disconnected systems.

How “Full” Is Full Modality?

Cosmos 3 natively supports the following modalities: text, image, video, environmental audio, and motion trajectory.

Note the last two—“environmental audio” and “motion trajectory”—which mark Cosmos 3’s biggest departure from typical multimodal models. GPT-4o and Gemini also feature multimodality, but they’re built for human interaction—image captioning, sound interpretation. Cosmos 3’s multimodality is built for robots: Can the sound of metal impact help infer material properties? How do you align joint torque time-series with visual flow? These are the real questions of physical AI.

According to NVIDIA, the training corpus includes “tens of billions of text, image, video, audio, and motion samples.” While no precise number was given, the sheer scale makes it likely the largest open physical AI dataset to date.

As for benchmarks, NVIDIA highlighted Artificial Analysis, Physics-IQ, PAI-Bench, and R-Bench. In the open-source category, Cosmos 3 achieved SOTA in world generation accuracy across these tests. Of course, note the qualifier "open-source"—closed-source contenders like Sora 2 and Veo 3 weren’t included in direct comparisons.

The Cosmos Coalition: Bringing Competitors Onboard

Perhaps more intriguing than the model itself is NVIDIA’s launch of the NVIDIA Cosmos Coalition.

Founding members include:

Agile Robots – European humanoid robotics startup
Black Forest Labs – Creators of the FLUX model, a leading image generator
Generalist – General-purpose robotics startup
LTX – Developers of the video generation model LTX-Video
Runway – Leader in commercial video generation
Skild AI – Robotics intelligence startup valued over $4B

The message is clear: NVIDIA has brought together top video-generation players (Runway, BFL, LTX) and cutting-edge robotics teams (Skild, Agile, Generalist). After all, a video generation model is essentially a world model—you can predict the next frame, you can predict the next event in the physical world. NVIDIA’s goal is to merge these two parallel technological tracks into one.

For companies like Runway and BFL, joining the coalition grants access to NVIDIA’s data, compute power, and robotics ecosystem; for NVIDIA, it turns potential world-model rivals into collaborators. A shrewd strategic move.

Is "Fully Open-Source" Really Full?

The phrase "fully open-source" has been overused, but Cosmos 3 genuinely qualifies: model weights, training code, and parts of the dataset are all available on Hugging Face, with commercial usage permitted.

By comparison:

Meta’s Llama series: weights open, partial code open, data closed
Mistral: weights open only
Most video generation models: either API-only or inference-only

In the physical AI domain, Cosmos 3 represents the most thorough open release to date. Considering physical AI’s extreme dependence on data quality and diversity, releasing datasets openly may be even more valuable than releasing weights.

What It Means for Developers

Cosmos 3’s value differs depending on the developer type:

For robotics teams: Use Cosmos 3 as the backbone, skipping the costly and time-consuming pretraining phase to focus on fine-tuning. NVIDIA claims this can shorten training and evaluation cycles from “months” to “days”—a major lifeline for robotics startups.

For autonomous driving teams: The world-model component is directly useful for corner-case data generation and closed-loop simulation testing. It’s no coincidence NVIDIA also pushed open-source simulation models at NeurIPS simultaneously.

For generative video teams: Although Cosmos 3’s video generation module emphasizes physical realism, it’s still a fully functional video foundation model. Members like LTX and Runway are clearly positioned to build derivative products on top of it.

For application-layer developers: Likely not relevant—Cosmos 3 isn’t a chatbot builder. Its intended users are developers building systems that can perceive, reason, plan, and act in the real world.

Things Left Unsaid

Several subtleties in the release are worth noting.

First, the relationship between Cosmos 3 and GR00T. GR00T is NVIDIA’s foundational model for humanoid robots, with version N1.7 newly available for early access. The two lines seem to advance in parallel—Cosmos being “general-world foundational,” GR00T being the “humanoid brain.” Think of Cosmos as the foundation, GR00T as the structure built on top.

Second, Isaac Lab 3.0 early access launched simultaneously, paired with the new Newton physics engine 1.0. Together, they aim to let robots train at much larger scales and speeds on DGX-level infrastructure. In short, Cosmos 3 tackles the modeling problem, and Isaac Lab 3.0 the training infrastructure—they interlock neatly.

Third, cloud integration has begun. Microsoft Azure and Nebius integrated NVIDIA’s physical AI data factory, Blueprint; CoreWeave integrated Isaac Lab; Alibaba Cloud directly embedded NVIDIA’s full physical AI stack into its AI platform. This means Cosmos 3 isn’t an isolated model confined to NVIDIA hardware—a key factor for ecosystem expansion.

Some Perspective

In the physical AI race, the last two years have seen lots of hype but few real-world deployments. Companies like Figure, 1X, and Agility have skyrocketing valuations, yet truly reliable, factory-ready robots remain rare. The bottleneck isn’t hardware—it’s model generalization: robots trained for 100 hours still freeze in new environments or fail to grasp unfamiliar objects.

Cosmos 3’s approach—large-scale pretraining plus full-modality integration—tackles this generalization problem much like LLMs did for NLP. Whether it succeeds depends on real-world deployment data. But one thing is clear: by releasing the model, training framework, simulation engine, physics engine, synthetic data pipeline, cloud integrations, and robotics coalition as one package, NVIDIA has dramatically lowered the engineering barrier for physical AI.

In the coming year, if you’re working on robotics or physical AI applications, Cosmos 3 will almost certainly become a baseline you can’t ignore.

The model is now on Hugging Face and available for direct download. For upper-layer applications requiring unified access to various closed and open models, OpenAI Hub already aggregates mainstream APIs in OpenAI-compatible format, accessible domestically. Combined with open physical foundations like Cosmos 3, this forms a streamlined toolchain for building end-to-end robotic systems.