Hugging Face releases MolmoMotion: Enabling language to command 3D actions

A single sentence makes a character move — MolmoMotion turns motion prediction into a VLM task

On June 17, the Hugging Face blog featured the Allen AI (AI2) team’s latest creation: MolmoMotion — a 3D human motion prediction model driven by natural language. Unlike the slew of past text-to-motion models, AI2 didn’t train yet another diffusion model, nor did they build another VQ-VAE+Transformer. Instead, they directly embedded motion prediction into the framework of their own Molmo vision-language model: you give it a history of already-occurring actions, a scene image, and a sentence describing intent, and it outputs the joint sequence for the next few seconds.

This approach is a clear signal in the multimodal landscape over the past six months. Text-to-motion has evolved over the last two years from MDM and MotionGPT to Tencent’s 1B-parameter HY-Motion at the end of last year, increasingly resembling video generation — stacking data, stacking parameters, matching flows. MolmoMotion takes the opposite route, treating motion as an “output modality” of a VLM, focusing on instruction following and scene understanding rather than pure kinematic fidelity. For developers, these are two completely different engineering paradigms.

What problem is it actually solving

Let’s clarify the scenario. Traditional 3D motion forecasting tasks give you the bone sequence of the first few frames and ask you to predict the next N frames — essentially a temporal regression problem. Such models are useful for pedestrian trajectory prediction in autonomous driving and avatar frame interpolation in VR, but they have a long-standing problem: the model doesn’t know why the person is moving this way.

For example: someone walks to a table and stops. In the next second, is he going to sit down, pick up a cup, or turn and leave? If you rely purely on historical joint data, the probability distribution gets flattened, and the model can only output an “averaged most likely” action — resulting in a limp-looking motion. Works along the MoMask and HumanML3D line try to alleviate this with textual conditions, but text is a “global description” and doesn’t solve instantaneous intent injection like “I want him to sit down right now.”

MolmoMotion’s entry point is exactly here. It redefines the task as:

• Input: Past K frames of 3D pose + current scene RGB image + a natural language instruction (e.g., “pick up the red mug and sit down”)
• Output: Future N frames of SMPL parameter sequence

The key lies in the second input — scene image. AI2 has long been working on visual grounding, and Molmo itself is one of the strongest in the open-source world for pointing tasks. MolmoMotion inherits this capability, meaning it can identify the exact location of “red mug” on the table from the image and then plan the trajectory of the hand. Previously, this either required a separate affordance model (see last year’s “Move as You Say”) or manually annotated goal points; now the VLM handles it end-to-end.

Tech stack breakdown

According to the blog, MolmoMotion’s core does a few things:

1. Redesigned motion tokenizer

Not VQ-VAE. AI2 used a scheme similar to residual quantization this time, cutting SMPL body pose (rotation of 22 joints + root translation) into tokens per frame, with multiple codebooks stacked per frame to restore details. This approach has already been validated in MoMask to yield much lower reconstruction error than single-codebook VQ. MolmoMotion directly appends these tokens behind Molmo’s text vocabulary, letting the LLM predict them as regular tokens.

2. Reusing Molmo’s vision branch

The image encoder is Molmo’s native SigLIP, unchanged. The clever part is that Molmo’s pointing training has made it highly sensitive to “which pixel corresponds to which object.” When generating motion tokens, attention naturally falls on the object region mentioned in the instruction — effectively providing implicit goal-conditioning for free.

3. Injecting historical motion via cross-attention

The past few frames of pose are not fed into the token sequence; instead, they are separately encoded and used for cross-attention. This design avoids lengthening the token sequence too much — at 30fps, one second is 30 frames × N codebooks, which would quickly eat up the context window.

# Pseudocode: Rough MolmoMotion inference flow
history_feat = pose_encoder(history_poses)  # [K, D]
image_feat = molmo_vision(scene_image)      # [P, D]
text_ids = tokenizer("pick up the mug and sit")

outputs = molmo_lm.generate(
    input_ids=text_ids,
    image_features=image_feat,
    cross_kv=history_feat,
    max_new_tokens=N_frames * n_codebooks,
)
future_poses = motion_detokenizer(outputs)

This architecture has a hidden advantage: the weights of instruction, scene, and history conditions are learned entirely through attention, unlike diffusion models where you must manually tune the classifier-free guidance scale, adjusting each time a new condition is added.

Data and evaluation

For training data, the blog mentions mixing HumanML3D, Motion-X, BEHAVE, and a batch of action data with scene images collected by AI2. For datasets like HumanML3D with only text-motion pairs, the scene input is a placeholder blank image; for BEHAVE, which has real scene and object interactions, it’s the main course.

The evaluation is particularly interesting. AI2 didn’t just run traditional text-to-motion metrics like FID and Diversity; they introduced Instruction Following Rate — human-annotated measure of whether the future motion achieves the target described by the language instruction. On their custom benchmark, MolmoMotion outperformed MotionGPT and LLM-integrated baselines by 15–20 points. This gap mainly comes from scene understanding: when instructions involve specific objects, text-only conditional models basically guess the object’s location.

For traditional metrics, MolmoMotion’s FID is slightly worse than the newest diffusion methods (e.g., HY-Motion, whose pure generation produces smoother motion), but the authors explicitly call this a trade-off: they want accuracy, not beauty. That’s a sensible choice — in downstream interactive applications, whether the motion completes the task matters far more than how silky it is.

How it compares to HY-Motion, MoMask, etc.

Here’s a direct positioning comparison:

| Model | Main task | Conditional input | Architecture | |-------|-----------|-------------------|--------------| | MoMask | Text → Motion generation | Text | Masked Transformer + VQ | | HY-Motion 1.0 | Text → Motion generation | Text | Flow Matching, 1B | | MotionGPT | Text ↔ Motion | Text | LLM + VQ token | | MolmoMotion | Motion prediction/continuation | Text + Image + History | VLM + RQ token |

Simply put, HY-Motion and MoMask “generate a motion sequence from scratch,” while MolmoMotion “predicts the next step based on the current state.” The former’s downstream is animation or game cutscenes; the latter’s downstream is real-time interaction, robotics, and pedestrian intent modeling in autonomous driving. These are fundamentally different tracks, despite both wearing the “3D motion” hat.

How developers can use it

Model weights and code are hosted on Hugging Face, released under Apache 2.0 like Molmo. Memory requirements are moderate — the 7B fp16 version runs on ~16GB, so A10/4090 are fine. The blog shows examples using transformers for loading, with interfaces almost identical to Molmo, except for an additional history_poses parameter and a motion_decoder post-processing step.

Some directions worth trying:

1. VR/AR avatar motion completion: Feed the last few frames before the user removes a headset + “sit down on the couch” instruction to smoothly transition the avatar into a sitting pose.
2. Game NPC behavior planning: Feed in a screenshot of the game scene and let NPCs plan motions according to natural language tasks.
3. Intention prediction in robot teleoperation: Predict what the operator intends to do next and pre-plan force feedback.
4. Enhanced pedestrian trajectory prediction: Traditional trajectory prediction outputs only root position; MolmoMotion outputs full-body pose, allowing finer intent modeling for autonomous driving.

Caveats: currently supports only SMPL body (no fingers or facial expressions), and object interaction geometry precision is still at the “looks right” level — actual robotic grasping will still require a dedicated grasp model for post-processing.

Some thoughts

This approach of embedding motion into a VLM is, long-term, the right direction. The reason is straightforward: the real bottleneck for motion generation has never been “does the motion look good,” but rather “does the model understand the instruction and the scene.” Diffusion has maxed out the former, but the latter still relies on the world knowledge of language models.

In the short term, MolmoMotion will certainly lose to HY-Motion in visual smoothness, but it opens up a new interface — controlling motion using prompt engineering familiar from VLMs. This is friendlier to application developers: tuning logic turns from mysterious parameter tweaking into simply writing prompts.

If AI2 can further integrate motion tokens with Molmo’s pointing ability (e.g., letting the model point to the target object before generating action), the system’s interpretability and controllability will step up another level — which would likely interest the robotics crowd.

References

• MolmoMotion official blog – Hugging Face: Technical blog from the AI2 team detailing architecture, training data, and evaluation results.
• Hugging Face Models: MolmoMotion model weights download page.
• MoMask paper – Hugging Face: Related work; masked modeling as a representative text-to-motion approach.
• HY-Motion 1.0 – GitHub: Tencent’s text-to-motion large model, offered as a comparison reference.