Xiaomi Open Sources ControlFoley: Shifting Video Dubbing from “Automatic Guessing” to “Intent-Driven”

Today, Xiaomi’s large-model application team dropped an interesting open-source project: ControlFoley, a unified and controllable video sound-effect generation framework. The code, model weights, technical report, online demo, and ready-to-use Skill are all released at once, with one clear goal—to turn video dubbing from “the model guessing” into “creator in control.”

It’s worth setting the context first. Over the past two years, video-to-audio generation (V2A) has become a crowded field. Meta’s AudioGen, Google’s V2A, Alibaba Tongyi’s ThinkSound, and more recently, Keling 2.6’s “visual-audio co-generation”—everyone is working on the same problem: how to automatically generate matching sounds for a silent video. But if you look closely, most approaches stay at “seeing determines sounding”—whatever is in the frame, the model guesses what sound should go with it. When the guess is right, great—but if the creator wants something else—say, “I want this cat to roar like a lion when it opens its mouth,” or “make the running footsteps sound metallic,”—the model has no idea what to do.

That’s exactly where ControlFoley steps in. It’s not here just to answer “is there sound,” but to make sure the sound is what you want.

One model, three tasks

ControlFoley breaks controllable sound generation into three scenarios and covers them all with one unified framework:

• TV2A (Text + Video to Audio): the video is primary, the text is supplemental. The model generates synchronized sound effects from visuals, while the text fills in semantics that the visuals alone don’t express clearly—e.g., if the video shows a bird flying, the text could add “birds chirping + distant wind.”
• TC-V2A (Text-Controlled V2A): this is the most interesting one. When text and video semantics conflict, the model obeys the text. For example, the video shows typing on a keyboard, but the text says “rain sound”—the output will be rain sounds, still in sync with the fingerprint typing motion. That’s the capability most creators want.
• AC-V2A (Audio-Controlled V2A): supply a reference audio clip, and the generated result will match its timbre and style without losing the video’s timing. In simple terms: “sound style transfer + video time alignment.”

Packing these three tasks into one model sounds reasonable but is actually tricky. Visual signals are inherently dominant—during multimodal fusion they can easily “overwhelm” the control intent from text or reference audio. You say “rain sound,” but because it sees a keyboard, it still gives you typing sounds. That’s a long-time issue in V2A, and one that ControlFoley specially addresses in its architecture.

CAV-MAE-ST: Why build a new visual encoder

ControlFoley doesn’t use CLIP but instead trained its own spatiotemporal audiovisual encoder, CAV-MAE-ST. The reason behind this choice is actually key.

CLIP excels at identifying “what it is”—it aligns images and text in a shared semantic space, recognizing whether an image shows a cat or dog, a city or a forest. But what V2A truly needs is “what action happens at what moment.” For example, if a person throws a punch, CLIP can tell you “this is punching” but not precisely whether the contact happens at frame 17 or 19. For sound effect generation, that temporal precision matters a lot—off by one or two frames, and the dubbing feels “off.”

CAV-MAE-ST follows the MAE (Masked Autoencoder) idea and does masked self-supervised pretraining on audiovisual data, emphasizing features in temporal dimensions such as motion rhythm, event boundaries, and audio-visual synchronization. In short, CLIP is the “semantic expert,” while CAV-MAE-ST is the “rhythm expert.” ControlFoley uses both together—CLIP ensures high-level semantic understanding and text alignment, while CAV-MAE-ST ensures proper temporal synchronization.

This “dual-visual-tower” strategy isn’t entirely new, but making it work in V2A—and keeping text control signals from being overridden—is a real engineering achievement.

Time–timbre disentanglement: making control like playing LEGO

Another design worth noting is time and timbre disentangled representation.

Before ControlFoley, most V2A models learned “when to sound” and “what timbre to sound” entangled together. The outcome: if you change the timbre, you mess up timing; if you tweak the timing, timbre changes too. After disentanglement, the timeline is governed by the video, and the timbre by the text or reference audio. This way, all three tasks can be expressed in the same architecture:

• TV2A: time from video + timbre from text semantics
• TC-V2A: time from video + timbre forced by text
• AC-V2A: time from video + timbre from reference audio

The beauty of this approach is that control becomes composable. Creators can adjust timbre without touching timing, or shift time anchors while keeping timbre fixed. In real editing workflows, this fine-grained control is far more valuable than “one-click generation.”

Multimodal robust training: keeping control signals from “going idle”

The third key point is the training strategy. Multimodal models often suffer a classic problem: if some modality makes life too easy, the model “cheats” and relies only on that one. In V2A, vision is the easiest to exploit, which makes the model ignore text or reference audio control.

During training, ControlFoley adopts modality dropout + conflict sample reinforcement strategies:

• Randomly silence one control modality to force the model to use the remaining ones;
• Intentionally create text–video semantic conflicts so the model learns “which modality to listen to”;
• Apply style perturbations to reference audio to prevent the model from simply copying it.

The outcome shows up in evaluations. Official benchmarks indicate that ControlFoley achieves open‑source SOTA on multiple V2A tasks, improving across four dimensions: semantic alignment, temporal synchronization, sound quality, and multimodal controllability. Especially for TC‑V2A—the “text overrides video” scenario—ControlFoley succeeds where most previous open‑source models failed.

Its position among competitors

Looking across the current controllable audio generation landscape:

• Ant Group’s Ming‑omni‑tts follows the “speech + music + sound effects unified generation” route, strong at natural‑language control, but video conditioning isn’t its focus;
• ByteDance’s OmniShow is based on the RAP2V framework (reference image + audio + pose → video), focusing on digital humans, where audio is input rather than output;
• Keling 2.6’s “visual‑audio co‑generation” produces sound alongside video generation, closed‑source and geared toward integrated content creation;
• ControlFoley, in contrast, targets the specific use case of “video already exists, generate sound effects per intent,” focusing purely on audio generation, and it’s open‑source.

This is actually an undervalued market—short‑video creators, post‑production teams, and game developers all need to dub existing footage rather than make new videos. In this niche, ControlFoley’s tri‑task unification + open‑source + SOTA combo is highly competitive.

Openness: genuine sincerity

This release shows a high level of openness:

• Complete source code available
• Model weights downloadable
• Technical report published simultaneously
• Online demo ready to try
• Ready‑to‑use Skill (likely a packaged inference tool)

For developers, that means you can run it right away, without rebuilding the ecosystem from scratch. For secondary developers, the unified architecture across tasks makes it easy to add new control modalities—like “emotion tags” or “scene type” in the future.

A quick observation

From the MiMo series to MiMo‑Audio and now ControlFoley, Xiaomi’s large‑model team has clearly accelerated its open‑source pace this year, with each project targeting a specific capability gap instead of spreading too wide. ControlFoley isn’t the biggest model nor the broadest, but in the niche of “controllable video sound effects,” it has achieved the best open‑source result. This niche‑focused open‑source strategy feels more grounded than endlessly chasing parameter size rankings.

The V2A field is likely to heat up in the coming year. As video‑generation models grow stronger and produce longer outputs, the dubbing part still largely relies on manual work or crude automatic methods. ControlFoley’s “control‑first” mindset may well become standard for future V2A models. After all, the goal of creative tools has never been just “automation,” but “automation plus control.”

The model weights and code are already available on GitHub and Hugging Face, and the online demo is live. If you want to try TC‑V2A, where “text overrides video,” you can experience it for yourself in just a few minutes.

---

References

• Xiaomi open‑sources controllable video sound‑effect generation model ControlFoley: make the sound “do what you want” – IT Home: ControlFoley launch article, including architecture diagram and task definitions