Xiaomi Open-Sources OmniVoice: One Model Covers 600+ Languages

This afternoon, Xiaomi AI Lab’s Next‑gen Kaldi team (that is, the k2‑fsa group) released OmniVoice on GitHub— a zero‑shot voice cloning TTS model claiming coverage of more than 600 languages. The repository already has 2.4K stars.

This marks the first time in the open‑source TTS field that anyone has pushed the number of supported languages into the triple digits. Over the past year, ChatTTS, CosyVoice, F5‑TTS, Fish Speech and others have been taking turns in the spotlight, but most have focused only on Chinese and English. Supporting a few dozen languages is rare, and anything approaching a hundred is almost unheard of. OmniVoice skips both “dozens” and “hundreds” altogether and jumps straight to 600+.

Start with the data: Chinese WER 0.84%, multilingual performance beats ElevenLabs

Metrics are hard currency. On the Seed‑TTS Chinese test set, OmniVoice achieved a word error rate (WER) of 0.84%, a number that’s essentially comparable to human recordings.

The real surprise comes from the multilingual results. Across 24 evaluated languages, the fully open‑source data–trained OmniVoice outperformed commercial systems ElevenLabs v2 and MiniMax in both speech similarity (SIM‑o) and WER. In an extended test of 102 languages, its intelligibility even approached or surpassed human recordings.

ElevenLabs’ reputation needs no explanation—multilingual cloning has been one of its moats. So an open‑source model beating it on multilingual evaluation, using only open‑source data, is far more noteworthy than any star count.

In terms of speed, OmniVoice’s real‑time factor (RTF) is as low as 0.025, meaning it needs only 0.025 seconds of GPU time to synthesize one second of audio—40× faster than real time. Training throughput is similarly ferocious: up to 100,000 hours of data per day, with a total of 580,000 hours of multilingual speech used for training.

Architecture: a bidirectional Transformer, non‑autoregressive, one‑step synthesis

OmniVoice’s appeal lies not only in its numbers but also in its architecture—so simple it borders on counterintuitive.

Mainstream TTS models fall into two camps:
The autoregressive (AR) camp, led by VALL‑E and CosyVoice, generates audio token‑by‑token with natural tone but slow speed;
The non‑autoregressive (NAR) camp is faster but usually requires a complex pipeline—text encoder + duration predictor + acoustic decoder + vocoder— to function, as seen in F5‑TTS and NaturalSpeech.

OmniVoice takes a third path: a single bidirectional Transformer generates discrete speech tokens directly from text—no intermediate modules at all. There’s no separate text‑modeling branch, no hierarchical token prediction, no fancy hybrid structure. The official description—“the simplest non‑autoregressive TTS model so far”—is not an exaggeration.

It adopts a Diffusion‑LM‑style discrete non‑autoregressive architecture, skipping the traditional “intermediate semantic tokens” stage and mapping text directly to acoustic tokens in one step.

This minimalist design reaches SOTA performance thanks to two key innovations:

• Full‑codebook random masking strategy. During training, random masking is applied across the entire codebook space, greatly improving efficiency so the model can see more data and learn more stable representations under the same compute budget. The impressive 100,000‑hour‑per‑day throughput owes much to this trick.
• LLM‑based parameter initialization. OmniVoice initializes its Transformer using pretrained large‑language‑model weights—the first time this has been done in a non‑autoregressive TTS. AR TTS aligns naturally with LLM structure, making initialization straightforward; for NAR, the structural gap had stopped anyone before. The biggest gain is a dramatic boost in intelligibility—whether the speech reads clearly and correctly. The benefit is especially striking for low‑resource languages, where text‑side priors matter most.

In short, non‑autoregressive TTS hadn’t been able to enjoy the LLM revolution—until Xiaomi pried the door open.

Features: describing a voice is enough to build one

Zero‑shot voice cloning is old news, but OmniVoice pushes usability forward further.

Custom voice design. No need to upload reference audio—pure text description works. Want “female, low‑pitch, British accent” or “male, high‑pitch, Sichuan dialect”, even “whispering style”? Just set the instruct field. For teams building TTS applications, this is liberating—no more maintaining massive voice libraries.

Noisy‑audio adaptation. Real user recordings often come from subways or offices with fan or keyboard noise. OmniVoice automatically filters out background noise and extracts only timbre features, producing clean voices from 3–10‑second samples recorded in noisy environments. More practical than many commercial systems.

Paralinguistic tags. Insert markers like [laughter] or [sigh] directly in the text; the synthesized voice includes actual laughter or sighs. Perfect for audiobooks, podcasts, or character dialogue—adds a whole new level of expressiveness.

Pronunciation correction. For Chinese English mixed texts, polyphonic characters and proper nouns are long‑standing pain points. OmniVoice lets users explicitly specify pronunciation using pinyin or phonetic symbols, fixing hard‑to‑pronounce words precisely.

What 600 languages mean

Putting the number 600+ on the industry map, the implication goes far beyond “just more.”

There are over 7,000 languages in the world, but commercial TTS systems usually cover only a few dozen to a hundred or so—the data‑rich, commercially valuable ones. Thousands more have less than ten hours of recorded speech and are simply ignored. OmniVoice has deliberately leveled up its low‑resource language generalization: even if a language has fewer than ten hours of training data, it can still generate high‑quality speech.

This is thanks to LLM‑based initialization: large language models have already seen vast amounts of multilingual text, giving them priors about vocabulary, phonetics, and prosody. Combined with the full‑codebook random‑masking strategy, the cold‑start cost becomes remarkably low.

Concrete value comes in at least three fronts:

1. Digital preservation of endangered languages. Languages with only a few thousand speakers can now be recorded and heard with minimal data.
2. Localization for cross‑border applications. One model covers almost every market worldwide, no need to train or buy separate models for minor languages.
3. Accessibility. Users with visual impairments who speak non‑mainstream languages can finally get high‑quality voice feedback.

Getting started: install via pip and run instantly

OmniVoice is shockingly easy to use. With PyTorch 2.8 and CUDA 12.8 installed, a single pip command fetches the model:

pip install torch==2.8.0 torchaudio==2.8.0 \
  --extra-index-url https://download.pytorch.org/whl/cu128
pip install git+https://github.com/k2-fsa/OmniVoice.git

Basic usage:

import torchaudio
from omnivoice import OmniVoice

model = OmniVoice(device_map="cuda:0")

# Zero-shot cloning: provide a 3–10 s reference audio
audio = model.generate(
    text="Hello, this is a voice cloning test.",
    ref_audio="ref.wav",
)
torchaudio.save("out.wav", audio[0], 24000)

# Descriptive voice design: no reference audio
audio = model.generate(
    text="Hello, this is a synthesized voice.",
    instruct="female, low pitch, british accent",
)

To see results immediately, the repo includes a web demo:

omnivoice-demo --ip 0.0.0.0 --port 8001

Open http://localhost:8001 in your browser for the full interactive interface. On Apple Silicon, change device_map to mps; it even runs locally on lightweight laptops.

A few thoughts

OmniVoice feels like k2‑fsa integrating years of Kaldi multilingual engineering experience with recent LLM‑for‑TTS thinking. Back in the Kaldi era they specialized in multilingual and low‑resource processing, and now they’ve extended that lineage into neural TTS—a solid technical direction.

The developer impact is direct: if your product needs to go global, you used to cobble together ElevenLabs, Azure, and various open‑source solutions per language; now, theoretically, one model and one inference stack can cover nearly all needs. While WER and SIM‑o data still need independent reproduction, the architecture and training approach make it promising beyond a “looks good on paper” model.

That said, there are caveats. Non‑autoregressive designs are fast but still lag behind top AR models (like MiniMax speech‑02 or ElevenLabs v3) in nuance and long‑text consistency. Between “reading correctly” and “reading with soul,” OmniVoice excels at the former. Also, the 580k‑hour training dataset is out of reach for most teams, and domain‑specific fine‑tuning workflows will need community exploration.

Still, it’s open‑sourced—code and weights shared, checkpoints on Hugging Face. Now it’s up to the community to see what new tricks emerge—ChatTTS rose in exactly this way half a year ago.

References

• Xiaomi open‑sources OmniVoice multilingual voice‑cloning TTS, claiming one model covers 600+ languages – IT Home: IT Home’s report summarizing Xiaomi’s official release with metrics, architecture, and feature descriptions.
• k2‑fsa/OmniVoice – GitHub: The OmniVoice open‑source repository containing full code, pretrained weights, installation guide, and web demo.
• OmniVoice model weights – Hugging Face: The k2‑fsa team’s Hugging Face page hosting OmniVoice checkpoints.