Colossus 2 simultaneously trains 7 models, with up to 10 trillion parameters.

Musk has dropped another bombshell.

On April 8, he directly posted the training task list of xAI Colossus 2 supercomputer on X—this is the world’s largest single-site AI training facility, and it’s training 7 models simultaneously. Not queueing—parallel.

The scale of the parameters on this list is quite staggering:

• Imagine V2: next-generation image generation model
• 1T variants × 2: two trillion-parameter models
• 1.5T variants × 2: two 1.5 trillion-parameter models
• 6T model: 6 trillion parameters, very likely the previously leaked Grok 5
• 10T model: 10 trillion parameters, the largest currently known model in training

When asked how long the 10T model would take to train, Musk replied: “About 2 months for pretraining.” Then added, “Still have some catching up to do.”

The subtext isn’t hard to read—xAI knows it hasn’t caught up to the first tier in some respects, but it’s showing its hand.

550,000 GPUs, 18 Billion USD—a story of insane engineering

To understand what it means to train 7 models at once, we first need to grasp what kind of monster Colossus 2 is.

In January, Musk announced Colossus 2 was officially online. Key numbers:

• About 550,000 NVIDIA GPUs (including H100/H200 and new-generation GB200/GB300)
• Total cost of around 18 billion USD
• 2-gigawatt (GW) power consumption
• 50 exaflops of compute—roughly 7× the combined speed of the world’s 10 fastest supercomputers

To put 2 GW in perspective—it’s the power consumption of a mid-sized city. Your entire neighborhood’s yearly electricity bill probably couldn’t power this machine for one afternoon.

Even crazier is the construction speed: Phase 1 of Colossus went from groundbreaking to 200,000 GPUs online in just 214 days. In the data center industry, that’s almost unimaginable—normally, even a cluster with 10,000 GPUs takes months to deploy. Musk’s approach was brute-force simple: pile up the hardware first, tune it as it runs. In his own words, it’s “first principles.”

Simply put—miracles by brute force.

Architecture breakdown: More than just stacking GPUs

Colossus 2 is not simply a warehouse stuffed with hundreds of thousands of GPUs. Running 7 models in parallel without collapsing requires engineering complexity far beyond imagination.

Compute layer: xAI didn’t use NVIDIA’s standard DGX SuperPOD design but worked with Supermicro for deep customization. Each compute node is an 8-GPU liquid-cooled tray, with cold plates directly covering the GPU, CPU, and PCIe switch chips—the three major heat sources. This chip-level direct-to-coolant (D2C) approach is practically the only choice at this scale—air cooling is a joke under 2 GW heat.

Network layer: using NVIDIA Spectrum-X 400G Ethernet instead of the traditional HPC-oriented InfiniBand. The entire network adopts a strict 1:1 non-blocking CLOS architecture built with Spectrum-4 SN5600 switches. Simply put, any two GPUs have equivalent communication bandwidth—no bottlenecks. This is crucial for gradient synchronization in large-model training.

An interesting detail: the 800G switches split their downlink ports via one-to-two AOC cables into two 400G ports for servers—a clever balance between cost and cabling complexity.

Storage layer: Phase 1 used VAST Data’s distributed storage for quick deployment; later expansion switched to DDN’s EXAScaler (Lustre-based parallel file system) plus Infinia object storage. The former serves high-throughput training I/O, the latter acts as a data lake storing raw datasets. Total storage capacity exceeds 1 exabyte (EB).

1 EB = 1000 PB = 1,000,000 TB. The 256 GB in your phone—4 million phones would equal this.

What does 10T parameters mean?

Now, back to the models themselves.

Currently, known top-tier model parameter scales look like this: GPT-4 is widely guessed at around 1.8T (MoE architecture), Meta’s Llama 3.1 reaches 405B, and Google’s Gemini Ultra is estimated in the trillion range. xAI jumps straight to 10T.

Ten trillion parameters—that number is an engineering statement: “I have the compute power to feed it.”

But size alone doesn’t equal quality. Industry insiders know this well. The key questions are:

1. Architecture? If it’s a dense model, training and inference costs would be astronomical. It’s almost certainly MoE (Mixture of Experts), where only a fraction of parameters are active per token—maybe one-tenth or less.
2. Enough data? Roughly applying Chinchilla scaling laws, a 10T model would need about 200T tokens of training data. Public high-quality internet text totals only around 10–15T tokens. So synthetic, multimodal, and proprietary data must dominate.
3. Inference strategy? Training it is one thing. Running inference affordably is another. Without heavy distillation or quantization, the inference cost of a 10T model would make most practical applications unviable.

Musk said pretraining would take two months. Given Colossus 2’s compute scale, that’s plausible. But beyond pretraining, there’s SFT, RLHF, and safety alignment—final usable models could take months more.

The 6T entry: very likely Grok 5

Among those 7 models, the standout isn’t the 10T—it’s the 6T.

When Colossus 2 launched in January, Musk clearly mentioned that Grok 5 would have 6 trillion parameters, expected for release in the first half of the year. Now the list includes one 6T model—by deduction, that’s Grok 5.

If Grok 5 indeed releases this half-year, it will directly compete with GPT-5, Claude 4, and Gemini 2.5. In parameter scale, Grok 5 might be the largest of its generation. But whether that leads to actual capability advantage depends on data quality, alignment effectiveness, and inference efficiency.

It’s worth noting that the list also includes Imagine V2, xAI’s image generation model—indicating that xAI isn’t only building language models but also expanding multimodal capability. Given the vast trove of image-text data on X, xAI has a natural advantage in visual understanding and generation.

Computation arms race: Who’s competing?

Musk isn’t alone in this madness.

• OpenAI + SoftBank + Oracle: “Stargate” project, also aiming at a GW-scale supercomputer.
• Meta: Project Prometheus, GW-scale, scheduled for early 2026.
• Google: plan to expand Cloud TPU to 1 million chips, contracts worth tens of billions USD.
• Microsoft + OpenAI: continuous Azure AI infrastructure expansion.

According to Epoch AI estimates, Colossus 2 delivers compute equivalent to 1.4 million H100s, currently the world’s most powerful. But most competitors target 2026–2027 go-live dates—so the ranking may reshuffle.

One commenter aptly put it: “The hardest part of AGI isn’t math anymore—it’s literally finding enough power outlets.”

This isn’t a joke. Baron Fung of Dell’Oro Group estimates that over the next 2–3 years, the world will add tens of gigawatts in data center power capacity. Meanwhile, residents across the U.S. have begun protesting data center construction—since 2022, household electricity prices have risen faster than inflation almost everywhere.

The cost of the compute race will ultimately show up on everyone’s electricity bill.

What does this mean for developers?

If you’re an AI application developer, this update is packed with implications:

Short-term: The Grok series will see a major capability leap. If Grok 5 launches on schedule and opens its API, it will become another strong option in tech stacks. The current Grok API is already OpenAI-compatible, so integration costs are low.

Mid-term: Emergence of 10T-scale models will push a rethink of distillation and inference optimization industry-wide. As base models grow, distilled smaller models inherit ever-stronger capabilities. For most developers, you’ll likely use distilled, efficient versions—not the 10T model itself.

Long-term: Compute centralization is accelerating. Fewer than five players globally can train 10T models. This means app-layer developers will increasingly depend on APIs from a handful of providers. In such an environment, using a unified interface layer to handle multi-provider API calls becomes valuable. Aggregation platforms like OpenAI Hub, which are OpenAI-compatible, let you flexibly switch among Grok, GPT, Claude, and Gemini without separate integration logic.

Here’s roughly how calling a Grok model via OpenAI Hub would look once available:

from openai import OpenAI

client = OpenAI(
    api_key="your-openai-hub-key",
    base_url="https://api.openai-hub.com/v1"
)

response = client.chat.completions.create(
    model="grok-5",  # actual model name per platform docs after release
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the advantages of MoE architecture over dense models."}
    ],
    temperature=0.7
)

print(response.choices[0].message.content)

The interface format is identical to GPT and Claude—just switch the model parameter.

Final thoughts

Musk’s style has always been: scale first, fix later. That’s how SpaceX works, how Tesla works, and now xAI too.

Seven models training in parallel, 10T parameters, 2-month pretraining—two years ago, those numbers would’ve sounded like bluffing. Yet Colossus 2 is built and running, with 550,000 GPUs indeed in place.

Whether these trained models are great is a separate question. Parameter count has never been the only metric—OpenAI proved reasoning compute value with o1, Anthropic proved the importance of alignment with Claude, and Google proved multimodal fusion potential with Gemini.

One thing’s certain: when a player can run seven parallel training pipelines, with the biggest reaching ten trillion parameters, competition in this industry has hit a new tier.

The coming months will be lively.

---

References

• Musk reveals xAI progress: Colossus 2 is simultaneously training 7 models — Linux.do community discussion post, including Musk’s original tweets and community interpretations