ByteDance Bets on a 10-Trillion-Parameter Model

ByteDance Pushes Model Scale Toward 10 Trillion Parameters

ByteDance is making its most aggressive foundation-model bet to date.

According to the latest report on August 7, ByteDance is pretraining a large model that could have as many as 10 trillion parameters, with the goal of competing against global frontier labs such as Anthropic. The project remains at an early stage, and ByteDance has not officially confirmed the final parameter count, model architecture, training completion date, or whether the model will be publicly released.

A day earlier, sources in China said ByteDance was discussing training a model with more than 5 trillion parameters. The two figures may appear to differ by a factor of two, but they are not necessarily contradictory: the former may refer to a baseline proposal that has already entered the discussion and resource-allocation stage, while the latter is more likely the upper limit of the training plan. Large-model pretraining typically goes through small-scale validation, medium-scale trial runs, and final large-scale training. Teams dynamically adjust their targets based on data quality, loss curves, and cluster stability.

The accurate way to put it, therefore, is not “ByteDance has already trained a 10-trillion-parameter model,” but rather: ByteDance is preparing to push the training ceiling for its next-generation foundation model into the 10-trillion-parameter range.

This distinction matters. What has surfaced so far is merely a training plan—not a model release—and there are no verifiable benchmarks, APIs, technical reports, or weights.

10 Trillion Parameters Is, First and Foremost, an Infrastructure Test

If the final scale lands near 10 trillion parameters, it will significantly exceed the trillion-parameter models currently known in China.

For comparison, Moonshot AI released Kimi K3 in July this year with approximately 2.8 trillion total parameters. It uses a mixture-of-experts (MoE) architecture with 896 routed experts, of which each token selects only 16, resulting in roughly 104 billion activated parameters. Alibaba’s Qwen 3.8-Max reportedly has around 2.4 trillion parameters. xAI is also continuing to expand the scale of Grok’s base model and has discussed pushing its next-generation model into the multi-trillion-parameter range.

If ByteDance achieves 10 trillion total parameters, it would push the upper limit of publicly visible model scale in China more than threefold. But that does not mean the model’s capabilities would improve threefold as well.

Parameter count is more like the total capacity of a warehouse than the entire volume of goods that must be moved every time an item is retrieved. In an MoE model, only a small number of experts are typically activated during each inference step:

• Total parameter count determines how much knowledge, how many patterns, and how much specialized capability the model can accommodate;
• Activated parameter count has a more direct impact on the compute cost and inference speed of each token;
• Expert-routing quality determines whether the model can assign different tasks to the appropriate subsets of parameters;
• Data and training methods determine whether those parameters contain useful capabilities or merely duplicated information and noise.

ByteDance has not disclosed whether the new model will use MoE, how many parameters will be activated per token, or whether it will incorporate multimodality or a native reasoning architecture. From an engineering-feasibility perspective, however, the cost of a dense 10-trillion-parameter model would be exorbitant, making a highly sparse MoE architecture the far more realistic route.

And the number of GPUs is not the only real challenge.

Training at this scale typically requires thousands or even tens of thousands of accelerators to run continuously for weeks or months. Communication failures, node outages, contaminated data, gradient anomalies, imbalanced expert loads, or corrupted checkpoints—any one of these problems will be magnified by the scale involved. An occasional hiccup in small-model training can translate into hours of wasted compute and substantial financial losses on a trillion-parameter cluster.

At a minimum, ByteDance will need to address:

1. Training stability: As model scale grows, loss spikes, numerical overflows, and convergence anomalies become harder to control;
2. Expert-parallel efficiency: MoE architectures require frequent token routing across machines, making network bandwidth an easy bottleneck;
3. Data supply: More parameters do not automatically create more high-quality data; duplicated corpora may instead reduce training efficiency;
4. Cluster utilization: The key is not how many accelerators have been purchased, but how much compute time is actually devoted to effective training;
5. Post-training capabilities: Pretraining only provides the foundation. Coding, tool use, agentic behavior, and long-horizon task performance still need to be shaped through reinforcement learning and environmental feedback.

In other words, 10 trillion parameters is first a test of organizational capability and infrastructure, and only then a competition in model capabilities.

The Parameter Race Is Back, but the Rules Have Changed

Over the past two years, the industry had begun to believe that the returns from simply increasing parameter counts were diminishing. Model companies shifted more resources toward reasoning-focused reinforcement learning, synthetic data, test-time compute, and agent engineering, hoping to achieve greater capabilities through better post-training.

By 2026, however, frontier labs had begun expanding their base models again.

The reason is straightforward: reinforcement learning can extract more of a model’s existing capabilities, but it is difficult for it to create representational capabilities that do not exist in the base model. Post-training is like teaching someone to answer questions more effectively, while pretraining determines how many books that person has read and how complex a pattern they can understand. If frontier intelligence remains constrained by foundation-model capacity, continued optimization on a smaller base model may hit a ceiling sooner.

This is the core logic behind ByteDance’s bet. It is not simply assuming that “more parameters mean more intelligence.” Rather, it is seeking to avoid a different risk: if next-generation general-purpose models still require extreme scale, failing to build the corresponding training capabilities today may mean not even qualifying for a seat at the table tomorrow.

External observers have also cited industry estimates—unconfirmed by Anthropic—that some of its next-generation models may contain 5 trillion to 8 trillion parameters. Because Anthropic has not disclosed exact architectures or parameter counts, the supposed comparison of “10 trillion versus 8 trillion” should not be treated as a rigorous head-to-head benchmark, much less as evidence that ByteDance has already surpassed Claude.

Different models may vary in total parameters, activated parameters, training tokens, data quality, and inference budgets. Ranking them solely by total parameter count is like evaluating database performance based only on server storage capacity: simple, but fundamentally unreliable.

Zhang Yiming Does Not Want to Replicate Claude

This plan also reflects a shift in ByteDance’s foundation-model strategy.

According to previous reports, Zhang Yiming recently emphasized within Seed that it would be acceptable for the company’s models to lag behind for a period of time, but that the goal should be to pursue the upper limits of intelligence. He also reportedly opposed relying on distillation from competitors’ models. In his view, distillation can help a model quickly imitate the output style and some capabilities of leading products such as Claude, but it is difficult to establish genuinely independent capability boundaries through distillation.

This is not an easy choice for ByteDance.

Distillation is currently one of the fastest ways to improve a model’s cost-performance ratio: use a stronger model to generate reasoning traces, code, and preference data, then train a smaller model on that data. This can significantly improve benchmark performance in the short term. Moving away from reliance on distillation means that the team must build its own pretraining data, training recipes, reinforcement-learning environments, and evaluation systems. Progress may be slower, and the cost of trial and error will be higher.

From the perspective of frontier competition, however, this direction is reasonable. An imitator can approach the teacher’s level, but it is difficult to use teacher-provided data to discover capabilities that the teacher itself has not yet mastered. If ByteDance truly wants to enter the global top tier, it must build its own closed loop of data and training rather than remain a long-term follower of Claude’s capability distribution.

This is especially true in coding. Coding tasks have relatively clear mechanisms for verifying results: whether a program compiles, whether tests pass, and whether an agent successfully completes repository-level modifications can all be evaluated by the environment. This makes coding one of the most suitable fields for early breakthroughs in reinforcement learning and long-horizon agent training. It is also a major battleground in which Anthropic, OpenAI, Google, and xAI continue to invest.

ByteDance already operates coding products such as Trae. If it can connect real-world development tasks and tool-use trajectories with foundation-model training, it would gain not merely a coding assistant, but an environment that continuously generates training feedback.

ByteDance’s Biggest Advantage Is Not Just Compute

Among Chinese technology companies, ByteDance is indeed one of the few players capable of undertaking a project of this kind.

On the one hand, it has sufficiently strong cash flow and capital-expenditure capacity to treat large-model R&D as a long-term infrastructure investment rather than a project budget that must deliver short-term returns. On the other hand, ByteDance has large-scale product entry points such as Doubao, Douyin, TikTok, Feishu, CapCut, and Trae, enabling it to observe how models fail in real-world tasks.

Doubao reportedly has approximately 324 million monthly active users. The value of this figure lies not only in the size of its user base, but also in the density of feedback it can generate. Competition among frontier models is shifting from “who can build a chatbot that is better at conversation” to “who can continuously identify problems, collect feedback, and iterate on models within real-world workflows.”

Many labs have models but lack sufficiently large user scenarios. Many internet companies have users but lack frontier pretraining capabilities. ByteDance has both, and that is the most noteworthy aspect of this enormous bet.

However, user data cannot be converted directly into a training advantage without processing. Privacy compliance, data authorization, quality filtering, and task structuring will all determine whether feedback can be used effectively. Hundreds of millions of monthly active users are merely raw material; whether that material can be processed into reliable training signals still depends on the engineering system.

For Developers, There Is No New Model to Use Yet

Although “10 trillion parameters” is an eye-catching figure, developers do not currently need to adjust their model selection or production architecture.

The reason is simple: this is not a model release. There is currently no API, pricing, context-window specification, rate limit, tool-calling protocol, or service-level commitment. It is not even clear whether the final model will be offered publicly under the Doubao brand.

Rather than parameter count, developers should pay closer attention to the following practical metrics in the future:

• Whether coding and repository-level agent tasks can be completed reliably, rather than merely achieving high scores on single-turn problems;
• Whether information retrieval and instruction following improve alongside a larger context window;
• Whether MoE routing causes output variability or unstable latency;
• Whether the total cost per task can be lower than competing products such as Claude, GPT, and Gemini;
• Whether the API remains compatible, and whether tool calling and structured outputs are suitable for production environments;
• Whether throughput, time to first token, and availability-zone coverage meet requirements under high concurrency.

For enterprise users, the strongest model is not necessarily the one with the most total parameters, but the one that can reliably complete tasks within a given budget. A medium-sized model that can handle 95% of support tickets at lower cost will generally offer more commercial value than an expensive, ultra-large model that solves only a small number of additional edge cases.

This Is a Ticket to the Table, Not Proof of Victory

Our assessment is that ByteDance’s training plan deserves attention, but not premature celebration.

Its significance lies in the fact that ByteDance has, for the first time, openly signaled its intention to push the scale of its foundation models into the same range as the world’s leading closed-source frontier labs. Compared with its previous approach of catching up through products, traffic, and rapid iteration, a 10-trillion-parameter project means it is preparing to tackle the most difficult problems in pretraining and systems engineering head-on.

But parameter count is only one prerequisite for capability. Without matching data, training algorithms, post-training environments, and product feedback, 10 trillion parameters may ultimately amount to nothing more than an extremely expensive machine. Conversely, if ByteDance can connect an ultra-large MoE model, reinforcement learning for coding, and feedback from hundreds of millions of users, it may truly develop competitive strengths distinct from those of Anthropic and OpenAI.

This is therefore not news that ByteDance has “surpassed Claude.” It is news that ByteDance has decided to spend heavily on a ticket to the next round of frontier-model competition.

Whether the model can be successfully trained, how many parameters it ultimately activates, whether inference costs can be reduced, and whether its capabilities enter the global top tier are the four issues that truly merit continued attention.

References

• IT Home: ByteDance Is Discussing Training a Model With More Than 5 Trillion Parameters — A summary of early reports from China covering the project’s stage, the head of the Seed team, and Zhang Yiming’s internal views on the foundation-model strategy.