ByteDance Is Betting on a 5 Trillion-Parameter Model

According to an August 6 report by LatePost, ByteDance is internally discussing training a model with more than 5 trillion parameters. If the plan is ultimately carried out, its total parameter count would surpass Alibaba’s Qwen 3.8-Max, with 2.4 trillion parameters, and Moonshot AI’s Kimi K3, with 2.8 trillion parameters, making it the largest known model by parameter count in China.

However, this news comes with two important qualifiers: “under discussion” and “not necessarily to be released.” The project is still in its early stages. ByteDance has not publicly confirmed the training plan, nor has it disclosed the model architecture, training data, chip scale, or expected launch date. More precisely, this is not the release of a new model, but a high-risk technological bet that ByteDance’s Seed team is considering.

The Scale Could Be Twice That of Its Peers, but Parameters Alone Do Not Guarantee Capability

Based on publicly available reports, 5 trillion parameters would be an extremely aggressive figure.

Most of the trillion-plus-parameter models disclosed in China so far use MoE, or mixture-of-experts, architectures. These models divide themselves into multiple expert modules and activate only a subset of them for each computation. As a result, the model’s “total parameter count” may appear enormous, while the number of parameters actually used during any single inference is much smaller.

This distinction is critically important. Suppose a model has 5 trillion total parameters but activates only 500 billion parameters for each token. Its training cost, inference memory requirements, and compute needs would be completely different from those of a dense 5 trillion-parameter model. The former is more like a factory with many specialized workshops, only a few of which operate at any given time; the latter is like the entire factory running simultaneously.

Therefore, at this stage, “5 trillion parameters” cannot simply be interpreted as “twice the capability of a 2.4 trillion-parameter model.” The final results will depend on at least the following variables:

• Effective parameter count: How many parameters are activated for each computation, and whether expert routing is stable;
• Number of training tokens: Whether the data is sufficient for the model to learn effectively as its parameter count expands;
• Data quality: The proportion of high-quality code, mathematics, reasoning, and multimodal data is often more important than the total volume of raw data;
• Training stability: The larger the model, the more likely it is to encounter systemic problems in optimization, communication, fault recovery, and long-context training;
• Post-training capabilities: Pretraining provides foundational capabilities, while instruction following, reasoning, tool use, and safety still require extensive post-training;
• Inference cost: Even if the model is more capable, activating too many parameters could make it difficult to serve large numbers of users at an acceptable cost.

In other words, 5 trillion parameters are more like an “upper-limit engineering project” than an automatic pass to the top of the capability rankings.

Why Is Seed Suddenly Considering Pushing the Scale to the Limit?

People close to Seed told LatePost that the Seed team conducted several rounds of reflection during the first half of this year. Its core conclusion was that if ByteDance continued competing with peers using similar model sizes and technical approaches, it might be able to keep narrowing the gap but would struggle to establish a genuine lead.

This led to another idea within the team: rather than making incremental improvements at the current scale, why not directly expand the parameter count to several times that of its competitors and attempt to trade model size for a higher capability ceiling?

This is a typical “long-pole” competitive strategy. Instead of seeking to lead in every metric, it concentrates resources on the direction most likely to create a generational advantage. ByteDance is not entirely without a foundation for training an ultra-large model. The company has accumulated extensive large-scale machine-learning expertise through its recommendation, advertising, search, and content-distribution systems. It has also continued investing in AI chips, data centers, and infrastructure, giving it the ability to coordinate data, algorithms, engineering, and product resources simultaneously.

But that is also where the risk lies. One ByteDance insider described expanding the parameter count to several times that of its peers in a single step as “something like a gamble.” The reason is straightforward: as model size increases, the returns curve may no longer grow linearly, while engineering complexity, financial investment, and the cost of failure are almost certain to rise rapidly.

The Project Is Led by Two Executives with Search, Advertising, and Recommendation Backgrounds

According to the report, the new model plan will be led by Xiang Liang, head of Seed Foundation, in collaboration with Shen Ke, who oversees pretraining data for large language models.

Both executives come from ByteDance’s search, advertising, and recommendation operations. Xiang Liang graduated from the University of Science and Technology of China and later pursued a PhD at the Institute of Automation of the Chinese Academy of Sciences. He joined ByteDance in 2016, where he previously led the AML machine-learning platform team before becoming head of the Foundation team for the Doubao large model. Shen Ke joined ByteDance after graduating from Tsinghua University in 2018 and currently focuses primarily on pretraining data for large language models.

This personnel combination reveals two key priorities for the project: first, the foundational model architecture and training systems; second, data supply and data governance. For a model at the 5 trillion-parameter scale, the algorithm lead is not the only central figure. The data lead will be equally important in determining whether the project can actually get off the ground.

As the parameter count grows, data requirements will not simply increase by multiplying the existing corpus several times. The model will require more high-quality, low-redundancy data that has been filtered and decontaminated. It will also have to address issues such as leakage between training and evaluation sets, distortion in synthetic data, copyright concerns involving code data, and multilingual coverage. Without sufficient high-quality data, an ultra-large model may simply memorize existing patterns more thoroughly rather than acquire stronger generalization capabilities.

Seed is reorganizing its structure, dividing responsibilities, and allocating resources. The report says ByteDance’s senior management is also pushing to reduce its internal horse-race mechanism and concentrate resources in the same direction as much as possible. Even when different teams’ work overlaps, the company intends to define boundaries clearly and strengthen collaboration.

This could represent an important shift in ByteDance’s AI organization. In the past, the horse-race mechanism allowed multiple teams to rapidly test and iterate, making it particularly effective in product competition. Training an ultra-large foundation model, however, is more like a long-term systems-engineering project. Repeatedly duplicating the same teams and resources can fragment compute, data, and talent. For a 5 trillion-parameter model, resource concentration may be more important than the speed of localized innovation.

Zhang Yiming’s Signal to Seed: Temporary Lag Is Acceptable, but Do Not Merely Chase Hot Trends

According to the report, at an all-hands Seed meeting two weeks ago, ByteDance founder Zhang Yiming reassured the team. He said that training large models is inherently extremely difficult, and that the team should not become excessively anxious over temporarily lagging results. ByteDance could accept falling behind the industry for a period of time, but its goal should be to pursue the upper limits of intelligence and ultimately enter the global first tier.

Zhang Yiming also identified programming as a key area at present and explicitly stated his opposition to distillation.

Here, “distillation” generally refers to having a smaller model learn from a larger model’s outputs, reasoning traces, or behavioral patterns, allowing it to reproduce some of the larger model’s capabilities at lower cost. This approach can be highly effective in the short term, particularly for improving code generation, mathematical question answering, and tool use—tasks that can be evaluated relatively easily. Zhang Yiming’s view, however, is that distillation essentially copies capabilities already demonstrated by leading models such as Claude. Continuing along this path may at most keep narrowing the gap, but it will be difficult to truly overtake them.

This also explains why ByteDance may be shifting its bet back toward pretraining scale. Rather than carrying out more “imitation” and incremental optimization around existing models, it could attempt to train a foundation model with a higher capability ceiling.

Of course, opposing distillation does not mean rejecting all external data or refusing to conduct post-training. The real question is whether the team is copying an answer that has already been validated or building its own source of capabilities. The latter requires stronger pretraining data, reasoning training, environmental interaction, and long-term evaluation systems, and it costs far more than follower-style optimization.

Coding Is a Breakthrough Point, but It Cannot Be the Only Direction

ByteDance’s internal emphasis on programming is consistent with the current reality of large-model competition. Coding tasks offer clear feedback, can be verified automatically, and have high commercial value. Whether a model can generate executable code, locate bugs, use tools, and understand large codebases is often easier to quantify than a general chat experience.

But the popularity of coding may also create new misconceptions.

A model’s strong performance on public coding benchmarks and short-range code completion does not necessarily mean that it possesses general intelligence. In real development environments, a model must also handle requirements decomposition, repository comprehension, dependency configuration, test failures, version rollbacks, and multi-turn collaboration. Future competition may focus not merely on whether a model “can write a piece of code,” but on whether it can function as an agent that continuously completes tasks in complex software-engineering environments.

From this perspective, the value of a 5 trillion-parameter model should not be measured solely by a single coding leaderboard. It should instead be evaluated across longer task chains: Can it maintain long-context consistency? Can it perform complex planning? Can it correct errors based on tool feedback? Can it handle uncertainty in real-world information across languages and modalities?

The Biggest Challenge Is Not Buying Chips, but Getting the System to Run

It is easy for outside observers to interpret this type of news as “ByteDance plans to buy more GPUs.” But the difficulties of a 5 trillion-parameter model extend far beyond the scale of its compute resources.

The first challenge is parallel training. Model parameters, optimizer states, and activations must be distributed across a large number of devices, involving data parallelism, tensor parallelism, pipeline parallelism, and expert parallelism. Once the number of devices becomes sufficiently large, communication is no longer a background detail; it directly determines training speed. A single network fluctuation, node failure, or imbalance in expert workloads can significantly reduce the efficiency of the entire cluster.

The second is GPU memory and checkpoint management. Even with MoE, model weights, gradients, optimizer states, and intermediate training results create enormous storage pressure. The model needs to save checkpoints frequently so that training can be restored after hardware failures or divergence. For an ultra-large model, a checkpoint is not simply “a file to save,” but an infrastructure project involving distributed storage, validation, version management, and rapid recovery.

The third is data throughput and data governance. The training cluster may consume an astonishing amount of data every second. If the data pipeline cannot keep up, expensive training chips will sit idle; if data cleaning is inadequate, duplicate content, low-quality text, and contaminated samples will undermine the model’s final capabilities.

The fourth is cost and inference deployment. Successful training is only the first hurdle. Once deployed, the model will also have to handle hundreds of millions or even more requests per day. The larger the total parameter count, the more important model compression, quantization, routing optimization, caching, and service orchestration become. A model that performs impressively in a laboratory but costs too much per call will struggle to generate actual revenue through Doubao, enterprise APIs, or other businesses.

ByteDance has already been steadily expanding its compute investments and advancing hardware deployment for both training and inference. Industry sources indicate that ByteDance is also discussing large-scale purchases with domestic chip manufacturers, with the training and inference supply chains gradually developing a division of labor. For a 5 trillion-parameter model, a multi-vendor cluster can reduce dependence on a single hardware path, but it will also introduce additional complexity in compilers, communication libraries, operator adaptation, and scheduling systems.

What This Bet Means for Competition Among Chinese AI Models

If ByteDance ultimately launches and completes the training, competition among large models in China will shift further from “who releases the next version first” toward “who can withstand a longer cycle and a more expensive infrastructure war.”

Previously, the industry placed greater emphasis on small models, distillation, and inference efficiency because these directions could be deployed more quickly and made costs easier to control. Now, leading companies in China and abroad are once again discussing models with trillions or even several trillion parameters. This suggests that the upper limit of foundation-model capabilities has not yet been fully explored.

That does not mean the industry will return to an era of simply piling on parameters. A more likely scenario is that the foundations of large models will continue to scale toward ultra-large sizes, while techniques such as MoE, sparse activation, quantization, speculative decoding, and specialized post-training will be used to reduce actual usage costs. Pursuing scale during training and sparsity and efficiency during inference may become the mainstream combination in the next stage.

For developers, the truly important question is not whether ByteDance will “exceed 5 trillion” as a number, but whether the following three signals emerge:

1. Whether clear model architecture and training progress are disclosed, especially the distinction between total parameters and activated parameters;
2. Whether reproducible public evaluations are provided, covering coding, reasoning, agents, multilingual capabilities, and long contexts rather than reporting only a single leaderboard;
3. Whether a stable API or product entry point can be opened to the public, with genuine competitiveness in price, latency, and context length.

For now, this remains a plan under early-stage discussion. Whether ByteDance can turn “we want to train a 5 trillion-parameter model” into a usable, deployable, and sustainably iterative product will depend on organizational adjustments, compute preparations, data development, and training results over the coming months.

Our assessment is that this is not an ordinary parameter race, but a renewed bet by ByteDance on the foundation-model route. It could raise the upper limit of model capabilities in China, or it could be scaled back midway if its data, compute, or engineering systems fail to keep pace. At least today, 5 trillion parameters are more like a chip placed on the table than a real product—the distance between the two remains considerable.

Sources

• ITHome: Largest Known Scale in China! ByteDance Reportedly Discussing Training a Model with More Than 5 Trillion Parameters: Cites a report by LatePost and introduces ByteDance’s ultra-large-model plan, the teams responsible, and Zhang Yiming’s related views at the all-hands Seed meeting.