iPhone Runs a 20B Model at 120 tok/s

Maple-Preview recently made its debut in the developer community. The project’s core selling point is straightforward: a ternary-quantized 20B MoE model can generate text on an iPhone at speeds of up to approximately 120 tok/s.

If this result can be consistently reproduced on mainstream iPhones under typical context lengths and sustained workloads, it would represent a significant advance in on-device large model deployment. At 120 tok/s, this is no longer merely a matter of “running a model on a phone.” Generation is faster than most people can read, making it sufficient for real-time rewriting, code completion, voice conversations, and background multi-agent tasks.

But do not rush to interpret this as “a 20B dense model has been fully loaded onto a phone and is running at full capacity at 120 tok/s.” The two key terms at work here are ternary quantization and MoE. The former reduces model weight size and memory bandwidth requirements, while the latter invokes only a subset of experts for each token. The 20B figure is closer to the model’s total parameter count; it does not mean that all 20 billion parameters are computed for every generated token.

The Bottom Line: Worth Watching, but Still Only a Benchmark Result

As of August 5, 2026, the publicly available summary that can be verified highlights the following information:

• The model has a total size of 20B parameters;
• It uses an MoE, or mixture-of-experts, architecture;
• Its weights use ternary quantization;
• It reaches approximately 120 tok/s on an iPhone.

These figures are highly appealing for on-device inference, but they are not enough on their own to prove that Maple-Preview has become “the best 20B model to use on a phone.” Developers still need to know at least the exact iPhone model, active parameter count, context length, time to first token, prompt-processing speed, sustained generation length, thermal conditions, and model accuracy.

In other words, 120 tok/s is an important metric, but it does not represent the complete product experience.

One common benchmark ambiguity is especially important to watch for: presenting peak decoding speed under a short context as the average speed of the entire inference pipeline. When a user submits a request containing several thousand tokens, the system must first complete prefill before it can begin decoding tokens one by one. Even if decoding reaches 120 tok/s, a slow prefill stage could still leave the user waiting several seconds for the first response.

For now, the most reasonable position is to view it as a valuable engineering preview, not as a fully validated conclusion about on-device models.

Ternary Quantization Is More Than Simply Compressing Weights into Three Values

Ternary quantization generally constrains or maps model weights to three discrete states, such as -1, 0, +1. Compared with FP16, INT8, and INT4, it further reduces the number of bits required to represent each weight.

From an information-theoretic perspective, three states require a minimum of approximately 1.585 bits to represent. Assuming all 20 billion parameters could be densely encoded at the ideal 1.585 bits, the theoretical size of the weight data alone would be:

• 20 billion parameters × 1.585 bits;
• Equivalent to approximately 3.96 billion bytes;
• A theoretical lower bound of nearly 4 GB.

In practice, however, a model on a phone consists of more than just this block of data. Quantization scale factors, grouping metadata, expert indices, token embeddings, normalization parameters, runtime buffers, and the KV cache all consume additional memory. The hardware may also be unable to read data natively at 1.585 bits, so the actual packing format may use 2 bits or some other aligned representation. Consequently, a “20B ternary model” cannot simply be equated with “only 4 GB of memory.”

What makes ternary quantization fast is not merely the smaller file size.

During token-by-token decoding, large models are often constrained by memory bandwidth: the chip must repeatedly fetch weights from memory, perform a round of computation, and then repeat the process for the next token. Smaller weights mean less data movement per decoding step. This is particularly beneficial on phones, whose unified memory capacity and bandwidth are far more constrained than those of desktop GPUs.

However, whether ternary weights translate into actual speed depends on the kernel implementation. If the runtime merely reads ternary weights and then reconstructs them into higher-precision values for computation, the overhead of unpacking may consume the gains from compression. An ideal implementation requires matrix multiplication, vectorized loading, and cache layouts designed specifically around the ternary format.

The most important technical question to ask about Maple-Preview, therefore, is not “how many bits does it use?” but rather: Does it have a genuine on-device execution path optimized for ternary weights?

MoE Is the Other Half of the Story Behind the 20B Figure

MoE can be understood as splitting some of a model’s feed-forward networks into multiple experts, with a router selecting a small subset of those experts to process each token.

For example, a model may contain a dozen or even dozens of experts, while activating only two of them for each token. This allows the model to have a large total parameter capacity while keeping the computational cost of each inference step relatively low.

This is why total parameter count alone is often insufficient when comparing MoE models. The more important metrics are:

| Metric | What It Represents | Impact on Mobile Deployment | |---|---|---| | Total parameters | The combined parameters of all experts and shared layers | Determines model file size and potential memory usage | | Active parameters | The parameters actually invoked for a single token | More closely reflects per-step computational cost | | Number of active experts | The number of experts selected during each routing step | Affects speed, quality, and memory access patterns | | Shared layer size | The portion that every token must pass through | Forms an unavoidable baseline cost | | Routing overhead | The cost of selecting and scheduling experts | May offset some of the gains on mobile devices |

Therefore, “a 20B MoE model reaching 120 tok/s” is not the same thing as “a 20B dense model reaching 120 tok/s.” The former may activate only a few billion parameters—or even fewer—for each token.

That does not mean the result is inflated. The purpose of MoE is precisely to provide greater parameter capacity at a lower computational cost, and it should not be dismissed simply because the entire model is not activated at once. The issue is that the project needs to disclose both total and active parameter counts; otherwise, developers cannot make meaningful comparisons.

MoE also presents a less obvious challenge on phones: although it reduces computation, it may create irregular memory access patterns. Different tokens may be routed to different experts, requiring the runtime to frequently locate and load scattered weights. Desktop GPUs can mask some of this problem with high bandwidth and large caches, but phones are more sensitive to it.

If Maple-Preview can indeed sustain 120 tok/s, its value likely comes not only from the model architecture but also from joint optimization of expert layout, weight packing, scheduling, and low-level kernels.

What Makes 120 tok/s So Fast?

For chat applications, 20 to 30 tok/s already produces a fairly fluid experience. Above 60 tok/s, the bottleneck often begins to shift from model generation to UI rendering, network transmission, and the user’s reading speed. The significance of 120 tok/s is more apparent in scenarios where the generated tokens are consumed by machines.

For example:

1. Local code completion: An editor can generate multiple candidates in advance rather than returning only one short completion.
2. Real-time voice assistants: The model can rapidly identify intent and draft a response before handing the result to local speech synthesis.
3. Structured information extraction: Emails, text messages, and local documents can be processed in batches without being uploaded to the cloud.
4. Background agent tasks: The model generates tool-call parameters, planning steps, and intermediate states rather than long-form text for humans to read, making higher speed more valuable.
5. Speculative decoding: A small on-device model can serve as a draft model, pregenerating candidate tokens for a more capable model.

However, developers need to distinguish among at least three kinds of speed:

• Prefill throughput: The speed at which the model reads and processes the input prompt;
• Decode throughput: The speed at which it generates new tokens one by one;
• End-to-end latency: The total time from the user tapping Send to receiving the complete result.

The 120 tok/s figure most likely refers to the decode stage, but it should not be treated as the average speed across all stages until the complete testing methodology is published. An on-device model may decode extremely quickly while still pausing noticeably during loading, long-prompt processing, or initial startup.

Memory, Thermals, and Battery Life Are Harder Problems Than Peak Speed

Phone-based model demos often show only two things: that the model loads successfully and that it can generate a short passage quickly. Turning that demo into a real product requires addressing three more difficult problems.

First, the KV Cache Continues to Grow with Context Length

Quantization mainly compresses model weights; it does not automatically shrink the KV cache by the same proportion. The longer the context, the more memory the cache consumes. For applications that support long conversations, document question answering, or codebase analysis, the KV cache may gradually become the main source of memory pressure.

Further quantizing the KV cache to save memory may introduce quality loss or additional conversion overhead. Developers ultimately need to choose among context length, concurrency, and output speed.

Second, Peak Speed Is Not the Same as Sustained Speed

An iPhone chip can deliver very high performance for short periods, but sustained inference produces substantial power consumption and heat. As the device warms up, the system may reduce clock speeds. If the model is also using the camera, microphone, display, and network, even less thermal headroom remains for inference.

A credible on-device benchmark should therefore generate at least several thousand tokens and record performance at startup, during the steady state, and after thermal throttling, rather than capturing only the peak speed of the first few dozen tokens.

Third, Unified Memory Is Not Unlimited Memory

An on-device model must share memory with the operating system, application UI, keyboard, background processes, and other media workloads. Even if the model file fits on the device, that does not mean the application can remain stable under system memory pressure.

From a product perspective, the most important metric is not whether the model can launch in a laboratory. It is whether the app must reload the model after being sent to the background, whether the system terminates it during a phone call or while taking a photo, and whether acceptable fallback options exist for devices with less memory.

Beyond Speed, Model Quality Remains the Biggest Open Question

The greatest risk of ternary quantization has never been whether the model can run, but how much capability remains after compression.

If an already-trained high-precision model is directly compressed into ternary weights, it will usually suffer a significant loss of accuracy. A more effective approach is quantization-aware training, or training the model from the outset to adapt to low-bit weight distributions. This allows the model to learn how to compensate for discretization errors during training.

MoE adds another layer of complexity. Different experts may specialize in different knowledge and capabilities, and quantization errors may affect not only the output of an individual layer but also routing decisions. Once a token is sent to the wrong expert, the damage is no longer merely a numerical deviation.

Determining whether Maple-Preview is genuinely usable therefore requires at least the following types of evaluation:

• Common-sense and knowledge-based question answering;
• Mathematical reasoning;
• Code generation and repair;
• Instruction following;
• Long-context retrieval;
• Multi-turn conversation stability;
• Capability retention before and after quantization.

The ideal comparison would not be against cloud models with similar parameter counts, but against the same weights in FP16, INT4, and ternary formats. Only then can developers determine whether the 120 tok/s result comes from better engineering or from a more aggressive trade-off in capability.

What a Credible Reproduction Report Should Disclose

If the project later releases its weights, runtime, or testing tools, the following information should be examined closely:

• The exact iPhone model, chip, and memory capacity;
• The iOS version and device power mode;
• Which parts of inference run on the CPU, GPU, or neural accelerator;
• The model’s total parameters, shared parameters, and active parameters per token;
• The ternary weight encoding, group size, and scaling precision;
• The context length and generation length used during testing;
• Whether 120 tok/s is an average, sustained rate, or instantaneous peak;
• Time to first token and prefill throughput;
• Temperature, power consumption, and throttling after ten minutes of operation;
• Quality retention compared with a high-precision baseline;
• Whether the model and runtime licenses permit commercial use.

Another frequently overlooked variable is the tokenizer. Different tokenizers produce different token counts for the same passage, particularly for Chinese, source code, and mixed-language content. Comparing tok/s alone can produce cases where one model appears faster but requires more tokens to generate the same text. Rigorous testing should also report characters per second or the time required to complete a fixed task.

What It Really Means for Developers: The Boundary Between Edge and Cloud Is Being Redrawn

The value of Maple-Preview does not necessarily lie in enabling phones to replace cloud-based large models entirely. A more realistic direction is to redefine the boundary between local and cloud execution.

Privacy-sensitive, latency-critical, and relatively simple tasks can remain on the device, including notification summarization, text classification, input rewriting, and local retrieval. Complex reasoning, long-context processing, and multimodal analysis can continue to be handled by cloud models. When simultaneous access to models such as GPT, Claude, Gemini, and DeepSeek is required, cloud routing can also be handled through OpenAI-compatible aggregation APIs such as OpenAI Hub. Maple represents a separate local execution path, and the two approaches are not in conflict.

Going a step further, an on-device model could first assess the difficulty of a request: simple questions would be answered locally, while low-confidence requests would be escalated to the cloud. This could both reduce API costs and minimize unnecessary data uploads.

In this architecture, the significance of 120 tok/s becomes clear. The on-device model does not need to be the most capable model, but it must be fast enough not to become a bottleneck in the overall request pipeline.

Conclusion: Not a 20B Miracle, but a Preview of Systems Engineering

The most shareable description of Maple-Preview is that “a 20B large model runs at 120 tok/s on an iPhone.” That statement is not wrong, but it omits the crucial conditions behind the result: ternary quantization reduces the cost of reading weights, MoE lowers the amount of active computation per token, and on-device optimization determines whether those theoretical advantages can be realized in practice.

It is better understood as a demonstration of systems engineering than as a miracle created solely by a new model architecture.

If subsequent data shows that it can maintain close to this speed under common context lengths, sustained workloads, and reasonable thermal conditions without a significant collapse in model quality, Maple-Preview will represent a substantial advance in on-device large models in 2026. Conversely, if 120 tok/s appears only with short contexts, specific device models, and brief peak-performance windows, it will remain an interesting technical demonstration—but one that is still some distance from production readiness.

At this stage, the speed deserves praise, but any firm conclusion must wait for independent reproduction.

References

• Hugging Face: Search for Ternary Models — A resource for understanding ternary-quantized models and the related weight ecosystem; it does not constitute independent validation of Maple-Preview’s performance figures.
• Maple-Preview project preview page — The primary source for the claims in this article regarding the 20B MoE architecture, ternary quantization, and 120 tok/s performance on an iPhone. The original external link has been omitted due to restrictions on linked domains.