Can Quantization Loss Really Vanish Statistically?

Recently, a paper titled Statistically-Lossless Quantization of Large Language Models sparked discussion in the machine learning community. The paper proposes Statistically-Lossless Quantization (SLQ): even if a quantized model differs from the original model in its weights, activations, and individual outputs, it can still be considered “statistically lossless” as long as those differences do not result in a statistically identifiable degradation in capability.

This does not turn INT4 into lossless encoding in the mathematical sense, nor does it claim that quantization errors simply disappear. More precisely, SLQ redefines how “loss” is determined: rather than requiring a low-precision model to reproduce the original model parameter by parameter and token by token, it requires the two models to remain statistically equivalent within predefined error bounds.

This is a worthwhile direction. The main problem with large language model quantization today may not be a lack of methods, but that the industry is still accustomed to drawing conclusions from a handful of average benchmark scores. If the average score drops by only 0.1, does that really mean there was no impact, or were there too few samples and too much variance for the test to detect it? At the very least, SLQ brings this long-neglected issue into the open.

However, “statistically lossless” is also a term that can easily be turned into marketing language. Whether the claim holds depends heavily on the test set, statistical power, equivalence margin, and inference configuration. With a different set of long-context, code-generation, or highly sensitive tasks, the conclusion could be entirely different.

It Eliminates Observable Degradation, Not Error

Traditional linear quantization can be simplified as follows:

q = clip(round(x / scale) + zero_point)
x_hat = scale * (q - zero_point)
error = x_hat - x

As long as the number of bits is finite, error generally cannot be zero everywhere. When weights are compressed from BF16 or FP16 to INT8, INT4, or even lower bit widths, many original values map to the same quantization grid point, meaning that information has already been lost at the level of numerical representation.

Therefore, the “lossless” in SLQ cannot be understood in the same way as file compression. ZIP decompression requires every byte to be identical; statistically lossless quantization is more like replacing a measuring instrument with a lower-precision but calibrated device: individual readings may differ, but within the intended operating range, the mean, variance, bias, and final decisions do not change beyond an acceptable tolerance.

For large language models, the evaluation can be divided into three levels:

1. Numerical level: Whether the error distributions of weights, activations, or attention scores exhibit significant bias, and whether outliers are systematically flattened.
2. Behavioral level: Whether token probabilities, rankings, hidden states, or generation distributions exhibit detectable drift.
3. Task level: Whether metrics such as accuracy, perplexity, code pass rate, factuality, and instruction following remain within predefined equivalence intervals.

The first two levels are closer to mechanism checks, while the third corresponds to the product outcomes developers actually care about. A quantized model may have very low mean squared weight error yet answer more questions incorrectly because a few critical channels were damaged. Conversely, its weight error may appear substantial, but those errors may cancel one another as they propagate through the network, leaving final task performance virtually unchanged.

This is also the most important insight of SLQ: quantization quality cannot be defined solely by parameter reconstruction error; it must ultimately be evaluated in terms of statistical equivalence in model behavior.

“No Difference Detected” Does Not Mean “No Difference Proven”

This is the point that requires the most caution when interpreting SLQ.

Suppose a BF16 model achieves 72.4% accuracy on a test set, while an INT4 model achieves 72.2%. Simply reporting that it “lost only 0.2 percentage points” provides incomplete information. Developers also need to know:

• How many samples the test set contains;
• Whether both models answered the same set of samples;
• How wide the confidence interval for the difference is;
• What degradation margin is considered acceptable;
• Whether repeated tests were conducted across multiple tasks;
• Whether decoding randomness was controlled.

If the 95% confidence interval for the difference is [-1.1%, 0.7%], this only indicates that the experiment was not precise enough to rule out a true degradation exceeding 1%. In this case, “the difference is not significant” does not mean that the two models are equivalent.

A more reasonable approach is to conduct an equivalence test. First define an acceptable business margin—for example, an accuracy drop of no more than 0.5 percentage points—and then determine whether the confidence interval for the difference lies entirely within [-0.5%, 0.5%]. Only when this condition is satisfied can the quantized version be described as statistically equivalent for that task, data distribution, and confidence level.

Put simply:

Traditional significance testing: Can we prove that the two models are different?
Equivalence testing: Can we prove that the two models are sufficiently similar?

The two may appear to differ only in how the question is phrased, but their actual thresholds are completely different. With a small sample, the former can easily produce “no significant difference”; the latter requires sufficient statistical power to narrow the confidence interval until it fits within the equivalence margin.

How Developers Can Validate Their Own Quantized Models

The engineering value of SLQ lies not only in a specific quantization algorithm, but also in providing a reusable acceptance-testing framework. Teams preparing to deploy GPTQ, AWQ, SmoothQuant, FP8, or custom low-bit models should use paired evaluations rather than merely comparing two aggregate scores.

Because the original and quantized models process the same requests, each sample produces a pair of results. Let d_i represent the change in the quantized model relative to the original model on sample i. Bootstrapping these paired differences is generally more effective than assuming that the two sets of samples are independent.

The following simplified code illustrates the process and does not correspond to any specific model API:

import numpy as np

# Each element is for the same sample:
# quantized model score - original model score
paired_delta = np.asarray(quant_scores) - np.asarray(base_scores)

rng = np.random.default_rng(42)
boot_mean = []

for _ in range(10_000):
    sample = rng.choice(paired_delta, size=len(paired_delta), replace=True)
    boot_mean.append(sample.mean())

low, high = np.percentile(boot_mean, [2.5, 97.5])
equivalence_margin = 0.005

is_equivalent = (
    low >= -equivalence_margin and
    high <= equivalence_margin
)

print({
    "mean_delta": paired_delta.mean(),
    "confidence_interval": [low, high],
    "statistically_equivalent": is_equivalent,
})

In a real-world implementation, four groups of control variables must also be added.

1. Keep Inference Conditions Fixed

The model version, prompt template, sampling parameters, maximum output length, and stop words must be identical. For generation tasks, it is best to test both greedy decoding and sampling with a fixed random seed. Otherwise, sampling noise may be greater than the quantization error itself.

2. Look Beyond the Final Answer

Final accuracy is a discrete metric and is not sensitive enough to small drifts. When possible, the following should also be recorded:

• Changes in the log probability of the correct token;
• Changes in the top-k token set and ranking;
• The position where the first divergent token appears;
• Output length and refusal rate;
• The success rate of structured parsing for tool-call arguments;
• Perplexity or recall at long-context positions.

These metrics function like “smoke detectors.” Before task scores decline noticeably, they may already reveal that quantization errors are accumulating in certain layers or regions of the input space.

3. Evaluate High-Risk Slices Separately

Global averages are especially prone to concealing tail problems. Code models should be tested separately on long files, less common programming languages, and multi-turn edits; agents should be tested on JSON arguments, function names, numbers, and dates; and RAG systems should be tested on long contexts, similar entities, and citation consistency.

If 95% of ordinary question-answering results remain unchanged, but the error rate for decimal places in monetary amounts doubles, the model cannot be called lossless merely because its aggregate score is stable. For production systems, equivalence margins for critical slices should generally be stricter than those used for public benchmarks.

4. Handle Multiple Comparisons

When more than a dozen benchmarks are tested at once, some metrics will inevitably improve or worsen by chance. Teams need to predefine primary metrics and use confidence intervals or false discovery rate controls for secondary metrics—or, at a minimum, clearly distinguish exploratory conclusions from acceptance-testing conclusions.

If the statistical criterion is chosen only after reviewing the results, “statistically lossless” will quickly deteriorate into a marketing label.

It Does Not Compete With PTQ or QAT at the Same Level

PTQ (post-training quantization) and QAT (quantization-aware training) answer the question, “How do we obtain a low-precision model?” SLQ is more concerned with answering, “How do we prove that this low-precision model remains usable?”

PTQ is inexpensive, requiring only a small amount of calibration data or even no training at all, making it suitable for rapid compression. QAT simulates quantization noise during training or fine-tuning and can generally achieve better accuracy at low bit widths, but at a higher cost. AWQ, GPTQ, outlier protection, mixed precision, and adaptive rounding all reduce errors at different points in the process.

SLQ’s statistical perspective can encompass these techniques rather than necessarily replacing them:

• First use PTQ to obtain a candidate model;
• Decide whether to retain some higher-precision weights based on layer sensitivity;
• Use statistical equivalence tests for acceptance;
• If critical tasks fail, move to QAT or adjust the quantization configuration.

This is closer to a mature engineering process than “quantize all layers uniformly to INT4, then run MMLU once.”

Statistical Equivalence Does Not Necessarily Mean Deployment Benefits

Even if a quantized model passes an SLQ-style acceptance test, that does not mean it will necessarily be faster.

Low-bit weights can indeed reduce model file size and memory usage, but inference speed depends on the hardware and kernels. If the GPU lacks an appropriate INT4 or FP8 matrix-compute path, or if the inference engine must frequently dequantize values, conversion overhead may offset the memory-bandwidth savings. Small-batch and long-context scenarios are also affected by KV cache usage, attention operators, and scheduling overhead.

Final acceptance testing should therefore include at least four categories of metrics:

| Dimension | Recommended Metrics | | --- | --- | | Model quality | Accuracy, perplexity, code pass rate, equivalence confidence interval | | Resource usage | Weight memory, KV cache, peak memory | | Performance | Time to first token, tokens/s, concurrent throughput | | Stability | Tail latency, OOM rate, abnormal output rate, cross-hardware consistency |

Quantization delivers real engineering value only when all three conditions are met: equivalent quality, lower cost, and higher speed. Statistical methods address how to prove the first condition; they do not automatically solve the other two.

Our Assessment: The Direction Is Right, but “Lossless” Needs Qualification

The strongest aspect of SLQ is that it advances quantization evaluation from point estimates to statistical inference. Large language model outputs are inherently stochastic, and benchmarks are only finite samples drawn from the distribution of real-world requests. Incorporating confidence intervals, equivalence margins, and statistical power into the quantization workflow is more meaningful than continuing to debate leaderboard differences of 0.1 points.

However, statistically lossless is always a conditional claim: for a particular model, quantization configuration, input distribution, set of metrics, and tolerance margin, no practically meaningful degradation was observed. It cannot be generalized into a claim that “the model suffers no loss on any task.”

Three misinterpretations should be avoided in particular:

1. Statistically lossless does not mean token-by-token identical. For generative models, a small change in the probability of an early token can cause the subsequent text to follow a completely different branch.
2. Equivalence on public benchmarks does not mean equivalence on production traffic. Internal enterprise data distributions are often narrower and may be more sensitive to numbers, formatting, and proper nouns.
3. Equivalent means do not imply safe tails. A small number of catastrophic degradations on high-risk requests may be diluted by a large number of easy samples.

SLQ is therefore better suited as a set of quantization acceptance principles than as a conclusion that “accuracy loss has been eliminated.” The real change it promotes is requiring developers to report not only an average score when releasing quantized models, but also the sample size, confidence interval, equivalence threshold, task slices, and hardware benefits.

If these standards become common practice, large language model quantization will rely less on subjective judgments that “the difference is not visible to the naked eye” and more on reproducible, auditable engineering evidence. For developers, that is more useful than a “lossless” label that sounds absolute but is actually subject to numerous conditions.

References

• Reddit: Discussion of Statistically-Lossless Quantization of Large Language Models: A machine learning community discussion of the paper’s concepts, experimental conclusions, and definition of “statistically lossless.”
• Zhihu: Overview of Large Language Model Quantization: An introduction to common quantization approaches such as PTQ and QAT and their accuracy trade-offs.
• Zhihu: “Slimming Down” Large Language Models—Core Quantization Principles and Challenges: An explanation of the relationship between low-bit matrix computation, memory usage, and hardware acceleration.