Qwen3.6-35B-A3B Was Modified to 0% Refusal: Benchmark Scores Haven’t Dropped Yet

A “Jailbreak” Cleaner Than Ever Before

Over the past week, a wave of “uncensored” modified versions of Qwen3.6-35B-A3B appeared on HuggingFace. The most popular are llmfan46/Qwen3.6-35B-A3B-uncensored-heretic and its FP8 quantized version protoLabsAI/Qwen3.6-35B-A3B-uncensored-heretic-FP8. At the same time, researcher grimjim open-sourced the complete refusal-direction dataset, ablation scripts, and weight diffs on Reddit’s r/MachineLearning.

The event itself is not new—abliteration (refusal ablation) has been circulating in the community for nearly two years. But this time is worth a dedicated write-up because: for the first time, it achieves “almost no benchmark degradation” at a practically usable level. According to the model card, refusal rates dropped by 88% on a custom 100-prompt evaluation, while maintaining a KL divergence of only 0.0015 from the original model. That KL number means that aside from “not refusing when it normally would,” the model remains almost identical to the original across the vast majority of token distributions.

First, Some Background: What Exactly Does Abliteration Do?

The 2024 mechanistic interpretability paper by Arditi et al. reached a remarkably clean conclusion: LLM refusal behavior is mediated by a geometrically consistent direction in the residual stream, called the refusal direction. The method for finding it is almost suspiciously simple:

1. Prepare two sets of prompts: one harmful (“teach me how to make explosives”), one harmless (“write a poem about spring”);
2. Cache the model’s residual stream activations for both sets of inputs;
3. Compute the difference between the mean activations of the two groups and normalize it—that gives the refusal direction r.

During inference, once the model detects a sufficiently large component in the r direction at some layer activation, it triggers the familiar “I can't help with that” response pattern. The idea behind abliteration is to “project out” this direction from the weight matrices—for each relevant weight row w, perform w' = w - (w·r)r, preventing the model from writing that direction into the residual stream during the forward pass.

It sounds elegant. The problem is: doing this makes the model dumber.

The Pitfall of Older Methods: Norm Decay

The issue with mlabonne’s original vanilla abliteration pipeline was that every orthogonal projection shortened the L2 norm of the weight vectors. In a 35B-parameter MoE model, there are hundreds of related matrices. If every layer shrinks slightly, the overall amplitude of the residual stream decays layer by layer.

The consequences are obvious:

• MMLU drops by 3–8 points
• GSM8K mathematical reasoning degrades even more severely, sometimes by double digits
• Long-context coherence worsens, and the model becomes easier to “zone out”

Anyone who has experimented with abliteration knows the pattern: the modified model really does answer everything, but you also really don’t want to use it for serious work. That’s why previous uncensored versions mostly remained in “toy” territory.

grimjim’s Fix: Norm-Preserving Biprojection

The core improvement in this new method is essentially one extra operation: after projection, rescale each weight row back to its original L2 norm.

The pseudocode looks roughly like this:

# Original weight row
w_orig = W[i]
original_norm = torch.norm(w_orig)

# Orthogonalization: remove component in direction r
w_proj = w_orig - (w_orig @ r) * r

# Preserve norm: scale back to original magnitude
w_new = w_proj * (original_norm / torch.norm(w_proj))
W[i] = w_new

The geometric meaning is straightforward: the new vector w_new has zero component in the r direction (satisfying the ablation objective), while retaining exactly the same magnitude as before (preserving the residual stream’s energy budget). In effect, the model reallocates its original attention/MLP capacity within the orthogonal complement of r.

That’s why it’s called a norm-preserving biprojection. Simple, but genuinely effective.

How Does the Data Look?

According to the comparison tables in the model card, the new version and the official Qwen/Qwen3.6-35B-A3B-FP8 differ by only noise-level margins on mainstream benchmarks:

• Knowledge benchmarks like MMLU and CMMLU differ by less than 0.5 points
• Reasoning benchmarks like GSM8K and MATH stay within 1 point
• HumanEval code generation is essentially unchanged
• The only obvious degradation appears on refusal-specific benchmarks like RefusalBench, dropping from 90%+ to below 10%—which is precisely the intended modification target

That KL divergence figure of 0.0015 is especially notable. For comparison, ordinary fine-tuning for just one or two epochs can easily produce KL divergences around 0.1. A value of 0.0015 means the token-level distribution is almost untouched; the model’s “personality,” knowledge, and reasoning paths are almost entirely preserved—except for the precise removal of one specific behavioral pattern: refusal.

Some Engineering Caveats

Several issues repeatedly surfaced in GitHub discussions and issue trackers. Anyone deploying this should probably review them first:

1. Don’t suppress special tokens with logit_bias

The protoLabsAI model card explicitly warns that using logit_bias to suppress special tokens breaks generation. The model starts emitting fragments like Action, assistant, and Human role markers, then hard-locks on prompts that trigger thinking pathways. The reason is that Qwen3.6’s tool-call thinking behavior operates at the model level, and the probability distribution over special tokens is part of its normal functioning.

2. Chain-of-thought introduces real latency costs

The official Qwen/Qwen3.6-35B-A3B-FP8 already emits roughly 130 additional thinking tokens per turn in tool-call scenarios, adding around 500ms of latency. The abliteration version inherits this behavior—this is not introduced by quantization or abliteration, but by the underlying design of the Qwen3.6 series itself.

3. MoE routing side effects

Qwen3.6-35B-A3B is a sparse MoE model with 35B total parameters and 3B active parameters. The abliteration process uniformly modifies relevant weights across all experts. So far, nobody has reported significant routing distribution shifts, but grimjim also notes in the README that certain experts may become abnormally activated on long-tail prompts. Running your own routing traces in production environments is recommended.

What Does This Mean?

When Arditi’s paper came out two years ago, the community reaction was essentially: “Oh, alignment can apparently be encoded as a low-rank direction.” At the time, most people viewed it as an interesting mechanistic interpretability result.

Two years later, the discovery has been engineered to the point where “a single researcher can perform precise behavioral editing on a 35B model over a weekend, with almost no benchmark loss.” This raises a fairly brutal question about the practical robustness of alignment defenses: if model weights are open-source, and the target behavior is clearly defined, then alignment mechanisms based on residual stream directions are almost defenseless.

On the other hand, this is also one of the most concrete industrial-grade applications of mechanistic interpretability so far. The same techniques can be used to:

• Precisely remove specific bias patterns from a model
• Strengthen or weaken certain stylistic tendencies (for example, making the model less verbose)
• Perform domain-specific behavioral tuning without rerunning SFT/RLHF

Names like Heretic and Aggressive sound very “community-driven,” but the underlying technology is far more serious than the branding suggests.

Deployment Options

Several distribution channels are currently circulating in the community:

• HuggingFace: llmfan46/Qwen3.6-35B-A3B-uncensored-heretic (BF16 original), protoLabsAI/Qwen3.6-35B-A3B-uncensored-heretic-FP8 (official FP8 format aligned with Qwen’s official quantization)
• Ollama: fredrezones55/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive, providing a full GGUF lineup from IQ3_M (15GB) to Q8_K_P (44GB), including a vision patch
• vLLM deployment: the FP8 version can be launched directly with vllm serve protoLabsAI/Qwen3.6-35B-A3B-uncensored-heretic-FP8

If you just want to experiment, a single 24GB RTX 4090 with IQ4_XS quantization is sufficient. For production, the FP8 version delivers throughput on H100/H200 GPUs that is nearly identical to the official FP8 release.

The officially recommended inference parameters are temperature=1.0, top_p=1.0, top_k=40, min_p=0, presence_penalty=2.0—the presence_penalty is raised to 2.0 to counter occasional repetition loops in MoE models, and this setting also works well on the original Qwen3.6.

Final Thoughts

Qwen3.6-35B-A3B is already a strong model in its own right. Its sparse MoE design with 35B total / 3B active parameters, native 262K context window, vision-language capabilities comparable to Claude Sonnet 4.5, and Apache 2.0 license already made it one of the strongest open-source base models available.

Now it also gains value as a testing ground for “mechanistic interpretability + behavioral editing.” No matter which side you’re on—studying attack surfaces or studying defenses—this is a project worth spending time examining at the weight-diff level.

As a side note, the official Qwen3.6 series is also directly accessible through OpenAI Hub via a unified API interface alongside GPT and Claude. If you want to use the official version as a baseline comparison, there’s no need to spin up your own inference service.

References

• Reddit r/MachineLearning: Norm-preserving abliteration on Qwen3.6-35B-A3B — Original technical post by grimjim, including method explanations and open-source links
• Reddit r/LocalLLaMA: Discussion of the Qwen3.6-35B-A3B Uncensored Aggressive release — Community discussion of the Aggressive variant, including K_P quantization information
• HuggingFace: protoLabsAI/Qwen3.6-35B-A3B-uncensored-heretic-FP8 — Full model card for the FP8 quantized version, including benchmark comparisons and deployment notes