You can extract the fine-tuning data just by looking at the logits — CDD black-box attacks are pretty ruthless.

Bad News: Your Finetuned Private Data Might Not Be as Private as You Think

In early July, a rather modestly titled post on Reddit’s r/MachineLearning stirred up the research community. The author released a model diffing method called Contrastive Decoding Diffing (CDD). With only gray-box access to logits—no weights, no activation values, not even any probe corpus—it can reconstruct the finetuning data verbatim from a narrow-domain finetuned LLM.

The keyword is verbatim. Not “it can roughly guess what domain you finetuned on,” but literally outputting the exact sentences that appeared in the training data.

Across evaluations involving 4 model families and 20 organism×model combinations, CDD achieved a reconstruction score of 4+/5 using a single default configuration—no per-sample tuning, no layer selection—succeeding on 19 out of 20 tests. Model sizes ranged from 1B to 32B. This level of robustness is almost counterintuitive in the model diffing field.

How It Works

To understand CDD’s idea, we need to set it alongside its predecessor—the one it aims to “outclass via dimensionality reduction.”

Early this year, Minder, Dumas, and others introduced a method called Activation Difference Lens (ADL) in their paper “Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences.” Their core observation: after a base model undergoes narrow-domain finetuning, the activation differences between the base and the finetuned model in certain layers contain clearly readable “traces.” ADL exploited these differences for steering, allowing inference of themes, domains, and approximate content of the finetuning data.

But ADL had two major flaws:

• White-box access required: You need complete access to both models’ weights and intermediate activations, which is impossible in API scenarios.
• Only retrieves domain info: It can tell you “this model seems finetuned on medical Q&A,” not the actual training sentences.

CDD, essentially, relocates ADL’s concept from activation space to output space, while minimizing complexity:

1. Run the same prompt on the base model and the finetuned model;
2. Align their logits token by token and perform contrastive decoding;
3. The amplified parts of the distribution indicate knowledge reinforced during finetuning;
4. Sampling from the contrastive distribution produces an approximate reconstruction of the finetuning corpus.

This process is mathematically analogous to the classic 2022 paper by Li et al. on Contrastive Decoding and the 2023 arXiv work “Contrastive Decoding Improves Reasoning in LLMs”—both operate on the principle of subtracting an amateur distribution from an expert one. The difference: the former improves generation quality, while CDD repurposes it as an audit tool—using the finetuned model minus the base model to expose memorized content.

It’s a neat perspective shift: the same formula can either make a model “speak better” or “confess.”

Why Developers Should Care

In short: if you’re finetuning private data into a model and then exposing that model through an API, you now need to reassess your threat model.

Previously, privacy attacks mostly fell into two classes:

• Membership Inference Attacks (MIA): Determine whether a specific data instance was used in training—just a yes/no.
• Extraction Attacks: Try to prompt the model into regurgitating training data—success rate highly variable, depending on overfitting and prompt craft.

CDD belongs to a third category, and its barrier to entry is shockingly low:

• No weights from the base model required—as long as it’s open source (Llama, Qwen, DeepSeek bases, etc.), anyone can run it locally;
• No weights from the target finetuned model required—just API access that returns top-k logits or logprobs, which OpenAI, Anthropic, and most open deployments already do;
• No probe corpus required—no need to guess the finetuning domain first to craft prompts;
• No per-model tuning required—a single default config works across four model families.

In other words, if an attacker knows which open-source base your model was finetuned from, they can recover your finetuning data for just tens of dollars in inference cost.

Many vertical SaaS teams should be worrying now: legal documents, medical records, internal company knowledge bases—if you’ve gone the finetuning route, you’re exposed to this attack surface.

The 1 Exception Out of 20 Might Be the Most Interesting Part

The author mentioned achieving 4+/5 in 19 out of 20 combos, but didn’t specify the exception. Likely possibilities include:

• The finetuning dataset was large and broad, leaving no clear narrow-domain drift, diluting the contrastive signal;
• Finetuning used RLHF or DPO rather than SFT, meaning the logits reflect preference differences rather than memory differences;
• Continued pretraining occurred between the base and finetuned models, breaking continuity between the two states.

If any of these hold true, CDD’s effective scope is likely narrow-domain SFT finetuning—consistent with ADL’s premise (its title literally says “narrow finetuning”).

For defenders, this leads to several practical suggestions:

• Mix in a sufficient proportion of general corpus (rehearsal) to dilute domain-specific signals;
• Don’t expose logits/logprobs, only return sampled tokens; this reduces CDD’s accuracy but also worsens user experience;
• Consider LoRA with added regularization before weight merging, to smooth output-space differences;
• Or simply use RAG, keeping private data in a retrievable index rather than baked into weights.

That last point is an old refrain—but with CDD, the balance between RAG and finetuning tilts further toward RAG.

From Research to Product Impact

Viewed in the larger picture, from late 2025 to early 2026, the model diffing subfield is clearly accelerating:

| Method | Access Type | Reconstruction Granularity | Applicable Scope | |---|---|---|---| | ADL (Activation Differences) | White-box | Domain/Theme | Requires full weights | | Traditional Membership Inference | Black-box | Yes/No | Single data entries | | CDD (this work) | Gray-box logits | Verbatim reconstruction | Narrow-domain SFT finetuning |

Granularity is increasing, access requirements dropping. That’s good news for researchers—but bad news for compliance teams.

One point worth noting: as of now, CDD only exists as a Reddit post—no official arXiv paper yet, open-sourcing status unknown. But the method is simple in essence—subtracting two models’ logits and resampling—so reproducing it is easy. Expect follow-up reproduction and attack-defense work in the coming weeks.

Direct takeaways for developers:

• If you provide closed-source finetuning services, audit whether your API exposes logprobs;
• If you run private finetuned deployments of open models, consider restricting logits output;
• If you conduct model safety evaluations, CDD is a very low-cost red-teaming tool worth integrating into your pipeline.

Incidentally, major closed models aggregated on OpenAI Hub (GPT, Claude, Gemini, etc.) differ in their logprob return policies—Anthropic never provides them, OpenAI does but with limits, Gemini partially does. That difference now takes on new meaning: it’s also a potential line of defense against CDD-type attacks.

A Bigger Question

CDD’s technical essence turns “difference between two models” from an abstract semantic notion into a precisely sample-able distribution. Could this technique be applied in reverse? For example:

• Model provenance: given a black-box API, use CDD to backtrack which open base model it derives from;
• Knowledge distillation audit: measure how much the student copied from the teacher;
• Version diffing: detect what changed between GPT-4 and GPT-4.5—CDD might highlight clues.

Any of these successful applications could redefine what “intellectual property boundaries” mean for models.

In an era when all major models are aggregated via APIs (as platforms like OpenAI Hub exemplify), logits are effectively a model’s second skin. CDD reminds us: that skin may be far more transparent than we imagined.

References

• Contrastive Decoding Diffing (CDD): recovering verbatim finetuning data from logits alone - Reddit — The author’s original post on r/MachineLearning, including experimental setup and 20 test combinations.
• AI Self-Improvement Capabilities: From Decoding to Reasoning - Zhihu — A Chinese overview of the Contrastive Decoding family of methods, helpful for understanding CDD’s technical lineage.