Meta paper shows watermarks can expose distilled AI models

Copying a large language model used to require stealing its weights. Now you can arguably do it by asking it a lot of questions and training a smaller model on the answers.

Researchers at Meta think they have a better way to catch that. A paper from Meta’s FAIR and Superintelligence Labs introduces WM-Basin, a black-box technique that identifies models distilled from a watermarked teacher. In testing, it beat the prior state-of-the-art token-level detector by a wide margin.

What the paper actually found

The study, titled “Watermark Basins: Black-Box Detection of Distilled LLMs,” was published on October 27, 2026 under arXiv:2610.37266. Its focus is a specific scenario: someone queries a watermarked model through an API, collects the outputs, and trains a “student” model on them.

The headline result is a 79% true-positive rate at a false-positive rate of just 0.1%, using n-gram=1. The previous best black-box token-level detector managed a true-positive rate of only 29% under the same conditions.

According to the authors, the gap between the two methods widens further at stricter detection thresholds and with larger n-gram windows.

The evaluation was not small. The team tested 1,200 student models ranging from 135 million to 8.2 billion parameters. These students were trained on mathematical reasoning traces from OpenR1-Math, across six student architecture families, multiple teachers, and several watermarking schemes.

How WM-Basin works

WM-Basin feeds the suspect model’s generated text into a fixed, clean probe model and converts that text into gradient representations. Put simply, it asks how the text would push the probe model’s parameters if the probe were trained on it.

From there, the method separates ordinary fine-tuning directions from alignment with what the authors call a key-specific “watermark basin” in parameter space. The watermark leaves a directional signature, and WM-Basin looks for that direction rather than for individual telltale words.

A crucial practical detail: the tool only needs text outputs from the suspect model via API. It does not require access to weights or logits.

Where the signal holds, and where it breaks

The researchers report that the detection signal survives several common post-distillation modifications. These include quantization, which compresses a model to run cheaper; pruning, which strips out parameters; and reinforcement learning.

WM-Basin also proves more reliable when token-level signals fade, such as during low-temperature sampling.

The paper is candid about its limits, though. Extensive fine-tuning on clean data can degrade the signal. The authors flag continued supervised fine-tuning on unwatermarked data and paraphrasing the teacher’s traces as potential evasion vectors.

What this means for AI developers

The 0.1% false-positive rate is arguably the more important number for that use case. Any legal or contractual claim built on watermark evidence will be scrutinized for wrongful accusations. A method that holds a low false-alarm rate while catching 79% of copies is far more defensible than one catching 29%.

The evaluation centered on mathematical reasoning traces, so performance on other kinds of content is an open question from the data presented. Real-world copiers may also combine several evasion tactics at once, including the paraphrasing and clean-data fine-tuning the authors themselves identified.