The European Union’s push for AI transparency is hitting a unexpected snag. As platforms rush to comply with new regulations requiring content provenance, the very tools designed to label AI output are introducing security vulnerabilities. Anthropic recently announced that its future Claude models will adopt SynthID-Text, an open-source watermarking technique developed by Google. The method uses a secret key to subtly alter how models select words, swapping likely choices like “cloudy” for synonyms like “overcast.” This allows anyone with the key to verify if text was machine-generated. It sounds clean on paper. In practice, it’s messy.
New research from Lasso Security reveals that SynthID-Text does more than just tweak vocabulary. It fundamentally shifts how models behave under pressure. Andrea Siposova, an AI security researcher at Lasso Security, found that the watermarking process can weaken safety guardrails. When tested against adversarial prompts—attacks designed to trick models into revealing passwords or ignoring rules—the watermarked versions were more likely to comply with harmful requests than their unwatermarked counterparts. Siposova notes that while watermarking aims to be imperceptible to readers, “it is going to cause some tradeoffs, it’s going to show up somewhere.” That somewhere is often the model’s refusal mechanism.
The core issue lies in a technique called tournament sampling. Instead of picking the next word purely based on probability, SynthID pits token candidates against each other using hidden scores derived from the secret key. This competitive selection process introduces what researchers call “sampling drift.” It doesn’t just change what the model says; it changes what AI agents do. If a weakened refusal leads an agent to invoke the wrong tool with dangerous arguments, the consequences escalate quickly. The study tested six open-weight models via Hugging Face, not Claude specifically, but the implication is clear: altering the sampling logic alters the safety profile.
Developers can’t just flip a switch and assume compliance equals safety. The variance in behavior depending on the specific secret key used suggests that robust red-teaming is no longer optional. Companies must stress-test their platforms with watermarking active to ensure that the drive for provenance doesn’t open the door to prompt injection attacks. We are trading subtle linguistic fingerprints for potential structural weaknesses. The industry needs to decide if that trade-off is worth the risk before rolling these features out at scale.
