Anthropic's 'Watermark' Text Adulteration in Claude Is a Perversion of Writing

Then bad news: LLMs already use randomness in a fundamental way. Each time they go to generate a token, they first generate a probability distribution of possible tokens. Then they pick one randomly according to this distribution. The technique described can be thought of as making the random number generator pseudo random. The output it generates is one of the possible outputs it would have generated before, just now it's deterministic and will generate the same thing every time.

Does the author think he is currently getting T=0 output from Claude? Is he under the impression that T=0 produces the "best" writing?

This entire article just seems so detached from the basics of how LLMs work.

I think that was intended, yes.

Then why are you using an LLM to write? They're not capable of understanding such nuance. They do pick randomly between two synonymous phrases, they do not use some super smart algorithm to pick the one that sounds the best.

This excuse doesn't hold any water at all - Occam's razor says the author is just super annoyed that his AI writing will be identifiable as AI writing.

I honestly can't stand the way Claude writes. This watermark change just makes it scarier.

So, you don't own the generated text, and can't use it freely then. What if I copy paste a section, or rewrite a section of text to my liking? What if I rewrite some lines of code that contains the mark?

Security theater, and vague enough to be used as a weapon against who the government wishes.

I hope it's left off for non-EU customers.

At the point that you're generating entire volumes of text from Claude you're not really trying to be a sophisticated writer. I don't see how it's going to hurt for it to choose random related words.

Who could have seen this coming???

The author has expressed a preference. Assume that there is a sequence of tokens, such that it is considered the absolute best by the author. This particular method of watermarking makes it less likely to generate that sequence, by definition.

I feel their argument would have been clearer and stronger if they had spent more time exploring the alternatives, and whether these alternatives would be just as effective. It is trivially easy to remove invisible tokens.

Like it or not, there is a public good to being able to identify AI generated content, and a small degredation in quality is tolerable in my opinion.

I don't think anybody has to worry about this issue though. Manual writing, coding, and proof reading continues to be an option. Where AI output is nothing to be ashamed of, the tools are available. For everyone else, there will be LLM providers that ignore EU law.

Reminds me of printer tracking dots.

https://en.wikipedia.org/wiki/Printer_tracking_dots

LLMs have never been the place I've thought to expect any commitment to the craft of writing, to be fair.

Try running an llm like qwen 3.8 27B in Q8 locally with an intentionally very low temperature setting, it will write like a caveman crossed with a robot. You may find that an extremely literal output does not look pleasant to read for humans.

Well, akshwally...

> Interoperability. Providers must implement an interoperability solution for watermark detection such as a standardized API access method, a publicly readable signpost mechanism embedded in content, or participation in a consortium detection solution by February 2, 2027

Either they false positve on pretty much everything ever written, or the chances of catching a true positive is so low as to be useless.

Basically Cinavia for text, and that often falls over and is easy to remove even when there is megabytes of data streaming over a long period of time rather than 2 or 3 bits per wall of text, let alone what most people use claude for, when there is a strict dictionary and other tight output constraints.

The "watermark" can be trivially defeated, but may be enough to satisfy the letter of the law, and like many people here, I would argue that if you are letting Claude write for you, you've already accepted getting the literary equivalent of turd soup, so the harm is — or at least could be — fairly minuscule.

[1]: https://digital-strategy.ec.europa.eu/en/policies/code-pract...

(FWIW I have a more favorable view than most people seem to of the EU's efforts to at least try tackle problems like this — but predictably, the bureaucratic "solutions" they come up with don't work, but do make things objectively worse)

Watermarking is bad not just because of the principled stance that your tool should not be working against your own interests (the passionate argument in TFA), but specifically because it lends credence to the idea that AI detection is a valid and possible thing to do perfectly.

As technologists of course we know "oh well yes but with some confidence interval we can detect AI token bias across a large corpus of text." To JimBob in charge of publishing your paper or reviewing your PhD submission, all he knows is "anthropic says AI detection is possible so this 30% chance your paper was written by AI means you've plagiarized." Do you really think you're winning the argument with the certified, law-approved plagiarism detection machine? No, you're not, and your career is over.

It's irresponsible to develop watermarking because it is not anywhere close to a perfect science, but it will be treated like one by people with the power to ruin your lives. Even if you've never touched AI in your life, your paper is going through the "maybe it says you cheated" box, and you better hope those dice don't come up snake eyes.

And what I mean by that is that companies that are at the top tend to make anti customer decisions because they have lost the concept that pleasing customers matters as priority one.

Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic announcement relies on. It becomes more difficult to reliably detect text as a fixed sample count without tweaking the distribution further. Either way, it's a minor problem that will be addressed over time, compared to the issue of who can detect this without guessing or developing their own sets: providers not releasing a way to detect any such watermarks without going through them makes this entire approach hostile to the public. The EU regulation on this subject is interesting, although again most certainly not the primary driver for these practices:

"1.1.2: Signatories will ensure that AI-generated or manipulated content is marked with an imperceptible watermark, with the exception of very short text. For free-form text longer than 200 tokens, watermarking still needs to be applied, even though it may have lower reliability compared to that of watermarking very long text"

A proper, effective and useful law would have required providers to regularly release datasets to run your own verification on any text released within a fixed interval of time, presumably once out of rotation. Instead, it only talks about exposing an user interface going through their own services:

"Signatories will ensure access to their detection solution through a user interface appropriate for the audience of end-users that may eventually be exposed to the content generated or manipulated by their AI system. [...] Any restriction to the access will be limited in time until more reliable and robust detection mechanisms have emerged and have been adopted as the state of the art for detection mechanisms for the watermarking of free-form text evolves."

Most interestingly, in line with the EU's mass-surveillance program, an alternative solution to watermarking where it may not be sufficient is also suggested, although only optional for now:

"Where appropriate and taking into account potential trade-offs related to privacy and security, as well as scalability challenges and costs, Signatories may implement as an optional supplementary measure fingerprinting or logging solutions for AI-generated or manipulated content which allow for checking whether content has been generated or manipulated by their AI system. For example, direct logging may be appropriate for text content, whereas fingerprinting approaches may be preferable for audio and visual content."

“Absurdly and insultingly”? Come on…

How Claude's text watermarking works

https://news.ycombinator.com/item?id=49303350

> The difference between watermarked and un-watermarked text will not be distinguishable to readers

https://www.anthropic.com/news/claude-text-watermark

Which is to say, it does not actually meet the EU AI act requirements which require transparency to humans. Not to mention that if the detection requires access to the base models, it makes anthropic the only entity who gets the say on if a piece of text comes out of Claude. Anthropic is both the player and the referee here.

If there is one takeaway you should have from this fiasco it is that you should be wary of using tools that doesn't serve your needs and your needs only.

I struggle to see how this could possibly be useful unless there's some sort of psy-op going on to trick people into uncritically accepting anything lacking a watermark as not being AI-generated.

Yeah, John. We're all OK with that.

It already fails. It randomly picks between close candidates. To help fool people into believing in intelligence claim, I guess.

Nanny state nonsense indeed.

As a non American/non European resident all I see from the Europeans are 0 contributions to software progress at any large scale while they surely do a lot of crying and huffing and puffing and demanding. Lots of complaining and rule making but not a lot of creating is a bad look.

FITFY.

I have no sympathy for writers whining about what the AI is doing to 'their' writing. It's only your writing when you write it. There's any easy way to avoid this: don't fucking use it. Use you own brain.

... and the opposite is also true, sometimes it will increase the probability of choosing the "best" word choice. So watermarking makes the LLM quality better then? /s