Can AI writing be detected? What the detectors actually measure

The company with the most to gain built one and switched it off

If AI-written text could be reliably detected, the organisation best placed to do it would be the one that built the model. OpenAI tried.

It launched an AI Text Classifier on 31 January 2023. In the announcement, it published its own accuracy figures, which is more than most detector vendors have ever done. The classifier correctly identified 26 percent of AI-written text as likely AI-written. It incorrectly labelled human-written text as AI-written 9 percent of the time.

On 20 July 2023, OpenAI withdrew it. The banner is still sitting on the original announcement: no longer available, due to its low rate of accuracy.

Read those numbers again in the order that matters. Three quarters of AI text walked straight past it. Nearly one in ten innocent people got flagged. That is not a tool that needs tuning. That is a tool measuring something other than what it claims to measure.

The false positives are not random

A detector that is wrong 9 percent of the time at random is a bad tool. A detector that is wrong about the same people every time is something worse than that.

In July 2023, Patterns, a Cell Press journal, published a paper by Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu and James Zou with a title that gives away the finding: GPT detectors are biased against non-native English writers.

They ran seven widely used GPT detectors over two sets of genuinely human writing: TOEFL essays written by non-native English speakers, and essays by American eighth-grade students.

The eighth-grade essays were classified accurately, close to perfectly.

The TOEFL essays were not. The average false-positive rate across the detectors was 61.3 percent. More than half of the essays, all written by real people, were called machine-generated. All seven detectors unanimously flagged 19.8 percent of them. At least one detector flagged 97.8 percent of them.

Then the researchers did the thing that settles the argument. They re-prompted the essays with richer vocabulary and ran them again. The average false-positive rate fell from 61.3 percent to 11.6 percent.

That single result tells you what the detectors are actually doing. They are not detecting authorship. They are scoring linguistic complexity, and calling anything plain-spoken machine-made. Which means they penalise exactly the people least able to argue back, and they can be defeated by asking for fancier words.

What institutions did once they looked at the numbers

Vanderbilt University disabled Turnitin's AI detector in August 2023 and published its reasoning, which is more useful than the decision itself because it shows the arithmetic.

Vanderbilt submitted around 75,000 papers to Turnitin in 2022. At Turnitin's own claimed false-positive rate of 1 percent, that implies roughly 750 papers wrongly flagged in a single year at a single university. The university also cited the bias against non-native English speakers, and noted that Turnitin would not explain its method beyond describing patterns common in AI writing. Its conclusion was that AI detection software is not an effective tool and should not be used.

Curtin University in Australia announced in September 2025 that from 1 January 2026 the AI writing detection feature in Turnitin would be disabled across all campuses and study periods. Ordinary text-matching originality checking stays on. Only the AI detection comes out.

You will find pages claiming that fifty or a hundred universities have done the same, usually with a tracker and a tidy number. Those numbers do not trace back to any authoritative count, and a striking proportion of the sites publishing them sell tools for making AI text undetectable. Two documented institutions and a clear direction of travel is what the evidence actually supports.

Watermarking is a real technology and still not the answer

The sophisticated version of the question is: never mind detectors, what about watermarking, where the model marks its own output?

Google DeepMind's SynthID Text does exactly that, by adjusting token probability scores during generation so the text carries an imperceptible signal. It is not a prototype. It has been applied to Gemini app output, and a production implementation shipped in Hugging Face Transformers, with reference code published openly.

Google's own documentation is admirably direct about where it stops working:

• It is less effective on short responses.
• It is less effective on factual prompts, because there is little room to vary word choice without hurting the answer.
• Confidence drops sharply when text is thoroughly rewritten or translated into another language.
• It is vulnerable to paraphrasing and word substitution.
• The detector returns three states, not two: watermarked, not watermarked, and uncertain.
• Google states that it is not designed to directly stop motivated adversaries.

And then the limit that ends the discussion for workplace use. A watermark can only be found in text produced by a model that put it there. Gemini output carries one. Output from ChatGPT, Claude, an open-weights model running on somebody's laptop, or a human being does not. The absence of a watermark tells you nothing at all.

Watermarking is a provenance tool for the company that owns the model. It was never a lie detector for your colleague's report.

So what actually gets noticed at work

Detection software is not what catches people. People catch people, and they do it on evidence that has nothing to do with statistics.

What gets noticed is a document that is fluent and empty. A confident paragraph about a client that gets a fact wrong that everyone in the room knows. A recommendation with no cost attached to it. A tone that does not match the four emails you sent last week. A report that reads as though it were written by somebody who had not been at the meeting, because it was.

None of that is a detector firing. It is a reader noticing that the thinking is missing, which is a much older skill than any of this.

Which points at the only durable answer. The question "will they be able to tell" is the wrong question to organise your work around, partly because the tools that claim to tell are not reliable, and mostly because the thing people actually notice is whether the work is good. Use the tool for the draft, the structure, the summary, the boring pass. Then supply the judgement that makes it worth reading, because that is the part no detector is needed to spot the absence of.

If the underlying worry is a workplace policy rather than a detector, the related questions are covered in can your employer see what you type into ChatGPT and what actually happens to the company data you paste into an AI tool.

The working version of this is in Using AI at Work.