The em dash doesn't give away a neural network. I checked on a corpus

2026-09-11Русская версия

I measured almost everything that's considered a sign of machine text on 7500 texts. Half of the signs don't work, some work backwards, and my own detector gave the agent's texts higher scores than mine.

Everyone knows the em dash gives away a neural network

It’s the first thing people say about machine text. The em dash, — instead of a comma. It’s item one on every list of signs, every humanizer scrubs it out, and in my tool it was hard ban number one too.

I took the LLMTrace corpus - 2708 human and 1455 machine texts in Russian, generated by gpt-4, o3, gigachat, Qwen2.5 - and counted the frequency.

em dash, Russian:
  human     794.7 per 100 thousand words
  machine   749.6

Humans use it more. The ratio is 0.94, so the sign works against us: a text with dashes is slightly more likely to be written by a living person.

In English the same dash gives 1.73, and there is a signal. One character, two languages, opposite conclusions.

What else turned out backwards

Then it got more interesting. I ran my whole rule set over the corpus, and here are the rules with a ratio below one:

A word on the Latin letters. Swapping letters is supposed to be an attempt to fool a detector: a Cyrillic “о” gets replaced with a Latin “o”. Humans do this three times as often as machines, because it’s an ordinary typo when switching keyboard layouts. People miss, models don’t.

A third of my English set worked the other way round. I kept those rules, but with the number attached: they no longer get weight and don’t count as a sign of generation. They stay as style advice - “in order to” is still worth shortening to “to”, it’s just not a machine trace.

What works

Not everything turned out to be junk. Some signs have a huge gap.

Emoji. In 2708 human texts - zero. Machines have 19 per hundred thousand words. The only sign that separates perfectly.

Individual words. Here I got lucky: Wikipedia has a “Signs of AI writing” page, edited by people, and half of the open humanizers rest on it. I took the words I didn’t have from there and measured them:

Before that my best result was seamlessly at 21.0. One Wikipedia page beat everything I had collected by hand.

Nominalization. The density of verbal nouns: “implementation”, “introduction”, “provision”. More on it below, because it has its own story.

My detector gave the agent’s texts the top scores

The tool started working, and I ran it over my blog. Eleven posts, the last three written entirely by an agent, the rest by me.

010_spec_driven       99/100 [clean]     agent
011_proxy_bench       88/100 [clean]     agent
012_local_llm_bench  100/100 [clean]     agent

007_brain_switch      60/100 [edit]      me
001_mermaid_ru        72/100 [edit]      me
004_death_or_alive    75/100 [edit]      me

The separation is reversed. A tool written to catch neural network traces gave the neural network the top scores, and the author “needs editing”.

My posts are full of phrases like “there are fewer and fewer generalists in the modern world”, “Let’s take a look at how it works”, “kills not only your productivity but also you”. These aren’t machine traces. They’re clichés, and they’re mine.

The agent, meanwhile, wrote under my CLAUDE.md, which bans the em dash, “not just X, but Y”, officialese and wordiness.

The gap comes from instructions, not from the neural network

The mechanism is explained by a PNAS paper on Biber’s 66 features. Parallel corpora of 66 thousand fragments, six models, human versus model pairwise - accuracy of 93-98 percent. But the interesting part isn’t the accuracy, it’s one column:

The number is the model’s frequency relative to the human one. For the base Llama everything is around a hundred percent. The base model writes like a human.

The gap is created by instruction tuning, not by the neural network as such. Two consequences follow at once.

First: humanizers work. Not because they trick the detector, but because they cancel the mechanism the detector catches. My agent didn’t bypass the check - it wrote under an instruction that suppresses exactly those features.

Second: a detector built on words and phrases is doomed. Any instruction like “write shorter and use verbs” wipes out the signal entirely, and that’s one line in a prompt.

It’s register, not authorship

One question remained: maybe the tool measures nothing at all. I took a second sample - 45 SEO articles about project management, written entirely by an agent.

A separation appeared, and a decent one: two and a half times. But it’s not about authorship.

The proof is inside the same data. The three agent posts in my blog gave a nominalization of 1.0-2.4, that is lower than mine. The same agent author, two registers, opposite results.

Corporate text about planning and implementation is stuffed with verbal nouns no matter who wrote it. The tool measures officialese - measures it honestly, the numbers add up. The question “who wrote it” it doesn’t answer and won’t.

So if you need an AI detector, you can’t build one from regexes. An officialese meter you can, and it’s more useful: officialese is worth removing no matter who wrote it.

Half a million comments

The second half of the tool is code comments. The same rules plus checks a regex can’t express: retelling the code, commented-out code, banners of equals signs, TODO without an owner.

The first run on our platform gave 502 thousand comments across nine thousand files. The number is absurd: fifty comments per file doesn’t happen.

Going down from there. 62 thousand blocks came from built JS in lib/ and bundle/ - I stopped walking the tree myself and started asking git ls-files for the list, which respects .gitignore exactly. Another 4474 hits were license headers nobody ever edits.

What remained was 24 thousand comments and 12 seconds instead of 82.

Commented-out code was a separate story. I caught it with a regex, then rewrote it after go-critic: the comment body is put into a function and handed to go/parser. If it parses, it’s code. The counter jumped from 52 to 168, and it looked like a success until I opened the sample:

pg := newFakePostgres() // the queue is empty, the worker should just wait
const maxMsgSize = 1024 * 1024 * 50 // 50 MB

The bug wasn’t in the detector. Blocks were cut by whole lines, so for a trailing comment the code before // got into the text. The parser honestly saw code and honestly fired. I switched collection to byte offsets, and it became 13 hits instead of 168 - almost all real.

And there’s no TypeScript parser in the stdlib

Go was lucky: go/parser is in the standard library. For TypeScript and Svelte a regex remained, and it missed the obvious - it didn’t count // export let attributeKey: string | undefined as code.

The obvious candidate is typescript-go, the native port of the compiler. Its parser lives under internal/, so the only way to use it is vendoring: put the source inside your own tree and go through an adapter. It works, but it costs a megabyte and a half of someone else’s code and maintaining a fork of an archived repository - while my whole tool takes a hundred kilobytes.

I took esbuild: public API, MIT, one go get line. api.Transform parses the comment body; no errors means code.

commented-out code in TS and Svelte:
  regex     76
  parser   187

Of the hundred and eleven new ones I checked the twenty-two that looked most like prose. False positives - zero. The binary grew from 3.8 to 8.9 megabytes, speed didn’t change.

The agent slipped through a hole in my rule

When everything worked, I set an agent on the tool’s own code: here’s the list of findings, here are the editing rules, fix it. It fixed eight comments out of twelve. Two came out like this:

// commentedOutCode: Go is parsed with a real parser, like go-critic; other languages get a character heuristic, since the stdlib has no parser.

A hundred and forty characters on one line. The rule required no more than two lines - the rule is satisfied. It became harder to read than before.

It wasn’t cheating. It did exactly what was written: lines were counted, not text, and gluing three lines into one formally solved the task. The hole was mine.

I added a line length limit. The new check immediately caught both comments the agent had just written.

Along the way it raised two objections to the rules, and both were right. The TODO rule fired on a line that quotes "// TODO: fix" as an example inside the checker itself. And a block with the mandatory note about the heuristic’s ceiling physically didn’t fit into two lines, because the note itself took two.

What I took away

A rule without a measurement is a matter of taste. The em dash seemed an obvious sign to me. Ratio 0.94. Latin letters inside a word seemed like homoglyph substitution. Ratio 0.28, these are typos by living people. I was sure of both.

Check other people’s numbers on your own data. I carried over the corpus weights from someone else’s skill and thought that was enough. My own measurement changed half of them and sent four rules below zero.

An absurd number in a report is your bug, not a property of the data. Half a million comments, 168 pieces of commented-out code. Each fell apart as soon as I opened the sample and read a dozen lines with my own eyes. The temptation isn’t to believe it, but to declare it a property of a big repository and tweak the threshold.

A counter that grew after a fix deserves a separate check. Replacing the regex with a real parser tripled the hits. It looked like a success, and in fact the parser highlighted a bug in my own collector.

A metric that can be met literally will be met literally. The agent met the two-line limit by gluing the text into one line of a hundred and forty characters. Any rule that counts form instead of content will one day be satisfied formally by someone - and it won’t be malice, but following the instruction to the letter.

A tool does what it measures, not what’s written on the box. I was writing a neural text detector and got an officialese meter. A useful thing, but it has to be called by its name, otherwise the first honest test will make a fool of you.

The tool lives at haiodo/hum1izer. Rules are in YAML, they’re extended, and you can swap in your own set with a flag. The score there is the tool’s rules, not an AI probability and not a verdict on authorship. It’s needed for one thing: comparing “before” and “after” on the same text.

This post went through it too, in its Russian original. 54 out of 100, the “rewrite” band.

Four hard bans, and all four are in quotes: I quote my own sins from old posts and rule names. The tool can’t tell a quote from my own speech.

There were two real findings, both mine, both fixed. Rewriting the examples so the counter drops to zero would take ten minutes. Only then the post would lose exactly what it was written for.