Is AI wrong half the time? Sometimes. That number needs a trial of its own.

Stated at its strongest: verification is not text generation. A fact-checker interrogates primary and offline sources, calls people, reads documents that were never digitised, weighs context and legal exposure, and decides what "true" even means for an ambiguous claim. Systems that answer from statistical patterns over text, plus whatever a search index returns, are not doing that job — and in tests they produce error rates that would end a human checker's career.

Two distinct things. First, a hands-on trial: asked to verify claims, the assistants mostly produced methodology — here's how you would check this — instead of finishing the task. That is a real and reproducible observation about product behaviour, and it matches the incentives of assistants tuned to be helpful and cautious rather than conclusive.

Second, citations to studies reporting high error rates. Each of those studies is presumably accurate about what it measured. The article's move — and it is the move nearly every downstream summary repeats — is to let heterogeneous results imply one background fact about AI reliability.

The vendor reply is that these evaluations often disable the tools that matter: no web search, no retrieval, no citations, no iteration. Run with search enabled and a prompt demanding sources, they'd argue, the numbers change substantially.

That reply is partly right and mostly beside the point. It does not touch the article's central claim, which is about professional verification, not about lookup. Retrieval helps with "what did the report say"; it does nothing for "is the person on the phone telling the truth." And per rule two, a vendor's own reliability measurement is testimony, not a ruling.

Two independent lines of evidence show how much the number moves with the setup.

Evaluation design changes behaviour. Work published in Nature makes the point that scoring models purely on answer accuracy rewards guessing: a model that says "I don't know" scores zero, a model that guesses sometimes scores. Hallucination rates are therefore partly a property of the exam, not only of the student.

Format and retrieval dominate. A study of commercial chatbots as news intermediaries put roughly 2,100 same-day news questions to six chatbots. Strong systems could exceed 90% on some multiple-choice setups yet fall materially on free-response versions of comparable material — and most errors traced back to retrieval failures rather than reasoning failures.

That is the whole argument in one place. A 2% figure, a 45% figure, a 60% figure and a 90% figure can each be honestly measured, and none of them describes "how often AI is wrong" for you, on your questions, in your product, with your tools switched on.

The warning is right; a single "AI error rate" is not. AI is not a substitute for professional fact-checking — especially where offline sources, human interviews, legal or ethical judgement, or genuinely ambiguous claims are involved. We concede that fully and without hedging.

We challenge only the compression: collapsing incompatible tests into an implied universal reliability figure. Error rates vary dramatically with task difficulty, evaluation format, web and retrieval access, prompt design, and whether abstention is rewarded or punished.

Treat an assistant as a research accelerator that surfaces sources and candidate explanations, then verify anything high-stakes against the primary material yourself. Some concrete habits:

• Ask for sources and open them. An uninspected citation is not evidence.
• Prefer free-response answers you can check over confident summaries you cannot.
• Say explicitly that "I don't know" is an acceptable answer — you are removing the incentive to guess.
• For anything legal, medical, financial or reputational, the AI's job is to find the document, not to be the document.

Retrieval is doing more of the work than most people assume — which is exactly why Case 004, on poisoning what AI retrieves, is the natural next read.