Can people tell whether a news text was written by a person or a language model? In our data, they cannot. Our poster paper “Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News”, with Martin Kappes and Marc-Oliver Pahl, is now published in the Companion Publication of the ACM Web Science Conference 2026 (WebSci ’26). The conference takes place in Braunschweig from May 26 to 29.
The setup
For each news fragment, JudgeGPT asks three questions on continuous sliders: written by a human or a machine, legitimate or fake, and how familiar the topic is. The machine-generated fragments come from RogueGPT and six models: GPT-4, GPT-3.5, GPT-4o, LLaMA-2 13B, Gemma 7B, and Mistral 7B. The analysis covers 2,318 judgments from 1,054 participants.
Decisively Digital
Discover the power of artificial intelligence and digital transformation in the #1 best-selling business book Decisively Digital.
Buy on Amazon Learn MoreFive findings
- No reliable distinction. Source scores for machine-generated and human-written text did not differ significantly (Welch’s t-test, p = .38).
- Across all models. The mean source scores of all six models fell between 0.44 and 0.55, around chance level. That includes the open-weight models with 7B parameters.
- Expertise, not politics. Self-reported familiarity with fake news predicted accuracy (r = .35). Political orientation did not (r = -.10, not significant).
- Two strategies. Clustering found “Skeptics,” who rate all content with low trust, and “Believers,” who start from high trust.
- Learning, then fatigue. Accuracy improved over the first 15 to 20 judgments and declined after about 30.
What follows
The paper draws three design implications:
- Provenance over detection. If readers cannot tell, the burden should not sit with them. Content credentials such as C2PA attach a verifiable history when content is created, and source-level signals such as CRED-1 add context about where a text comes from.
- Short inoculation. The early learning effect supports pre-bunking exercises, but fatigue limits them to short, focused sessions.
- Different users, different signals. Skeptics and Believers may need different trust indicators.
Limitations
The sample skews toward educated Europeans: 68% have a university degree, and 74% are from Europe. By design, about 98% of the stimuli are machine-generated, because the study compares models with each other. Political orientation was measured with a single item, so finding 3 needs a study with politically balanced stimuli.
Read the paper
- Paper (open access, CC BY 4.0): doi.org/10.1145/3795513.3807431
- Code: JudgeGPT and RogueGPT
Two more of our papers appear in the same proceedings: Origin Lens, on verifying image provenance on the device (more in my post on the Origin Lens app), and my PhD Symposium paper, The Indistinguishability Threshold.
