Today we released version 1.0.0 of two open-source tools for our research on AI-generated news: JudgeGPT and RogueGPT. With this release, the JudgeGPT survey officially opens for data collection, and its data structure is now stable.
Why two tools
In our survey Blessing or Curse?, Martin Kappes, Marc-Oliver Pahl, and I named the scarcity of comprehensive datasets as one of the main challenges for research on generative AI and fake news. To measure how people judge AI-generated news, you need two things: news fragments produced under known conditions, and a way to collect judgments on them. RogueGPT handles the first part, JudgeGPT the second.
Decisively Digital
Discover the power of artificial intelligence and digital transformation in the #1 best-selling business book Decisively Digital.
Buy on Amazon Learn MoreDespite the names, neither tool is tied to OpenAI. โGPTโ stands for large language models in general.
RogueGPT: producing the fragments
RogueGPT is a Streamlit application that generates short news fragments with large language models and saves them to a MongoDB database, together with the model and the prompt that produced each one. A configuration file defines the templates, styles, and components for each language.
Version 1.0.0 covers four languages (English, German, French, and Spanish) and three formats: tweet, headline, and short article. The generator configuration lists models from small open-weight ones such as Phi-3-mini, Gemma 7B, and Mistral 7B up to GPT-4o, o1, o3-mini, and DeepSeek-R1. A second tab takes human-written fragments with their outlet and URL, so the survey can mix both origins.
JudgeGPT: collecting the judgments
JudgeGPT shows participants fragments from that database. For each one, they rate two things separately:
- whether the text was written by a human or a machine, and
- whether it is legitimate news or fake news.
They also say how well they know the topic. Keeping source and authenticity apart is the point of the design: a fake story can be written by a person, and a machine-written text can be accurate.
Before the first fragment, participants answer a few questions about themselves, for example age, education, political view, and how familiar they are with fake news. After every five answers, they see how often they were right on each of the two questions. The survey runs in English, German, French, and Spanish. It detects the language automatically, and you can switch it in the app.
Take part
If you want to test your own judgment, the survey is at judgegpt.streamlit.app. An earlier version of the app served as the quiz in How to Spot Fake News This Election. The responses feed our next studies on how people perceive AI-generated news.
Open source
Both tools are on GitHub under the GNU GPLv3:
The README files explain how to run them locally. Issues, suggestions, and pull requests are welcome.
