″“Did you lie?” Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms” by Alan Cooney, David Africa, Geoffrey Irving

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded18 Jun 2026

Publisherfeeds.type3.audio ↗

Handlerec:18968269

Show notes

TL;DR. Lie detectors for LLMs could be valuable for auditing and monitoring. But evaluating them requires testbeds where the model verifiably believes the opposite of what it says, which isn’t straightforward. We determine that most existing trained model organisms don't clear this bar. We train 13 reasoning model organisms, with evidence they hold the alternative belief in chain-of-thought, as well as evidence that they have generalised out of distribution. We also build a broad prompted-lying testbed. On these, detectors scale positively with model capability when lying is prompted, but every activation- and logprob-based detector drops sharply when lying is trained in.Existing lie detectors are therefore not capable enough for making high-confidence claims about when models are lying, though they may still be useful as part of a broader toolkit. View our paper here and our datasets, model organisms and trained detectors here. Overview. We evaluate four lie detectors (a) — a chain-of-thought monitor, two activation probes (Apollo and our new Did-You-Lie probe), and a logprob classifier — in two regimes. In the first, subject models are prompted to lie: we evaluate on our Varied Deception testbed across 31 open-weight subject models spanning 2B–1000B parameters (b). In the [...] ---Outline:(00:16) TL;DR.(02:42) Two hopes for lie detection(04:03) The need for better model organisms of deception(09:14) Results The original text contained 1 footnote which was omitted from this narration. --- First published: June 17th, 2026 Source: https://www.lesswrong.com/posts/xFbMTnmDAucwE8Cdm/did-you-lie-evaluating-lie-detectors-across-model-scale-and --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.