“Evaluating honesty and lie detection techniques on a diverse suite of dishonest models” by Sam Marks, Johannes Treutlein, evhub, Fabien Roger

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded26 Nov 2025

Publisherfeeds.type3.audio ↗

Handlerec:12178257

Show notes

TL;DR: We use a suite of testbed settings where models lie—i.e. generate statements they believe to be false—to evaluate honesty and lie detection techniques. The best techniques we studied involved fine-tuning on generic anti-deception data and using prompts that encourage honesty. Read the full Anthropic Alignment Science blog post and the X thread. Introduction: Suppose we had a “truth serum for AIs”: a technique that reliably transforms a language model $M$ into an honest model $M_H$ that generates text which is truthful to the best of its own knowledge. How useful would this discovery be for AI safety? We believe it would be a major boon. Most obviously, we could deploy $M_H$ in place of $M$. Or, if our “truth serum” caused side-effects that limited $M_H$'s commercial value (like capabilities degradation or refusal to engage in harmless fictional roleplay), $M_H$ could still be used by AI developers as a tool for ensuring $M$'s safety. For example, we could use $M_H$ to audit $M$ for alignment pre-deployment. More ambitiously (and speculatively), while training $M$, we could leverage $M_H$ for oversight by incorporating $M_H$'s honest assessment when assigning rewards. Generally, we could hope to use $M_H$ to detect [...] --- First published: November 25th, 2025 Source: https://www.lesswrong.com/posts/9f7JmoaMfwymgsW9S/evaluating-honesty-and-lie-detection-techniques-on-a-diverse --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.