“A system overview for near-term, low-trust AI compute verification” by Naci Cankaya

episode

LessWrong (30+ Karma) not yet transcribed

▲ 0

Transcript

This episode hasn't been transcribed yet Upvotes decide what gets transcribed next — 0 so far. Sign in to upvote

More from LessWrong (30+ Karma)

“Claude Opus 4.5 Achieves 50%-Time Horizon Of Around 4 hrs 49 Mins” by Michaël Trazzi 3 min · 20 Dec 2025

All episodes from LessWrong (30+ Karma) →

Report any issue

Source

Recorded23 Jun 2026

Publisherfeeds.type3.audio ↗

Handlerec:19035099

Show notes

Version 0.2, working draft This is a working draft of my current best idea for a privacy-preserving, retrofittable AI compute verification system, for confidence-building in an arms-control-like AI agreement between rival nation states. The purpose of this draft is to elicit community engagement by making use of Cunningham's law: I make assertions about what the (emerging) field of AI verification should aim for, and people with experience in international policy, cybersecurity and any relevant field of engineering can point out what this draft gets wrong. Thank you to everyone who has provided feedback to version 0.1, especially Aaron Scher, Mauricio Baker and Jonathan Ng. 1. Introduction and summary In order to plan and execute under tight timelines, one needs to make some strategic bets, instead of hedging too much and keeping all options open. The field of research on AI verification is bottlenecked partly by a lack of shared vision (as well as human capital, but having clear goals helps hiring and fundraising). With this post, I aim to: Make technical objectives for verification in high-stakes AI governance more specific and actionable (section 2).Contribute a first, high-level reference architecture for meeting these goals (section 3 and [...] ---Outline:(00:54) 1. Introduction and summary(06:31) 2. Problem statement and motivation(06:41) 2a. Low-trust AI governance(09:46) 2b. Threat model(11:09) Covert adversary and the inversion of the fortress problem(12:21) The attribution problem and plausible deniability(13:26) Assumptions about physical security and inspection(15:08) Discussion of attack surfaces(18:19) 2c. Practical requirements(23:05) 3. System overview and operation(23:10) 3.1. Brief introduction(27:14) 3.2. End-to-end execution trace(28:00) 3.2.1. Evidence capture(30:22) 3.2.2. Evidence evaluation(33:57) 4. Subsystem designs for eliminating the need for mutually trusted silicon(34:29) 4.1. Trust in silicon is hard(35:58) 4.2. Analog data movement control: passive splitters, data diodes, enclosures(37:52) 4.3. Building blocks for a mutually secure verification system(38:53) 4.3.1. Controlled ingress(40:02) 4.3.2. Output cross-checks(41:46) Prior work(43:01) 4.3.3. Sanitized egress(44:26) Prior work(45:19) 4.3.4. Instructor-executor(48:26) 5. Engineering approaches for evidence capture and evaluation(48:32) 5.1. Evidence generation, capture and commitment(50:29) 5.1.1. Network taps and active wardens(51:18) Prior work(54:03) Open research questions(55:55) 5.1.2. Memory challenging and memory wiping(58:19) Prior work(01:00:19) Open research questions(01:01:32) 5.2. Evidence evaluation and disclosure(01:01:37) 5.2.1. Secure auditing environments (tentative plan A)(01:04:20) Prior work(01:06:22) Open research questions(01:07:53) 5.2.2. Replay and the determinism challenge(01:10:10) Prior work(01:10:49) Open research questions(01:11:43) 5.2.3. Inspection software, inspector agents(01:12:38) Prior work(01:13:58) Open research questions(01:14:58) 5.2.4. Zero Knowledge Proofs (tentative plan B)(01:16:22) Prior work(01:18:55) Open research questions(01:20:14) 5.3. Support mechanisms(01:20:19) 5.3.1. Side-channel defense(01:20:51) Prior work(01:22:43) Open research questions(01:24:39) 5.3.2. Resource accounting(01:25:18) Prior work(01:25:30) Appeal to the reader(01:26:28) Appendices(01:26:32) A1. The statistics of random sampling The original text contained 23 footnotes which were omitted from this narration. --- First published: June 23rd, 2026 Source: https://www.lesswrong.com/posts/fgvmKqRGvBteKeDoc/a-system-overview-for-near-term-low-trust-ai-compute --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.