Roman Yampolskiy vs Emad Mostaque: I Was The Only Optimist

The man who open-sourced the most-used AI image model in history sat down with the man who has spent a decade proving superintelligence cannot be controlled. Brian set up a debate. What emerged was something more unsettling than any debate.

Roman Yampolskiy is the computer scientist who coined the term AI safety and author of AI: Unexplainable, Unpredictable, Uncontrollable. Emad Mostaque is the co-founder of Stable Diffusion and the only AI CEO who signed the pause letter. He now says he doesn’t know how a pause could work. Yampolskiy thinks that is the only option left.

The question underneath everything is simple: if you have a 50% chance of wiping out civilization and you build it anyway, what are you actually doing? We cover what AI safety researchers actually think the danger is, why nobody has published a paper, filed a patent, or shipped a prototype for controlling a superintelligence, what the Qwen weights being out means for the pause argument, and why Mostaque thinks swarm intelligence is the most dangerous and most unpredictable risk vector we have.

What you’ll hear:
-Why both guests think P(doom) tells you less than you’d hope
-What it means that no company, no lab, and no team has a patent on controlling superintelligence
-Why the models the public receives are slightly lobotomized
-The difference between an AI swarm and the ASI everyone is debating
-Why giving every psychopath access to a cutting-edge intelligence weapon is incoherent safety strategy
-What it would actually take to update Yampolskiy’s assessment

“We either do it, or we die. There is nothing for you to gain by doing it.” — Roman Yampolskiy

CHAPTERS
00:00 A debate that wasn't a debate
00:36 Turing test, AGI, superintelligence: where are we?
02:02 The open source argument nobody wins
03:32 Pause frontier AI forever. Which button?
05:04 P(doom): parameterizing our ignorance
09:02 Nukes are inefficient. AI isn't.
11:46 The lobotomized model problem
18:04 Decade-old problems solved weekly now
26:18 We either do it or we die
31:10 Stop the training or stop the funders
33:34 Lipstick on a Shoggoth
35:50 No paper. No patent. No framework.
39:42 Train only on what you need
44:14 What lowers Mostaque's p(doom)?
52:00 Same future. Two perspectives.

Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt

Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu

Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1

Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating

Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join

Featured Guests:
Roman Yampolskiy on Twitter/X: https://x.com/romanyam?lang=en
AI: Unexplainable, Unpredictable, Uncontrollable (book): https://www.amazon.com/dp/103257626X
MIRI: https://intelligence.org

Emad Mostaque on Twitter/X: https://x.com/EMostaque
I.I.I. Inc.: https://ii.inc/
Stable Diffusion: https://stability.ai/

My books:
Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA
Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu
Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U
Galileo’s Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un

Twitter/X: https://x.com/BrianKeating
Substack: https://briankeating.substack.com
Blog: https://briankeating.com/blog
Audio-only: https://briankeating.com/podcast

#AIrisk #aisafety #stablediffusion #RomanYampolskiy #EmadMostaque #briankeating #intotheimpossible