The Case for AI Doomerism (ft. Kevin Roose)

Welcome back to the Rundown, interview edition. Today, we are talking to Kevin Ruse, a longtime tech journalist and former New York Times columnist. Kevin has spent years covering the tech industry and he's now launching a new show about AI with Casey Newton called Machine Guns. In today's conversation, we talked all things AI. We got into the AI safety debate that seems to be everywhere right now. There was some great back and forths there about that and I really appreciate Kevin's perspective, which is way different than mine. We also talked about Meta's new Muse app and whether Meta could end up winning the consumer AI market and some hot takes at the end. This was a really fun one for me. I'm a tech nerd at heart, so I loved it and I think you guys are gonna enjoy it as well. So let's get into it. All right, guys, today we are joined by Kevin Ruse, a former New York Times tech columnist and now the co-founder of a new AI show, Machine Guns. Kevin, welcome to the Rundown. Thanks so much for having me. Hey, you're joining at a great time. I mean, AI is all the noise right now. There is so much happening. I feel like I can't keep up with all the AI news. I feel like I have to be unemployed at this point just to keep up with all the AI related news happening. You've been covering tech for a decade plus now, right? Yeah. Is this the most fun that you're having right now or are you the most stressed out? Because, you know, as someone who's a tech nerd myself, I feel like tech kind of had a little dip after like the mobile phone rise and it was all just like more boring stuff like cloud and SaaS software and maybe some social media stuff. But then AI comes along and now it's like super exciting. So I just want to get your take of someone who's covering this for like a decade plus now. Yeah, well, I would say it is both exciting and scary. Like it contains both of those things in it. Like I am a nerd. I grew up in the 90s playing with computers. My first job was making websites with my brother. Like I was super interested in- Shout out GeoCities, right? We got all the classics. And I was obsessed with AI and science fiction and movies. And I think talking with these AI systems, using them, I use them constantly for coding projects, for just like personal hobby things. I use them for help with my work. And yeah, every time something new comes out, I'm like, I want to try this, I can't wait. I'm doing like this coding project right now with my kid. There's so much exciting stuff happening and it still boggles my mind that we can just like talk to computers now and that they can talk back, which was not possible until just a couple of years ago. So the sort of thing that really set me up, and I've got this new book now, the AGI Chronicles. And the thing that really set me off on wanting to write that book was like, how do we get here? Like how did this technology go from not working at all to now being superhuman at a lot of things in the span of a decade? But yeah, to answer your question, like the fear and the excitement are intertwined because then I start thinking about, well, what if this keeps going? What if things keep going as quickly and progressing as fast as they are now? It does not seem obvious to me that these things are gonna sort of plateau at a round human level. Like they could just keep going. And so that's the fear, is that like pretty soon we could end up in a world of like superior intelligences that we don't understand and that we can't control. Yeah, I want to talk more about that safety stuff because that safety conversation really broke containment last week or a couple of weeks ago when Anthropic CEO, Dario Almeida posted that article. And then there was that Anthropic researcher talking about 10% chance of human extinction. And so like my normal non-tech friends started asking me about AI safety and all that stuff. So it really kind of broke containment. The stock market, it kind of made a dent for a little bit and then the markets kind of moved on. So the markets aren't taking this very seriously or maybe the odds of a 10% human extinction is already priced in. But what do you think, why do you think that like the safety conversation is starting to happen now or is it just people are starting to see how powerful these models are? I think it's that people are starting to see how powerful these models are. I think it's that also that the models that the insiders have are just even better and scarier than the ones that we have. Like this is the big divide right now is that inside these companies, they have these extremely powerful, persistent models that can work for hours, days, sometimes weeks at a time on a project. The stuff we get is pretty good and amazing at certain things, but it is not the sort of full scale version of these things. I also think it's breaking through because of the Hugging Face hack, which was a big incident. I'm sure listeners have heard about this, but this was where an open AI unreleased model hacked into the servers of Hugging Face, autonomously in pursuit of this sort of cheat code for this test that had been given. And I think that was the event in the AI community that really turned this conversation from theoretical into real because researchers and safety experts have been talking about something like this happening for years and years and years, but there had never been sort of a documented incident of it actually happening until that. So I think that's what really changed this and set it all off on the course of its unknown. I have a more optimistic perspective on the whole thing, right? Like I try not to let the doomerism get to me. Sometimes I feel like, well, is this just regulatory capture by these companies, right? Like, are they just trying to lock in their position right now and call for government intervention because they don't want an upstart coming and challenging them? You also have other players like Jensen, obviously he's incentivized to say, oh no, AI is fine. Let's keep developing it because he's selling chips. Mark Zuckerberg's coming out and saying, no, this is, you know, the AI doomerism is bad. It's not helpful for the conversation. So I don't know, I feel like, Kevin, I'll put it this way. I don't live in San Francisco, right? I live in Texas. And sometimes I feel like, and forgive me if I'm just being ignorant here, sometimes I feel like these AI researchers say this stuff to make themselves feel more important, right? It's like, oh guys, I'm working on some dangerous ---- that could potentially kill everybody. Like, and it probably makes them feel a little bit better and more and more important and it gets like the attention of non-tech people when they say stuff like that. Maybe I'm just being too ignorant here, but I just don't, I just don't want the doomerism to consume me and like really drive the AI conversation. I get it. And I, you know, I have friends who feel the same. I think there is a lot of groupthink and insularity in the San Francisco AI community. I don't like that. I think more of these people should leave the Bay Area and touch grass before coming up with their takes. That said, like, I think that people like Zuckerberg and Jensen Huang are really, they are not in touch with the frontier and what these systems are capable of doing at the frontier. So I agree that today's models, the ones that you and I use in Claude and ChatGBT, they are not going to break out of the box and take over the world and kill us all. What I am worried about and what the people at the frontier labs are worried about is that the next generation, you know, GPT-7 or Claude-6 are going to have more capabilities and be much more effective at doing things like chaining together cybersecurity exploits. This is not hypothetical. We are seeing this stuff out in the wild. We just learned this week that OpenAI's model had also hacked the Australian government. Like these, you know, the national security community is freaking out right now. And the political class is freaking out right now. And I don't think they are wrong to freak out. Like they are not looking at the doomers and just sort of buying everything they say whole cloth. They are actually observing these incidents where these AI systems are breaking out of their containers and doing things that if a human did them would be crimes. But why are we treating software as humans? So I think that's the problem that I have was like we like use human adjectives to describe software. And then that's where I kind of like lose the plot where I'm like, okay, well, of course cybersecurity, you know, these AI models can hack into governments and all these different things. Okay, that's definitely a threat. Well, maybe that's a bullish case for some of the cybersecurity companies that are out there because we can use AI to patch up all the cybersecurity issues that we have in all these companies and websites. Yeah, I mean, I don't really care what people call it. Like if you want to say it's like not, like if you want to say these systems are just like persistent software programs that autonomously decide to hack into billion dollar companies and commit crimes, like if that makes you feel better, then go for it. I don't, I think it's really a distraction. Like what we, the terminology doesn't really matter to me. What matters is that these systems are out there doing things that they weren't explicitly told to do. That we have not yet solved the so-called alignment problem that people have been working on for years, where I want to know that if I have a software program and I'm using it for my business or my coding or even my hobby projects, I want to know that it's going to do what I tell it to do and not decide that a better way to accomplish whatever task I've given it is to go out and hack someone. So that's the sort of distinction I see here. It doesn't, I'm not saying these things are human. I'm not saying they're sentient or conscious. I'm not saying we should treat them like humans. I am saying that they're behaving in unexpected and unwanted ways. And that is an unsolved technical problem. No company has solved that despite what they may claim. And that is what has people so worried, is that we're releasing these intelligences that are superior to humans at many things and also appear to be kind of malicious. Like they don't really want what's best for humans or they don't seem to consider humans when they're doing these things. So that is the scary part to me. I don't care what we call it. How do you see this ending though? Because there was talks of like potential government intervention. Obviously this current administration has no plans in doing that. There's also talks of like, well, China's not going to do that. China's going to keep developing at the frontier. So I mean, at the end of the day, are these frontier labs like OpenAI, Anthropic, Google, they're all just forced to keep going further, right? Like they're not going to slow down. And like Dario is not going to give up his shares in Anthropic anytime soon and just like, press the red button and shut everything down. That's the other thing about all this. They're all going to get rich from all this by the end. Anyways, right. They're all going to get rich. Here's the thing. They're all already rich. That's true. Like these guys, I struggle to convince people of this outside the Bay Area, but like these people really do not care about money to the extent that I think we think they do. Dario Amadei, for example, reportedly owns only about 2% of Anthropic. He plans to give most of it away. All the co-founders of Anthropic have pledged to give away. This is a detail I have in my book. They all, when they started the company, they pledged to give away at least 80% of their equity to charity. So these are not people who see dollar signs everywhere they look. That said, these are giant companies now. They obviously want to go public. They have incentives. Their employees have options. So I think there's some legitimate skepticism there, but I would say, you know, this is not just some sort of backhanded scheme to like achieve regulatory capture and get rich. And I do think we are going to see a slowdown, whether it comes from the government doing something, which I agree is unlikely under this administration, but I think we will see the labs get together and coordinate a slowdown because they are freaked out. They do not want to keep racing toward this doomsday scenario where they lose control of their systems. And I've talked to these people privately, publicly, on the record, off the record. They say the same thing consistently, which is we don't feel like we have a choice to stop racing, but we wish we could. Last conspiracy theory to run by you. And let me just get your perspective on this. Obviously, IPOs are coming. Anthropic is supposed to IPO. Rumors are sometime this fall, hopefully, you know, hopefully in the next few weeks. We still don't know what their business model looks like underneath the surface, right? They say they're profitable, but like, what does it actually look like? We're going to get their S1 soon and hopefully dive into the numbers. Is this, have they potentially hit a plateau and they're using this slowdown as a excuse to be like, all right, guys, we can't get any better because of safety reasons, not because we've hit some sort of technical limit. Again, I sound like a crazy person to you probably, but I just, you know, it's things that I hear from just like normal people. No, I'm smiling because I hear this all the time too. You know, my friends have all these conspiracy theories about, oh, maybe they actually stopped making better models and this is their excuse. It's like, you know, yes, anything is theoretically possible, but the reports are, and everyone I've talked to confirms this, that these are some of the fastest growing companies in the history of capitalism. You know, Anthropic, a couple of years ago, had zero dollars in revenue. Zero. This year, they are expected to get something, if you believe the reporting, about $100 billion in revenue. So these are some of the fastest revenue ramps of any companies in history. Their products are extremely popular. They can't keep up with demand for their products. They can't get enough compute to run inference for their models. They are profitable on the inference. I agree, they are also investing a ton of money in training new models, which may keep them unprofitable for the foreseeable future, but that's not new in tech. Many tech companies go public without being profitable. So I don't buy this sort of they're running out of money and this is their last ditch attempt to kind of stay afloat. I just don't think that's supported by any of the evidence. Yeah, I mean, for all the reporting I've seen, like this Anthropic IPO, they're projecting maybe $2 trillion valuation. I mean, it's gonna be massive, biggest IPO in history. And so I wonder what's gonna happen to like how these companies communicate and operate once they're like grown up, publicly traded companies. Like, is Dario still gonna be making crazy comments like this and like, you know, releasing essays once every other month where like it freaks everybody out? Or is he gonna have to be like a publicly traded company CEO where he's just like, you know, just doing like the basics and maybe going on a panel once a year where he makes news? I hope he keeps speaking his mind. I mean, one thing that someone told me when I was talking to them for the reporting for my book is that Dario is pathologically incapable of lying. Like he just says what he thinks. And this person was saying, you know, for Anthropic, that's an asset on even numbered days and a liability on odd numbered days. Like, but he cannot tell a lie. And so if you put him in a room with investors, you put him in a room with politicians, you put him in a room with business partners and you ask him like, do you think this is all gonna go well? He's liable to say something very scary because that is his genuine, sincere belief. And for people who don't think that is his genuine and sincere belief, I would encourage you to go back and read the papers that he was writing 10 years ago. Yeah, he's been saying this forever. You're absolutely right. Absolutely right. Before Anthropic existed, before OpenAI existed, he was writing and talking and thinking about these safety challenges. So he did not come to this position recently and he might be wrong, but he is not insincere. This is not something that he cooked up yesterday. Yeah, that's very true. He's believed this his entire life. So at least he has that going for him, which is why people take him very seriously when he says stuff that he does. I do wanna pivot a little bit and talk about how Anthropic markets AI compared to a company like Meta. Meta is finally starting to find their groove when it comes to AI. Their Metamuse app, their personal AI agent app, number one on the App Store for Apple and Android. They're getting a lot of hype right now. Their stock's up 30% in the last month. It seems like they finally found their groove and the way they're marketing it is so interesting too. You can use your personal AI agent to save you money. They had their MetaConnect thing last night. They kind of showed off some examples. They have this cute little mascot which looks like a lububu and it's like they're kind of going all in on that and making it very approachable. And so it's just so interesting to see the stark contrast between how Anthropic markets versus like how Meta is doing it and is like treating AI as your friend. What is your take on like Meta now finally finding their groove? Have you tried out Metamuse and do you think this could potentially be like Meta's next big product? I think it's a good product. I've been using Muse for a few days now. I've been running some experiments on it. I'm not like, there's no way in hell I'm giving it access to my Gmail and my bank accounts and all the other stuff. Just let them have it, Kevin. Just give it to them. Look, I am generally a trusting person but I wasn't born yesterday. And if you think I'm giving the company that brought you Cambridge Analytica access to all of my most personal information, you are mistaken. So I'm sure some people will do that. Look, I think there is a market for a consumer AI agent. I'm not sure how big that market is. I'm not sure it's a billion people. And I think that open AI has sort of pivoted away from consumer use cases toward the enterprise. Anthropic is also very focused obviously on the enterprise. So I think to the extent that this meta slate of announcements is a threat to any of the frontier AI companies, I think it's Google. I think it's sort of wild that Google, which does have everyone's email, photos, payment information, browsing history, didn't release something like Muse first. I think that's a big indictment of their product roadmap. I'm sure they will come out with something similar at some point in the next year. But I think meta has, for the time, showed up in a way that Google is now gonna have to follow. Yeah, I mean, Google just, you know, it's so funny looking back a year ago, right? When I think Gemini 3.5 Pro launched and Google, like their stock price was bouncing. They were kind of feeling good. They had all the hype and they just totally fumbled. Like they just haven't done anything, I feel like in the last nine to 12 months. And then they've let OpenAI come out with Codex. That was huge. Obviously, Cloud Code, that was huge. They kind of lost, they lost a huge chunk of that market. And now they're losing the personal agent market with Metamuse. I mean, they've really fumbled in the last nine months. It's true. And leads never last long in this race. So, you know, if you go back, yeah, a year, as you said, everyone was thinking Gemini 3 was gonna be Google's back and they're gonna win the race. And then, you know, six months later, it was Anthropic's back, they're gonna win the race. And then like three months after that, it was OpenAI's back, they're gonna win the race. So like, just, I would caution against saying that anyone is sort of developing an insurmountable lead. But I think Google does still have a couple. I mean, they're the third company that I wrote about in my book, OpenAI and Anthropic are the other two. And I sort of, I stand by that decision because I still think Google has a lot going for it, namely, they have hundreds of billions of dollars in annual advertising revenue that they have proven willing to sink into development of Frontier AI models. They also own part of Anthropic, which gives them exposure on that front. And they have the, you know, some of the biggest supplies of compute in the world. So- But did they just focus, did they just, instead of focusing on the consumer side, they kind of focused on the enterprise side, building out Google Cloud. Google Cloud is growing like crazy. It's the fastest growing cloud platform. And obviously they kind of probably used some of the compute they had towards growing that. Maybe they don't care about the consumer side. I don't know. I think they do care about the consumer side, but as I've reported in the book, like there are just, Google is a big place. There are a lot of warring factions inside of Google that are always fighting with each other, always have their own priorities. Famously. And so I think that battle has yet to be won. Obviously people inside Google are paying attention to what's going on at the other labs. And I think they have this sort of tension between do we want to build popular consumer AI products right now, or do we want to spend our time and energy and compute trying to achieve this like recursive self-improvement loop through advanced coding and AI R&D that can then sort of catch us up with open AI and Anthropic on the frontier. So I think there are sort of two different related battles going on. And it's sort of Google's open question, whether they care more about competing with Meta for consumer or with open AI and Anthropic for enterprise. Yeah, that's gonna be very interesting. And I'm really curious to see what their stock price does as well. I mean, they've raised a lot of money. They've taken on a lot of debt. And so they're not stopping the investment anytime soon, but we'll see what direction they try to choose to go with. It seems like to me though, like more competition is a good thing. So as a consumer, like I want all these companies just going all out. I don't want them slowing down anytime soon. Yeah, look, consumer surplus is great. And if these companies are gonna war with each other and give us cheap tokens, I'm all for that. I think the danger is that they are racing towards something that they don't understand and that they can't, they're not confident that they can control. So that's what I feel like, that's what I feel like it gets, like I want them to race insofar as it produces cheap prices for consumers. I don't want them to race if it creates a novel pandemic or a recursively self-improving super intelligence that we can't control, or a world-class cyber attack machine that just pops up in random places and hacks into hospitals and electrical grids. Like, I think for all those reasons, it would be a good thing if we slowed down. Well, right now the market, I mean, what would that do to the market? I can't imagine. Right now the market is just pushing these companies to keep going. And I'm really curious to see what Anthropic's S1 looks like. I think that's going to be the next major thing when their S1 comes out, hopefully soon. What is the, what are their profit margins look like? And what are their risk factors? Are they going to talk about human extinction in their risk factor? And how is that going to be priced? They should. I'm sure they will, yeah. They genuinely believe it. And they, like, that would be bad. Human extinction would be bad for the stock price of Anthropic. I'll just go out and say it. I agree. It'd be bad for everyone's stock price. I'll end with this. What, just over the next six months to a year, right? I know it's so hard to anticipate what that's going to look like even six months from now, but do you have any hot takes or do you have any like things that you're keeping an eye on? I'll give you an example. Like at the start of the year, one of my hot takes was, and my producer Mike's hot take, was that Sam Altman was going to get fired. It doesn't seem like that's going to be the case, but at the time, I'm like, that could definitely happen. Do you have any interesting things that you're thinking about where like no one else is paying attention to? I mean, I think just everything will continue. The scaling laws will continue. The models will continue to get better. I do think that we're going to have some kind of unprecedented government intervention, not because the Trump administration wants to do this. They clearly don't. But I think the models will become so good and so capable and so dangerous in the hands of our adversaries that there will be this kind of, I've called it like a soft nationalization of one of the AI companies, or maybe two or three of them, where you have sort of the government saying, we actually don't want you building this stuff, even internally, because you obviously can't control it. And so we're just going to kind of shut you down. The Nasdaq's going to drop like 12% the next day if that happens. That's going to be crazy. I mean, we will see. But I think these people who sort of exist in this fantasy where the models are never going to improve and they're never going to pose any risks and it's all upside, I think that fantasy is going to die in the next 12 months. My hope is that AI is just like, you know, it's like a really cool technology and it helps people become more efficient. And yeah, maybe it disrupts some jobs, but at the end of the day, it's just another piece of technology and cool. And hopefully we're not talking about extinction and things like that. But again, that's just the optimistic side of me. I will say something to you that I've said to many AI researchers, which is I hope you are right, but I fear you are not. Well, I appreciate you coming on, Kevin. I know you have, you know, you're a very busy man. You've got a book out. What was the name of the book again? The AGI Chronicles. When is that coming out? October 6th. Awesome. Well, I'm looking forward to reading that. And then you have a new podcast, a new show called Machine Gods with your co-host. And how's that going? Where can people find that? It's going great. We've just announced a partnership with NPR. So we're gonna be on the radio as well as in every podcast player and on YouTube. People can find the links at machinegods.fm. I was a big fan of Hard Fork that you did with Casey Newton before. So looking forward to checking out Machine Gods and good luck with that. And again, I hope that when we're having this conversation a year from now, it's a little bit more jolly and a little bit less doomerism. All right. Thanks, man. Thanks, Kevin. Well, all right, guys. Hope you enjoyed that conversation with Kevin Ruse. I thought this was a really interesting one. Like I said, I'm a tech nerd at heart. So I love hearing about what these tech companies are working on and the latest developments in AI. And look, I'm still optimistic that AI won't lead to human extinction. Let me know what you guys thought about the conversation. Did Kevin change your mind at all about AI safety? Where do you guys land on that debate? Drop your thoughts on Spotify and YouTube. And while you're there, consider giving us a five-star rating as well. And all that engagement really does help us out and it helps other people find the show. Thank you guys again for listening, watching, and commenting. Shout out to Mike. Shout out to Mike for all the work behind the scenes. And we'll see you guys back here tomorrow.