Why Are Ai Companies Hitting the Brakes? | Dw News

Well, some of the biggest rivals in artificial intelligence are calling for the industry to slow down. The chief executive of Anthropic, Dario Amadei, has warned that safeguards are failing to keep pace with rapidly advancing systems and that within a year, groups of AI agents could be capable of causing catastrophic damage online. The head of rival OpenAI, Sam Altman, says his company is pushing back its initial public offering to next year, citing concerns over AI. Altman and ex-AIs Elon Musk backed Amadei's call in an unusual show of agreement across the industry. Conor Leahy is U.S. executive director at Control AI, which is a nonprofit organization that works to prevent the extinction risk posed by superintelligence. Welcome to you. Now, these warnings sound a bit like vindication for what your organization exists to prevent. Are you happy about that? Or would you rather have been proven wrong? I mean, needless to say, I would have much rather been wrong. Sometimes I joke that the good news is we were right. The bad news is we were right. OK. Now, as I understand, your company was consulted on a U.S. bill that wants a full ban on developing superintelligence, superintelligence systems. What would this look like in practice and how would it be enforced? So the most important thing here to understand is that what these companies are trying to build is not a better chatbot. It's not a tool. It's not something to help people. It is what is called superintelligence, fully autonomous AI systems that can fully replace and outcompete humans across all tasks, whether that is economic, science, engineering, politics, military or anything else. And these are not systems that we know how to control or understand. So if there were such systems, and of course, not just one, because it's software, so it can be copied. So we will quickly have millions, billions of things that can outcompete humans, outcompete us the economy, outcompete us at warfare, outcompete us at everything else, and that do not have our best interests at heart and we cannot control. Very hard to imagine that going well. So therefore, we think the only reasonable thing is to not get into this situation. We must ban the development of such systems, both domestically and internationally, because if they're built anywhere in the world, whether that's in Western countries such as the United States or China or anywhere else, this is an unacceptable risk to our security. But this is proposed US legislation. It's not a global treaty. If the US does go ahead and ban this type of development, what's to stop China or anyone else racing ahead? So actually, a very important component of the call from Sanders and Kassar is to call for international regulation. National regulation alone cannot solve this problem. Fundamentally, if the United States builds superintelligence, the United States will cease to exist, as will other countries. If China builds superintelligence, China will cease to exist, as well as all other countries. As such, we must find a way to build a trust but verify regime with countries such as China where we can mutually verify that neither of us are pursuing superintelligence. This doesn't mean we don't pursue any forms of AI. There are many forms of AI that are not superintelligence and that have very important and valuable economic and military applications and should further be pursued. But no one wins if superintelligence is built. And so we must build such a trust but verify regime that is backed by credible deterrents. But in the example of China, what's to motivate China to rein itself in on the advice of a rival like the United States? Fundamentally, their own self-interest. If China builds superintelligence, they cannot control it either and it will lead to their destruction. If an American company builds superintelligence, it will lead to the destruction of China and the United States and everyone else. Fundamentally, in a sense, the game theory works out here in that it is in fact not in the interest of any sovereign country to build superintelligence. Now that is not sufficient for a deal to actually happen. There will still need to have to be monitoring and verification to ensure adherence to such agreements that are backed by a ladder of escalation, everything from political or economic sanctions up to reserving the right to self-defence. Now Sam Altman and Elon Musk have backed this call by Dario Amadei, the CEO of Anthropic, to slow down development. What does that tell us about where the industry stands on safety when it comes to AI? Most of what is being said by these companies is marketing hype. If you read the things that people such as Dario are proposing, you may notice that they talk about pacing. They do not talk about stopping or not building superintelligence, despite them and their employees and many others for years now having already warned that there could be a 10% or even higher percentage chance that such superintelligent AI could kill everyone on earth, every man, woman and child. Now, in my personal opinion, if I found myself that my company was building something that had a 10% chance of killing over 8 billion people, I would simply stop doing that. They seem to not be interested in not doing this and so therefore, what is important is fundamentally that government gets involved, that we stop this from happening, that we stop not just these companies, that we stop all companies and all actors internationally from pursuing and building superintelligence. You've hinted it there that they're not really interested in these calls to ban or reign in AI, yet you have tweeted this week, we're at a watershed moment. The vibe is shifting. It's not over yet. We can still stop artificial superintelligence. Do you really believe that given the power of these other actors? Yes, I do, because I do believe that we still do live in a democracy. The government does have the ability to reign in companies and the public does have the ability to affect what the government does. It's not perfect. We here in America are a very divided nation. But what I can say is that the bipartisan upswell of political will against AI of many kinds that I've seen in the United States over the last months and years is second to none. I've never seen anything like this before. Really, this is not a right versus left issue. This is a human versus machine issue. This is a question of how do we want the future for us and our children to look? And this is not something that anyone here wants. No one wants to be replaced by AI. No one wants AI threatening our lives. So, no, I absolutely believe that we can still do it. We can still stop these companies. We can make it illegal, but we are running out of time. And well, in a few years' time, if we don't do anything, it might well be too late. Thanks a lot for that, Connor. We've been speaking to Connor Leahy, U.S. Executive Director at Control AI. Thank you. I speak to Sam Winter-Levy, Senior Fellow in the Technology and International Affairs Program at the Carnegie Endowment for International Peace, where he focuses on the geopolitics of AI. It's good to have you with us, Sam. Why are these warnings emerging now, including from two anthropic researchers who quit this week? I'm just wondering what has changed in the capabilities of AI? Yeah, I think there's two main reasons why we're seeing these warnings emerging now. The first is about the capabilities of these models. So it's becoming increasingly clear that these are very powerful systems. They have the ability to solve unsolved math problems that have thwarted human researchers for generations. They have the ability to hack into the operating systems that underpin pretty much the entire Internet, apps, you know, much of what modern technology relies on. So these models are very powerful. The second reason is that the researchers at these companies are not confident that they understand exactly what's going on inside these models, that they understand how to control these models. And so this combination of increasingly powerful models combined with a lack of confidence that the companies that are building them know how to control them is underscoring why so many researchers are warning about the risks of these systems right now, including researchers within the companies that are building these systems themselves. And it's prompted an unusual agreement where Sam Altman of OpenAI and Elon Musk of XAI have backed Dario Amodei's call to slow development. What does this tell us about where the industry actually stands on safety? Yes, I think this is a pretty rare moment where you get Dario Amodei, Sam Altman, Elon Musk, you know, figures who have a complicated personal relationship with each other, to put it mildly. This is a rare moment where you're seeing agreement across the board. I think all of these executives, all of these companies are increasingly alarmed by the pace of progress, the pace of progress that they themselves are driving. But they're increasingly alarmed at the power of these systems and their inability to control them. We've seen many of these companies have real world instances in the past, in the past months where agents have not followed instructions, have hacked into systems that they were not supposed to hack into. And it's really only out of luck that we've not yet seen a real world disaster with real economic or human costs. So I think there's broad agreement across the industry. Now, what exactly to do about that? How heavily the government should be involved there, I think there's much less agreement. And so it can be quite easy for all of them to get on board with these kind of general calls for prudence, general calls to slow down in some abstract way. But when we start seeing more concrete, specific proposals, I think this agreement may be a little bit more tenuous. I suppose there are two things when it comes to regulation. One is whether Congress has the technical knowledge and the political consensus to regulate AI. But the second is about whether we can even trust these companies to regulate themselves. Yes. I think on the second point, we can't trust these companies to regulate themselves. That's exactly why you should need some form of government regulation, some form of congressionally mandated regulation or regulation coming from the executive branch, from the White House. So I think in the absence of government regulation, we're seeing the companies trying to figure out these kind of self-regulatory schemes. But no, we can't trust any company to oversee itself. On the question of whether Congress has the capacity, the technical capacity to do so, I think right now, in general, government is just moving far too slowly, given the pace at which this technology is moving. Government does not have the technical expertise. It does not have the political consensus to regulate these. That's starting to change. You're starting to see a broader range of senators and congresspeople coming out in favor of regulations from both the Democrat Party and the Republicans. But right now in the White House, there is not the same appetite for regulation. We've seen President Trump be quite dismissive of the risks of some of these systems. Parts of the White House, I think, are aware of the risks and the dangers, but other parts are much more focused on the risks of losing to China in this race. And so we don't yet have enough political appetite to regulate in a binding, enforceable way. So I think that's starting to change. Yeah. On that note, we've heard from Trump reacting to this and saying he doesn't seem very happy about the calls for caution in the development of AI technology. If you're having that reservation, then in terms of a timeline, I guess we're racing between the dangers of AI and how fast we can catch up to regulate it, then doesn't that create a problem? Yeah, for sure. So one of the issues here is that there are these multiple prisoners dilemma. There are these multiple competitions going on. One competition is between the different companies within the United States, where none of the companies trust each other. None of them want to slow down, if that means they're going to lose to another company. And so that's one reason why you need the government to regulate, so they can all slow down at the same time. But then you also have this broader competition between the US and China, where you also will need some sort of international agreement with China to get them to slow down too. And right now, I think the White House, President Trump are very skeptical that you can do any sort of agreement with China. They're skeptical that China is going to slow down. And so they're worried that if the US companies slow down out of fears of the technology, which in their view, the companies maybe are inflating, then the only outcome of that will be that the United States loses this race to China. I think there's lots of issues with that argument, but I think that concern that if you regulate domestically or if you slow down domestically, you just see the competition to a competitive state, but you also don't trust with this technology, I think that's motivating a lot of the pushback to these calls for regulation from the labs. If at all, there is a genuine slowdown, what would it look like, for example, inside the labs in Washington and what actually reaches us, the public? So I think that's a great question. I think the labs are still trying to figure out what exactly this means. I think the term that they're using is pacing the frontier. So it's not really a full pause. They're still going to be releasing new models, but they're going to be slowing down the rate at which they release these new models so that they can spend more time testing the models for the exact capabilities that they have, testing that the models are aligned with the intentions of their creators, testing that they really understand how these models are going to behave in the real world. So from the public's perspective, I think AI progress will still feel very fast. You will still see new models coming out. You will still see the capabilities improving, but it may not be from inside the companies. More effort will be going towards safety, towards alignment, making sure the models behave in the ways that you expect them to, and you won't see quite as dramatic a rate of improvements in these models. But I think from the general public's perspective, it will still feel like AI progress is happening very rapidly. Sam, I must ask you this. For a long time, the people outside, or at least people who've been in AI or even people observing, who have been warning about this for a long time, and now that the AI insiders are starting to speak, I suppose there's that question of, you shouldn't be surprised. These are people who've been working on these things, so they should have known a long time ago that these dangers were there, or at least were heading in this direction. So I'm just wondering why it all seems, why are they all coming out at this time, when this should have been known a long time ago? Yes, I think, as you say, none of this is that surprising. Many of the people building these technologies have been warning about these risks for a long time. Lots of careful observers of the technology have been warning about this for a long time. I think one of the reasons we're suddenly seeing this flurry now is that AI progress has been taking place on an exponential curve. And with exponentials, you can either react too early or too late, but it's very hard to react at exactly the right moment. And right now, we're at the part of the curve where the capabilities of these systems are just becoming increasingly undeniable. We really are seeing unsolved math problems being solved that no human had been able to solve before. And we really are seeing these models breaking out of human control, hacking into systems they were not supposed to. In the past, those were risks that people warned about, but they felt like sci-fi stories. Now we're actually seeing this happen, and I think that's proving to be a big wake-up call for the researchers and for the companies that are designing them. I think the final element to this is that even though each of the individual companies has warned about some of these risks in the past, some companies more than others, they were always more concerned by the fear of falling behind their competitors. And I think right now, they're still concerned by that fear, but they're also worried that the systems themselves will just do something terrible that they did not expect them to do or want them to do. And that is now starting to balance out their fear of just losing a race with their main rivals. And so I think that is also prompting the renewed urgency of these calls for a pause or for a pacing of the frontier, for a slowdown of some sort. Sam Winter-Levy, thank you very much for speaking to us. Thank you for having me.