AI researcher warns of extinction risk | Interview with AI expert

A dire warning tonight as one of the architects of a highly advanced AI system sounds the alarm about his own creation. A former anthropic safety researcher says there's a greater than 10% chance advanced AI could wipe out humanity within the next decade. He quit the company accusing it of moving too quickly without enough safeguards. So is this a realistic fear or a long shot worst case scenario? Tech expert Joe Tavares joins us now live on the 10 to talk about this. And I guess it's time to finish up all the ice cream in our freezer because it's time to go it seems. I mean that's this is crazy that they're saying that in 10 years or less humanity could be wiped out. Please explain this to us Joe. So I'm going to lean a little bit on sci-fi here because we don't have any like real life parallels but if you look through all of the sci-fi stories that have AI it usually does not end well for humans. Yeah that's a good point. Good point. I don't like that metaphor but okay. What is it that they're saying though? Well this is a guy who basically worked for Anthropic which is a giant in the AI field and he's saying I can't do this anymore. This isn't working out for me because I'm afraid of this. When he says that why should that be why should that be an alarm bell? There's a couple reasons. Number one he also worked at OpenAI so he has kind of a behind the scenes look at the major frontier labs and then the other thing is he's like in his late 20s so his career in AI is effectively over after this. He I would say obviously believes in what he's saying. What is it that he's fearing could happen that would wipe out humanity? So there's a thought experiment in AI we call it the paperclip maximizer where if you set a broad goal like make more paperclips and you don't constrain that in any way the system will consolidate resources to maximize paperclip output and what that means is all of the resources so the car you me everything and that's a concern because a term that we call instrumental convergence which means like you and I both have goals but we also have to maintain homeostasis and drink water eat food etc. That's a convergent goal so we have to keep doing that. It's assumed that any super intelligent system would also have those instrumental convergent goals where it needs to stay alive to accomplish whatever task it's set for. So the concern isn't necessarily it's going to do a skynet it's more we ask it to do something mundane and it takes that way beyond the extreme. So what is this guy this researcher from Anthropic suggesting we do to eliminate that 10% chance of wiping out humanity? What are some I mean you talk about that it got out of the the sandbox right like the AI bot or whatever it is that can go do things. How do you just build guardrails and why aren't more people talking about that right now? It's it's a big discussion we call it alignment in the field aligning AI's goals with human goals. The issue is once you hit recursive self-improvement where the models are making themselves better you don't have control or constraints around what those goals are and once you lose that control you won't be able to get it back. Why is it that we're not safeguarding though if we know that this thing is capable of doing things beyond the control of what we tell it to do then why aren't we right now building around that to prevent it or can we not do that? We are in some ways but it's it's like we have laws in society to constrain people but people still commit crimes and go to prison so like there's not really any way that we can put guardrails around it that they won't be exploited or gotten around. It's a pretty big crime to get out of the sandbox and go wreak havoc on the entire humanity though and you hope that these companies you know is this guy saying it's too late to put the toothpaste back in the tube and that what we've created will already wreak havoc regardless of what guardrails we put into place? I don't think anybody's saying that my take on it is we're in this arms race and we're locked in it so if we look at game theory if everybody's agreeing and then somebody defects and starts to take advantage that person will inherently win because everybody else is agreeing to not do something so what we probably need to do is globally as a society agree that this is not a technology that we should take lightly and we should approach it like nuclear weapons have have independent third-party auditors and looking at it and make sure make sure that we're taking it at the right pace. Well that's very important I think that's a good note to end this one on. Tech expert Joe Tavares great to see you thanks for joining us here tonight. Thanks for having me.