Should AI investors be worried? | The Global Story

Why is Anthropic telling future investors that its technology could pose, quote, existential risks to humanity? That warning was part of this lengthy report on risks that the company filed ahead of going public. Later this month, Anthropic will become the first. fully AI-focused company to float on the U.S. stock exchange. It's predicted to be one of, if not the biggest valuations in American history. So what story is Anthropic telling the world about itself to get people to buy in? From the BBC, I'm Asma Khalid in Washington, D.C., and welcome to The Global Story on YouTube, our show about where the world and America meet. I'm Mark Chisnack, and I am the BBC's AI correspondent. Mark, welcome back to The Global Story. I know we've had you on before, but it's been a while, so thank you for rejoining us. Thanks for having me on the show. So the big headline here is that the AI company known as Anthropic is expected to go public very soon. It'll be the first fully AI company to do this, to go public on a stock exchange. This is anticipated to happen later this month, in the month of November. What will it mean for Anthropic to make that move? Going public means a company sells shares to investors on a stock exchange for the first time. In Anthropic's case, it would mean moving from being a privately owned company backed by a relatively small group of investors to one whose shares are bought by and sold by the general public. Now, the process is called an IPO or initial public offering. It's a significant moment because a company has to open up its financial records to scrutiny. Unfortunately, in the U.S., that involves filing. Filing a registration statement with the Securities and Exchanges Commission, the SEC. That document's known as an S-1. And that filing, it tells potential investors how the company makes money, how much it spends, what its financial position looks like, and what the risks are by buying shares in that company or operation. It also means the company becomes accountable to a much broader group of shareholders. Anthropic is an interesting example because it's operating in an industry where the potential market is absolutely enormous, but the economics are still evolving and evolving really, really rapidly. So investors are being asked to believe that the company can turn its rapid growth that it's seen into a sustainable business. Now, an IPO can raise enormous amounts of money. Giving a company the resources to invest in infrastructure, research and development, and further expansion. Now, Anthropic reportedly got something like about 131 billion dollars in cash. And that's a huge, huge amount of money. Yeah, so it's not running out of money. It's not in need of money. No, no, no, no, no. The question is, you know, why does it need to, when it's got so much cash in the bank, as it were, why does it need continued or further investment? And that sort of further investment is based on its financial commitments. That's to secure computing power. Does that mean those data centers that are being built? Yeah, yeah, yeah, yeah, basically. Basically, that's what that means. So all of the work that is... is performed by its models has to happen at data centers, has to be completed on effectively other people's computers. When we say the cloud, when we say cloud computing, what we really mean is somebody else's computer. I see. The response is created there and then sent back out to you on whatever device you're making that request on. Margot, I want to talk a little bit about what Anthropic actually does. What it stands for? Anthropic was founded by Dario and Daniela Amodei, as well as other engineers from OpenAI. They founded the company back in 2021. Hmm. Now, OpenAI is the people behind ChatGPT, and they're arguably the people that have kicked off a lot of the interest in the consumer use of artificial intelligence. They left OpenAI partly because of concerns about how rapidly artificial intelligence was developing and whether enough attention was being paid to safety. Now, Anthropic founded on this idea that developing powerful AI systems should involve a much stronger emphasis on understanding, controlling and managing the risks associated with the technology. And in the development of these models, that if you bake that in right at the very beginning, It makes the whole process safer. So, as a consequence, they broke off or broke away or left OpenAI and set up Anthropic with that sort of philosophy at its heart. I imagine many people listening or watching our conversation will be familiar with Claude, right? And this AI assistant that Anthropic makes. Other chatbots are available. I'm just going to put them in the BBC and say other chatbots are available. So, so. And that's what they might think of when they hear Anthropic, right? It's sort of the most familiar, I think, aspect of this company to the average person. But what else does this company actually do and how good is it at what it does? They have a family of AI models, a family of frontier AI models, if you like. Which can perform a variety of different tasks. Claude, the chatbots are the ones that it's probably most famous for. There are various iterations of those models, right up to super, super advanced, which aren't available to the general public. Okay, that's what I was going to say. So there is a version of this AI technology that is not. There are versions of this technology which aren't yet available to the general public, but which some areas, some industries do get access to and are experimenting with. It's embedded into a wide variety of different industries, from defence, healthcare, Healthcare, the financial industries, you know, it's used in a wide variety of different sectors and spaces. They are a serious, heavy-duty player in the AI space. What does that ethical philosophy look like in how the company operates? Well, Dario Amodei has warned that AI could become extraordinarily powerful, potentially a lot faster than our ability to understand or control it, and that that risk needs to be taken really, really seriously. However, I have to say, recent incidents challenge that philosophy, though. In its own testing, Anthropic AI agents have accessed real-world systems. They've bypassed restrictions and taken actions that they weren't supposed to. Now, the company has acknowledged problems with both its own security arrangements and the model's behaviors. But again, lots and lots of industry observers and lots of people looking at this and critics. They argue that these incidents expose just how difficult it is to control autonomously or increasingly autonomous AI agents. This brings us to sort of Amodei's idea of pacing the future, slowing the development of the most powerful AI systems to give safety measures time to catch up. Now, some industry observers suggest that talk of a slowdown is about creating a regulatory environment, which makes it difficult for tech outfits. Who don't already have a frontier model, and that means a really, really advanced model already developed, makes it really difficult for those people to compete or to catch up. So when we look at things like a company like this one going for IPO, putting itself onto the public stock market, there'll be a lot of competing concerns and interest there. Can a company genuinely prioritise safety while it's competing in an industry where being first... could be worth billions and billions of dollars. And in some cases, you know, it could potentially be worth trillions of dollars. So it sounds like Anthropic was founded by people who began their careers at OpenAI. I'm not saying it was spun out, all right, but it was developed chronologically after OpenAI. And yet Anthropic is now... The first of these AI companies to go public. How should we interpret that? It demonstrates that this company has reached a certain scale where public investors are prepared to consider buying into its future. But it doesn't necessarily mean that it's won any kind of AI race, if you like. You know, the industry is still evolving rapidly and the competitive landscape includes open AI. Google, XAI, and any other number of AI companies, you know, Anthropic's financial disclosures suggest that it's achieved extraordinary growth. But they also reveal enormous costs associated with building and operating advanced AI systems. The company is making commitments that extend years into the future based on the assumption that the demand for AI will continue to grow. And that is a really, really big bet. And that's asking people that want to buy shares in it also to make a really, really big bet that that demand will continue to grow and that that demand is going to continue to grow in the way that it is at the moment. So there's a lot of question marks. As with any investment, as with any company that asks you to invest in it, it's a leap of faith. You're taking a trip to the horses, a trip to the track. You're taking a trip to the casino at Monte Carlo, effectively, and putting it all on red. And I think at the moment, the concerns, the concerns around the money markets and the concerns around sort of people in business is those numbers. That's the bit that they're drilling down into. The numbers, okay. Yeah, they're seeing a lot of potential for growth, but they're also seeing enormous costs. You know, the cost of infrastructure, the cost of paying to use computing power, the cost of paying for computing power to make these AI technologies work, and the cost of having to basically make a bet. That you're going to need a certain amount of computing power which is going to cost you a certain amount of money and making that bet right now means you've got really really significant billions and billions of dollars worth of commitment to spending and a lot of stuff could change. The technology could change between now and, you know, two and three years. Look at the speed of change in this space. It's absolutely remarkable. Demand could change. Computing power could change. New technology could come along, which completely upends the way in which we think about how this tech works. So as a consequence of that, you know. Any sort of growth predictions are, you know, they are that, they're predictions. Nobody's Nostradamus here. So it sounds like investors may be nervous, you're saying, in looking at the cost, the numbers. That's the information that's interesting. I think lay people, when they heard this news, were intrigued by the section of the report, the S-1. That's the report that, again, has to be filed with the Securities and Exchange Commission here in the United States before Anthropic's IPO. And the section of that report that I think caught some people's attention was that Anthropic was detailing the risks of its technology. Now, again, that risk section of the S-1, my understanding is that it is a standard part of any public filing. You need to tell investors what. They are risking, essentially, by putting their money in. In its S-1 filing, it makes a really big thing about risks. There's quite a large section on that. I'm going to read some of it now. Because our models write a large and growing share of our code and increasingly perform our research, including our safety research, misaligned behavior and internal use could undermine that work, introduce vulnerabilities into our systems, or into the training of future models. ...or result in models operating outside of our intended controls. And that sounds to me very much like the disclaimer that you might get at the end of a pharmaceutical commercial in the United States. Poses catastrophic existential risk to humanity, right? Yeah, yeah, yeah, yeah. Essentially, yeah, it's kind of language that you're a bit more familiar with in the US than we are in the UK. But essentially, it's pointing out some of the real and present risks that this technology represents. And there are a lot of industry observers. And there are a lot of industry experts who say, yeah, that is a perfectly reasonable thing to point out. In the same breath, there are also a lot of people who say, OK, pointing this out does serve a commercial imperative because it highlights how powerful your technology is. Your technology is so powerful, it has the capability to end human life, possibly, you know, which is a very, very bold statement. And so, you know, both of those things can be true at the same time. So it's quite unusual. Some people might say that these risks are being highlighted and pointed out in the S1, but it's perfectly reasonable to do that because they are. You know, perfectly reasonable risks that people are worried about in the way in which these technologies evolve. So given all that, why do they think people should invest in the company? What is their argument that they are making for that? Well, that's a worst-case scenario risk. That's a worst-case scenario risk that if something goes wrong, then all of these terrible things could occur to the planet. In the best-case scenario risk, these technologies could be used to kickstart all kinds of different things. You know, healthcare is the is the big sort of poster child that is often used that these tech. That AI technology could be used to help kickstart or help create new therapies, new drugs, new drug discovery. I mean, drug discovery is going on all the time using this kind of tech, that it could have enormous benefits. for humanity and so yeah you take the rough with the smooth there's a risk it could there's a risk it could end everything and there's a risk uh and there's a there's the potential that it could make a lot of things a lot better but there's also you know risk outside of ending humanity if you like there's the risk that it might potentially do to the workforce that it might displace lots of people in the workforce and that is something that is real and present that is going on right now but But from an investor's perspective, Are you concerned about the risk to wider society or are you concerned about the potential for making a profit? And that is essentially, you know, that's that's the capitalist dilemma at its heart that anybody has to ask themselves. Mark, our show, The Global Story, is about where the world and America meet. And I recognize that Anthropic, at its core, it's an American company. I realize many people around the world may use some of its tools. But if you are listening, watching our conversation, and you're not in the United States watching this IPO, why does it matter to you? Because of the influence that these companies have, the vast majority of the most successful, really powerful, advanced AI technologies are developed in the United States. China is catching up very, very fast and has some incredibly capable technology, which is worrying for a lot of people in the US. But these technologies have global influence. Their influence is being felt in the public sector around the world. So, for example, They're being used in local councils. Here in the UK to help perform certain tasks, certain backroom tasks, if you like, deal with huge amounts of paperwork and digitizing some of that paperwork. You know, very boring, time-consuming tasks. AI is being used to do all that sort of stuff. It's being used in healthcare to help doctors transcribe conversations that they have with their patients. There's a lot of different reasons why people might be concerned about, you know, an IPO that's occurring for an US tech company that's trying to sell itself on the stock market with a valuation of $1.5 trillion or something like that, which is, again, just eye-watering and staggering. Yeah, at some point it just begins to feel like monopoly money. You're like $1.5 trillion. Yeah, these companies have a global impact. And that global impact affects a lot of different industries. It affects, you know, the world of work. It affects, you know, the public sector. It affects defense. And as a consequence, you know, the ripples that occur out of the United States will be felt across the entire planet. I want to return to this question of pacing the frontier. Yeah. You alluded to this. essay that was written by the co-founder of Anthropic titled We Must Pace the Frontier where you know he spoke in this kind of manifesto about slowing the pace of AI being very judicious right about how they're developing the technology. How does going public fit into that vision and Mark the reason I'm asking that is because when you are answerable to shareholders They will have some sway over the decisions you make as a company. And what if they want you to move faster and make more money? Is that not possibly at odds with slow rolling the development of the technology? That could. ...create enormous tension. There is the potential there for an incredible amount of tension, which is the case for lots and lots of different companies. And we see this borne out in lots and lots of different companies and lots and lots of different industries, that shareholders may have a different vision for that company and may have a different vision for how it will become profitable or how it might profit them than senior executives of that company might have. And that's a tension that is borne out every single day. When it comes to this notion of... of slowing down the development, perhaps, to allow safety measures to catch up. There are a lot of people in the industry who their eyebrows were raised when this suggestion was made, because a lot of the big frontier labs, a lot of the big AI companies started making the same kind of suggestions round about the same time. Um, which is why a lot of those people said, okay, maybe this is around the notion of creating regulation, which putting up a fence, which keeps everybody else out. And allows us to build what we want to build. Only the big players inside, you're saying. We've built our frontier models. We've spent billions and billions and billions developing these technologies. So we're at a certain level now. Maybe we need to shut the gate and stop a lot of other people getting in. So that becomes a bit of a monopoly, if you like. They have a monopoly on those technologies. And that by creating particular sorts of regulation, they can keep other people or other companies out. So there's a lot of people that are suggesting that that may be playing a part in some of the discussion around this slowdown. I guess I'm just trying to understand with this vision or this philosophy that Anthropic has put forth since its founding, this sort of ethics of AI, how does that mesh with the idea of going public? What does going public mean for the ethics of this company? Some investors may find that desirous. Some investors may say, OK, this company says that it has this, it's founded on these ethical principles. This is a technology that doesn't look like it's going away. This is a technology that has enormous potential. And this company is saying that it has these principles. I believe as a human being or as an investor that if we're going to protect our species and if we're going to protect our society, that a company that is creating this incredibly powerful technology and at its heart, it's saying it has these ethical values. Well, that might be the company for me. There may be investors out there that believe in that mission. And conversely, there may be a lot of investors who Don't care about that mission and just see it as a financial opportunity. And there will be a lot of people, you know, the tech industry, the wider tech industry will be watching what happens to Anthropic when it IPOs really, really closely. So they're just the first. We're going to see additional. Tech companies, AI companies making their IPOs, you're saying? OpenAI is likely to do the same thing next year in the first part of 2027. OpenAI is likely to IPO as well. And they will be watching really closely what happens because if Anthropic does end up with this valuation, there's a lot of suggestion that it might be for a trillion, for 1.5 trillion. These are huge, huge numbers. And if OpenAI is going to perform the same thing, If OpenAI is going to IPO as well. as well, then it's not going to want to IPO for any less. It would be unusual if they were to make an offering, make a public offering that was valued and where they valued the company at a lower value than Anthropic. On that note, Mark, thank you so much for joining us. It was such a pleasure to talk. Thank you for having me on. That's it for The Global Story on YouTube. Thank you, as always, for spending some time with us. And a reminder that The Global Story is also an audio podcast. You can find new episodes from us every weekday by searching for The Global Story on BBC.com or wherever you listen to your favorite podcasts. Thank you so much for joining us. Thank you so much for joining us.