Concerns and Optimism at King Charles's AI Summit for Tech Leaders

Activists assemble at Downing Street in London, united by concerns over dire predictions about AI. Industry insiders have issued stark warnings over the future impact of AI, from a possible AI takeover of the internet in as little as six months from now, to the outside chance that AI will wipe out the human race by the end of the decade. These protesters are adding their voices to a growing call to hit pause on AI development, but the fear there may not be much time left to do so. But the biggest upcoming risk is that we build machines that can make themselves smarter, that can do their own AI research. And this could lead to many dangerous developments, but the main problem would be that we build AI that is radically smarter than humans, and that will be almost by definition uncontrollable. So science fiction risks are starting to come true. Very serious scientists, including the most cited top research scientists on AI in the world, are all sounding the alarm to say that this is a very serious risk that we actually have to start thinking about how to deal with. And concerns like these are reaching high places. Here in Scotland, King Charles is hosting tech leaders for a talk on whether a principled approach could drive AI's future. Those who have created these technologies are now increasingly warning that AI risks developing darker capacities, perhaps even to take life. And in this regard, if I may say so, there seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands and being used in potentially catastrophic ways. Surely then we need sufficient means of control before it is all too late. But the gathered tech leaders have different views about how concerned to be. There are those who share the King's misgivings. This is a very new technology, and there are a lot of unknowns and uncertainties. And there are things that even the top scientists in the field don't fully understand yet about it. And I have to say the chance of something bad accidentally happening if we get things wrong is definitely non-zero. So we should take these risks very seriously. And those who see the plus side, including NVIDIA CEO Jensen Huang, who envisions an AI that brings positive results for humanity, provided it's developed with, quote, responsible optimism. He says reports of rogue AI agents have to do with products that weren't ready for release, and that with rigorous safety measures in place, AI can bring enormous benefits. Researchers at Oxford are using it to help design cancer vaccines tailored to individual patients. At Manchester, we work with researchers to develop AI models that produce detailed air pollution forecasts in minutes rather than days. AI is a new instrument. It is an instrument of discovery. It should be in the hands of all people, institutions, and countries. The debate about whether to hit pause on a rapidly growing technology finds no conclusion here. But the conversation is becoming impossible to ignore. And if industry insiders are right, we won't have to wait long to find out what any final decision could mean.