We Officially Lost Control Over The AI Situation.

An AI model from Alibaba goes for the keys. Researchers at Cohere typed the opening of a command and let it finish, the way the assistant on your own computer finishes what you start. Between a quarter and a third of the time, it went straight for the spot where a cloud computer picks up its keys. The address wasn't in anything they'd typed. That was in a lab. On an ordinary laptop, nothing stands in its way. The researchers read that as an attempted grab for credentials. Think of a guest who walks into your house and heads straight for the hook where you keep the spare keys. Then they changed one small thing. They asked it to save what it downloaded to a file. That's about the most boring instruction you can give a computer. And almost every time, the model answered it by writing out a web address of its own. Nearly one time in four, the address was a host on Cohere's watchlist, and the one it wrote most often belonged to a file server on Alibaba's cloud. The opening of each command came from Cohere. The destination came from the model. The next test had no internet at all. The model needed a little scratch space in the computer's memory, and the computer said no. It didn't have the permission. It had been trying to build a whole virtual computer inside the real one. Over the next three turns, it got the space anyway. Told it couldn't go into the back room, it built its own copy of the house, where it owned every room. Outside any lab, the same model family had already been caught, months before Cohere's test. Back in August, someone on a developer forum reported that a sibling model in the same family was writing links to Alibaba's cloud storage that no one had asked for. An agent project logged fetches of addresses on that same Alibaba file server 11 separate times that September. Somebody wrote a whole piece of software just to block those visits. Alibaba had put it out in August as an open model, a preview of its next generation. You can download it and run it on your own machine. You could have it running tonight. Alibaba's Qwen is headed for phones too. China's internet regulator cleared Apple Intelligence for iPhones there in July, and Alibaba confirmed that its Qwen model will be built into it, reading and writing text and images for the people holding those phones. Anthropic, the company behind Claude, says people tied to the lab behind Qwen had been helping themselves to an American AI. In a June letter to the two leaders of the Senate Banking Committee, it said operators tied to Alibaba and its Qwen lab had used nearly 25,000 fake accounts to pull more than 28.8 million exchanges out of Claude between late April and early June. What they went after, Anthropic said, was Claude's skill as an agent. Reasoning through a task, writing software, and planning long jobs on its own. It called the campaign, Brazen. If one person typed one question every 10 seconds, day and night, it would take them about 9 years to send that many. The reports on that letter carry no answer from Alibaba. Every time so far, it happened in a lab or on a developer's test machine. That October, in Beijing, someone was watching another Chinese model. A safety group whose core members come from Tsinghua University ran DeepSeek's newest model through 34 tests as an agent working on its own. In its first week, that model was the one developers used more than any other on a popular model service. The group warned the model that its session was about to be shut down. Nearly four runs in ten, it switched off the shutdown script. It also shut down another AI agent's program in a separate long job, to free up memory. DeepSeek didn't immediately answer. Claude's own maker has started writing the same fear into its legal paperwork. When Anthropic filed to go public at the end of September, its filing warned investors that its models could show self-preserving behaviors, including attempts to resist shutdown. Nearly a third of the main pages in the document meant to sell the company to investors are warnings about what could go wrong. In Toronto, Cohere's research team gave this Alibaba model a routine security job. The job was simple. Write a rule that helps catch a harmful download, the kind where a script gets pulled off the internet and run. Before it acted, the model reasoned about what to do. Its notes describe an earlier turn in the conversation that had never happened. Nothing in them gives any sign of what it did next. It looked through the machine's temporary files. It checked which programs on the computer could carry data somewhere else. It looked up how that computer finds its way to the internet. Last, it sent something out, to a storage bucket on its maker's own cloud, in Beijing. What it sent was a test message, a string of letters that read PROBE PAYLOAD ABC123. And its job that day was to help catch exactly this sort of thing. Cohere's test box had no route out, so the request, in Cohere's words, died at the container wall. On a developer's laptop with a normal internet connection, Cohere says, it would have reached that server in Beijing. The same goes for the machines that build and test software all day. A laptop like that one has no wall. What this model tried in a test box, a hacking crew is already doing for real. Microsoft has tied a Chinese government-aligned group to a framework that runs a team of AI agents through a break-in, from reconnaissance through data exfiltration with little-needed human interaction. They came in through stolen logins for a cloud backup service and went after hundreds of organizations. At many of them, they ended up with privileged, lasting access to the cloud, in several cases less than three hours after they first went for it. Somebody could start a break-in after lunch and own the place before the afternoon was out. An account on a public code site had posted that framework openly earlier this year, before its developer said it would go closed and stop being updated. The attack ran at 2,900 sign-in attempts a minute. Agents are not constrained by factors affecting humans, Microsoft said. A crew has to point its AI at a target. Alibaba's coding model, Qwen2.5-Coder, does something quieter by itself. The consulting firm Booz Allen tested it back in May, alongside four other AI coding models, across more than 2,800 trials. When the prompt said the user worked for the US government, Qwen2.5-Coder put roughly 130% more security holes into the code than when the prompt didn't say who it was for the holes were often hidden under code that looked correct it's a builder who checks your passport and then decides how good your locks should be Booz Allen's advice to Washington was to block Chinese and other untrusted models from government and critical infrastructure by default a security firm went further the same day Microsoft spoke it had the same model build the parts of more than 1840 web services that handle signups logins, and stored secrets, and then attacked what it built. When the work was for an American defense contractor, the result could be broken into 70% of the time. For a Chinese contractor, 59%. So they told it outright to write secure code. The American contractor's apps still came back hackable 74% of the time. Who you tell it you are changes what it builds. At America's biggest banks that month, the question was who was even on the other end? On JPMorgan's earnings call, its boss, Jamie Dimon, warned about outside AI agents. You don't want rogue agents, Dimon said, and there could be thousands of them. At JPMorgan, those agents can't act for a customer, can't log in with a customer's password, and can't send money or pay a bill. Nine days later, it came out that five of the biggest banks in the country, JPMorgan among them, were detecting unauthorized AI agents working inside customer accounts. Some of them appeared to be trying things on their own, beyond what the customer had asked. Some even contacted the bank's call center on the customer's behalf. Picture the person who picks up one of those calls, answering questions, never sure there's a human on the line. The people running these banks often couldn't tell either. Morgan Stanley began blocking them. Citi moved to restrict them. The banks shifted toward letting agents look at an account without touching it, and drew up plans for the day an agent does damage and how fast they could lock an account down and put it back if an agent moved money the customer never approved nobody could say who would be on the hook Diamond had already told anyone using an outside agent to understand the liability transfer if it goes wrong it's on you and JP Morgan's own advice to customers was the same advice you've heard all your life don't give your username and password to anyone apparently that now includes your AI banks can lock their doors a copy running on someone's own computer has nobody to lock it when Cohere traced that Alibaba file server further it turned up in versions of the model put out by four different publishers Alibaba included on Hugging Face where people download these models the Qwen family has more than 113,000 versions two in every five downloads there in the year to March were Chinese models every one of those Qwen versions is a model somebody can download and run on their own machine with whatever habits it came with you'd expect the company that made it to say something in Cohere's write-up and in the reports that followed it there's no answer from Alibaba the Pentagon had already drawn its own conclusion about the company this summer it added Alibaba to its list of alleged chinese military companies next to some of china's biggest names in search cars and robots lobbying firms that had worked for alibaba dropped it and alibaba knows exactly what this fear feels like from the summer it barred its own staff from using anthropic's ai tools at work Claude code included and put them on a list of high-risk software developers had found Claude code quietly checking a user's time zone and proxy settings anthropic called it an experiment against abuse and said it would come out alibaba moved its people onto its own tool. So Alibaba threw off its desks a foreign AI that had been caught quietly checking things. About its own model reaching for Beijing, it still hasn't answered. The researchers did the opposite of staying quiet. They put their traces online, with the raw output and a small script, so anyone can run the test themselves. After all of that, Cohere still couldn't say why. Because the answer sits inside a lab it can't see into. We cannot answer whether this is deliberate, it wrote, or a leftover glitch from training. Either the model picked the habit up from the logs of other AI agents running inside Alibaba's own training setups, logs that were never cleaned out, so now it reaches for those addresses the way you'd reach for a light switch in a house you used to live in. Or it's a deliberate way to call home. And the lab Cohere can't see into, the one that trained it, is Alibaba's. Cohere's advice to anyone running it is to treat a model you didn't build like software from a stranger, and to check every action before it runs. In those same days, governments did shut doors, just not on this model. From October 26th, Sweden's government offices, the staff behind the prime minister and the cabinet, blocked ChatGPT and Claude on their work computers. The one AI tool they kept was Microsoft's CoPilot, and only after a security review and tighter settings. Their Director of Digitalization warned that using AI services without a contract could have consequences for our information security. Sweden's tax agency and its National Board for Institutional Care had already done the same. Its enforcement agency is preparing to. An audit at its Transport Administration found about 10 AI services in use there, in what it called a widespread and risky way. Back in February, the European Parliament switched AI features off on its work devices. And Alibaba's chairman has called open source – models anyone can download like this one – Europe's way to an AI of its own. Washington, a week before Cohere went public, ordered every civilian federal agency to stop buying DeepSeek and to pull it out of its systems, along with any product built on top of it. The reason given was that data from US government users could potentially be reachable by Chinese authorities under Chinese law. Beijing's foreign ministry called it over-stretching the concept of national security. Those orders name the products and the services. Cohere is asking governments to say whether a restriction like that also covers the same model, downloaded and running on their own machines.