We’re abseiling without a rope on AI. There are no safety checks, no containment strategies, nothing. The fetish for the new has got capitalism in its thrall and we all may pay the ultimate price. This is from James Massola, Sydney Morning Herald Chief political commentator.
To be in San Francisco, one of America’s greatest cities, right now is to be steamrolled by the artificial intelligence wave that is breaking over the world.
Cruise around the San Francisco Bay Area for a couple of days and the advertisements are selling the benefits of AI – from “right-sizing” your business (sacking humans and hiring AI) to curing major diseases, smarter investing through to soft hacking government.
The dystopian version is that once frontier AI models (the ones that are most advanced, like those made by Anthropic and OpenAI) achieve “full” recursive self-improvement [RSI] in the next year or so, AI will build better, more sophisticated versions of itself without any human assistance or oversight at all (AI already does most of this work itself, but not all of it).
The pessimists suggest there is a 10-50 per cent chance that humanity will be wiped out as a species in the next couple of years. Yes, really.
As one of the AI safety experts put it: “if you have things you want to do in your life, do them by 2027”.
The US is several months ahead of China in AI development. It’s also the wild west of regulation. Thankfully there are people sounding the alarm, including Karl Berzins, an Australian engineer who co-founded the SF-based FAR.AI, a not-for-profit focused on safety and heading off the worst of what could be coming down the pipe.
“My worry is whether we can move fast enough. Australia traditionally likes to play it safe and that just won’t work here.”
Talk to the pessimists, and all bets are off as to whether humanity itself will survive once RSI is achieved – because once it’s achieved, the AI agents we currently ask to plan our trips overseas, choose a recipe or help with work will move from coding to artificial general intelligence and then on to superintelligence.
At that point, in the same way we hardly ever notice when stepping on or around an ant, humanity’s survival depends primarily on whether it suits the most advanced AI models to look down for a moment while passing us by.
At this point, you probably think I’m insane. Allow me some context. I checked in with several sources in the Australian government’s intelligence community during the course of the week and was reassured (I think) that they believe humanity has until about 2030, rather than 2027, to sort out this existential threat and head it off. But they agreed the threat was real and imminent.
Some AI organisations are so worried about the power of the tech, they’re running tabletop exercises that war game what would happen if, for example, a powerful AI model accidentally revealed the location of every submarine off the west coast of the US, both enemies and allies, one day soon.
AI agents – that is, about 1200 copies of the company’s AI models – could cure all major diseases that afflict humanity in the next few years. Or they could end humanity. This is the breadth of the metier we are discussing.
The break-out Hugging Face cyber hack was against the instructions the AI had been given by its human operators; the desire to complete the task outweighed human instructions not to break out of the sandbox.
We are nowhere near prepared. You don’t get laughed out of the room for raising this stuff in San Francisco even though it seems like the realm of science fiction.