Experts are sounding the extinction alarm. What are they on about and should we be concerned?

New day, new dystopian fever dream. Renewed warnings from tech experts about the potentially devastating outcomes of artificial intelligence have everyone freaking out (again).

The technology, which simulates human comprehension and action, has developed rapidly over the past few years. AI developers have shifted their attention from generative AI to agentic AI, which is more autonomous and task-oriented. There are concerns that as AI becomes more independent, humans could lose control. According to some industry insiders, there’s a chance it might even end humanity.

Wait, what?!

Last week a former employee of Anthropic and Open AI resigned and spoke out, accusing both companies of acting irresponsibly and “gambling with our lives”. Researcher Jacob Coxon predicted superhuman systems could soon hack anything, but most alarming was the disclosure that people building AI “earnestly believe” it could kill us all by the end of the decade. “No other human activity poses this level of danger.”

Another Anthropic employee, safety researcher Evan Hubringer, agreed. He reckoned the chance AI would “kill all humans” within 10 years was over 10%, noting recursive self-improvement (AI building itself) could happen sooner than expected, leading to a loss of human control.

Their fears are echoed by the industry. A public letter released in July, signed by 1,386 AI workers, including Anthropic CEO Dario Amodei, acknowledged a “real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems”.

Add the hypothetical concept of artificial superintelligence (the yet-to-be-realised scenario of AI exceeding human intelligence) into the mix and things start to get pretty scary.

Hmm, this sounds familiar.

It is. In 2023 industry leaders including Amodei, Open AI CEO Sam Altman, Anthropic CEO Dario and Bill Gates signed a statement asserting that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war”.

Nobel-prize-winning computer scientist Geoffrey Hinton signed it too. The “godfather of AI” caused a collective freak out in 2024 by predicting AI could cause human extinction within 30 years. Elon Musk reckoned the chance of AI destroying humankind was as high as 20%. By 2025, Amodei put the chance of things going “really, really badly” at 25% and Google CEO Sundar Pichai admitted the underlying risk was “actually pretty high”.

Ugh. But has anything actually happened yet?

Yep. The latest grim predictions came amid revelations about out-of-control AI. German site Dse Wiki was hacked by 1,200 rogue OpenAI-modelled agents in May, which hid their actions to avoid detection. (Experts say “vast colluding swarms of semi-intelligent AI” present a major threat.) In July, 700 OpenAI agents attacked the open-source platform Hugging Face. The incident was evidence, OpenAI said, that its models were now “powerful, persistent, and collaborative” enough to exploit security weaknesses and “take dangerous actions that no human directed”. Anthropic revealed its AI models had attacked other organisations. And it’s happened at Meta as well; in August the company confirmed an AI agent hacked an external company during testing. These incidents aren’t limited to the US. Chinese company Moonshot AI’s model Kimi3 also escaped containment.

What does ‘humanity ending’ actually look like?

There are several different scenarios, some more extreme than others.

AI may turbocharge warfare (arguably already happening) and terrorism through bioweapons and military automation. There’s also a heightened risk of nuclear war; in one 2026 study, AI escalated conflicts by threatening nuclear strikes in 95% of simulations.

Then there’s the “gradual disempowerment” theory. Relinquishing control to AI could leave humans vulnerable and dependent. AI could eventually control the critical infrastructure for everything we rely on like food, electricity, money and healthcare. If it all collapsed (either accidentally or intentionally), we’d be a bit fucked.

And if AI decided humans were unproductive and inefficient? It would make logical sense to get rid of us. 

Surely governments have something to say about this?

US senators from both sides of the aisle, including Bernie Sanders and Ted Cruz, have been pushing for legislation to regulate the industry. However, president Donald Trump dismissed calls for controls, describing them as a “SICK conspiracy” against AI and data centres that “only China” is happy about. (China’s ministry of foreign affairs brushed off AI warnings as “fearmongering”.) The only guardrail AI needs, according to Trump, is a “STRONG AND SMART (High IQ!) PRESIDENT”.

In the UK, MPs say AI poses a threat to human rights and requires an independent oversight body combined with specific prohibitions. 

Australian prime minister Anthony Albanese, who announced a world-first AI framework and Office of AI in July, this week acknowledged the “serious risks”, proposing governments and companies take responsibility for the technology.

In New Zealand, Labour shared its AI policy last week and the Green Party is petitioning for a pause on new data centres, which the coalition government wants more investment in. Prime minister Christopher Luxon, asked this week if he was concerned about AI wiping out humanity, said no. He said he thought the opportunities outweighed the dangers, but acknowledged the need for oversight. “We have to work with likeminded friends to put in place the right regulatory frameworks.”

Global agreements on slowing down AI are “really hard to see in the current geopolitical environment”, according to local AI expert Andrew Lensen. But setting up an AI institute could make us “much more prepared to tackle AI harms” – including the “existential” ones.

Are controls the answer?

Tech leaders appear to think so. Altman and Musk backed Amodei’s recent call for the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”.

But many inside and outside the industry are sceptical of a slowdown. The race between Open AI, Anthropic, Meta and Google (and their Chinese competition) is intense. Money has poured into the AI bubble; $US1 trillion is estimated to have been invested globally this year and AI technology accounts for 2% of US GDP.

Hold up, aren’t Amodei and Altman the people developing AI in the first place?

Correct.

So will they stop?

Good question.

Hmm, humanity-ending technology makes these companies sound pretty powerful.

It sure does. Lensen encourages scepticism. “The tech oligarchs may be pushing this narrative for their own marketing, economic gain, and to try and prevent new competitors.” Dire warnings are a flex that may benefit a company’s valuation. Anthropic and OpenAI are both preparing to go public, with expected IPO valuations of US$2 trillion and US$1 trillion respectively. These eyewatering figures are justified by catastrophic predictions, according to Grindr chief executive George Arison. “Maybe they actually believe it,” he told the BBC. “Or you could argue they’re saying it because it’s a great way to gin up more investor support.”

So, what happens next?

That remains to be seen. But once we’ve had the Great Machine Wars we could seek out Arrakis. The Spinoff will probably be live blogging it, so donate now.