Serious people are putting troubling odds on the machines wiping out the humans in the near future. New Zealand needs a bit of work on its own AI policy, reckons one expert.
Across recent days the warnings about the risks of artificial intelligence have reached an unprecedented scale and breadth. The particular worry, stemming from a post by an AI researcher who quit his job at AI giant Anthropic, is, well, the wipeout of the human race.
The alarm bells are ringing almost everywhere, in the wake of startling derailments such as the Hugging Face hack, in which AI agents went “rogue”, with their commitment to completing tasks leading them to defy instructions and leap out of their virtual sandbox.
A growing parade of people in the AI world are offering estimates on the risk to humanity of a technology approaching “recursive self-improvement”. Even the bosses of the American AI giants are asking for more regulation – calls that have in turn generated suspicion from some that they might be motivated as much by consolidating their own strength and attracting investment as anything else.
Donald Trump, for his part, has denounced a “hoax” and “conspiracy”, saying the introduction of stronger regulation would only play into China’s hands in the race for technological advantage. “The only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades!” reflected the president of the United States.
For the mere mortals among us trying to process the existential risk, it’s worth taking a breath, suggested AI expert Tom Barraclough, speaking to The Spinoff for an episode of At Large with Toby Manhire.
When an AI agent goes off-course, such as in the Hugging Face example, “it’s really important not to get away from the fact that somebody started this process”, said Barraclough, founder of New Zealand’s Brainbox Institute and tech startup Syncopate. “Somebody, somewhere hit the enter key, and triggered this training run. And before they did that, they made a bunch of decisions about what models they were going to use, what resources it had, what those safeguards were going to be.
“Somebody made a call about the supervision mechanism. Somebody made a call about how they were going to log what was happening and what the agent had access to in terms of changing those logs. There’s a really long and complicated process involved to setting off something that could do massive catastrophic harm. So it’s important not to get away from that and distract from the accountability of the people, the companies et cetera who are doing this kind of thing.”
That wasn’t, however, cause to quit worrying. “At the same time, you don’t have to go too far [to get to] the humanity wipeout thing. All you have to do is just think about the capacity of technology and computers as they are now, right? And then think, well, this thing is really good at doing stuff with computers … If we still lived in this agrarian society where we didn’t really interact with digital technologies that much, and we were mainly using hand tools in the fields, the fact that AI was really good at computers wouldn’t matter so much. The fact that computers run through our health system, our infrastructure, our cars, our communication systems, our banking systems; that is what creates a huge amount of risk.”
In the face of these gathering storm clouds, New Zealand is hardly equipped to sit at the vanguard of global regulation, but there is still stuff we could do, said Barraclough. “The first and most important thing is to equip people and businesses in New Zealand to understand this stuff and make informed decisions about it,” he said. That was in many ways a literacy challenge, and could be as simple as understanding the difference between a web-linked or isolated AI chatbot.
At a wider level, there was a need for “some level of national accountability and coordination”, he said. “These are really serious technologies … You can do incredible things with them. It’s going to be really hard to anticipate what impact that has for us. So I’ve explored that through the lens of what’s called sovereign AI.”
Sovereignty in this context is not so much about a “Kiwi GPT”, he said. “I think that’s unrealistic.” It was more about questions like “do we have enough computers onshore to run these things, such that if all of our internet cables were cut, we wouldn’t immediately lose access to all of the AI?” Then there was the “broader digital infrastructure, like the data centres, the cables, the power grid – and part of that is also standards. So a big part of making all this work is the ability to make all the systems talk to each other and have some level of alignment and compatibility between different organisations.”
From there New Zealand would be well placed to “play an outsized role in standards development” across borders.
While there was a pool of people within the public sector “who are doing extraordinary things with AI”, said Barraclough, that did not visibly ladder up into the top tier of officialdom or politics. “I’m a little bit worried that I can’t point to the MP who can keep up with it,” he said. “I really want to see the leaders politically, in business, in the public service, in the community, who are leaning super hard into this and really want to dive into the detail.”
What would he like to see politicians in an election campaign discussing? “The question is how are we going to set up all of our systems that surround the AI such that we can use the AI reliably and responsibly,” he said. “So that’s thinking about what we’re doing with our data to set it up and make it available and structure it.”
As for the portents and risk appraisals of the last week, Barraclough said he was disinclined to dive too deep into the doomsday pond. “I’m optimistic,” he said. “Because we’re having the conversation now. It’s not a linear trajectory. Something like this happens, we all have a chat about it, and we course-correct. So, I hope we can continue to do that. My optimism rests on – and I think this is what we really need to think about – our ability to help people and organisations to use the technology well and understand what they’re doing with it… and thinking about how we protect all of our systems.”
For more from At Large with Toby Manhire, follow here for Spotify, or here for Apple. For YouTube, subscribe to the Spinoff here and find all the episodes here.
