Cautionary tales from abroad show that when automation and AI goes wrong, it does so at speed and scale – with huge human and financial costs. 

If overseas experiences are anything to go by, the cutting of public sector jobs in favour of AI models will create more problems than it solves, argues Alexandra Sinclair.

The New Zealand government announced last week the removal of roughly 8,500 public sector jobs as part of a bid to save costs and replace many of these roles with AI and automation. Finance minister Nicola Willis has claimed the changes will save the government $2.4 billion over four years. AI is increasingly framed as the solution to the costs and complexities of government. However, this government is likely to find that the cutting of public sector jobs in favour of AI models will create more problems than it solves. 

New Zealand has so far avoided the high-profile automation scandals seen in counterparts such as Australia (Robodebt), the Netherlands (child benefits scandal) and the UK (Horizon Post Office scandal). In those countries the drive to automate for cost savings resulted in the hasty introduction of automated systems. These systems had enormous human costs and were a false economy for government due to the large amounts of compensation paid out as a result of automation failings. 

In the Netherlands, the government of the day resigned after the extent of the child benefits scandal was revealed. Then Australian prime minister Scott Morrison had projected the Robodebt scheme would save the government $1.7 billion. It has cost them $2.4 billion and counting in compensation payouts and has hugely affected public trust in government and in new technologies. When automation and AI goes wrong, it does so at speed and scale. 

It’s also important to contextualise this announcement in light of the fact that New Zealand has no AI specific regulation in place. Canada, the United Kingdom and Australia all have mandatory AI transparency measures for the public service and/or have invested in AI safety institutes. South Korea has recently passed legislation regulating AI. The European Union has an extensive regulatory framework that enables individuals to object and contest solely automated decisions, while also putting significant compliance obligations on AI developers wishing to enter the EU single market. The lack of AI-specific regulation in New Zealand means the public has no individual rights of challenge in the courts for AI-specific harms. 

There is no public mandate to cede huge amounts of power in designing government infrastructure and providing government services to overseas technology monopolies. New Zealand’s data sovereignty context is highly distinct. AI models licensed from overseas are not designed for an indigenous context. There are also significant data sovereignty concerns with training overseas general-purpose models with Māori data. 

We have seen the current government dump its own in-house digital systems crafted for an indigenous healthcare context in favour of private AI scribe providers in healthcare. Last week’s announcement suggests the ceding of further power in designing government infrastructure and providing frontline government services to big tech. New Zealand’s size means it’s going to be a taker of new technologies. It has little power to negotiate – it will have to accept the technology on the terms it’s given, subject to the whims of whatever updates Open AI or Anthropic roll out. We should be concerned about the delivery of government services left to the impulses of overseas technology companies that do not necessarily have the same values or concerns for what is in New Zealand’s best interests. 

Moreover, once these contracts are entered into and companies are providing the digital infrastructure, this lock-in dynamic makes it very expensive for government to exit if it’s not happy. The government has said that AI models will save money on frontline staff, but AI procurement is itself an expensive and ongoing exercise and if prices rise once our services are structured in reliance on them, we will be unable to say no. 

There are many studies casting doubt on the productivity gains of AI in the workplace. MIT found that 95% of companies have failed to find revenue generation from AI use. Other studies have shown that AI scribes offer minimal time-savings for doctors. Unsurprisingly, most studies touting productivity benefits are from the developers themselves. Willis has said the civil service has poor digital systems and data governance. This is a sign that government needs to be cautious when introducing AI.

AI is only as good as the quality of the datasets it is running on. Fast productivity gains are more difficult to achieve in government where dataflows are incredibly sensitive and where information is classified and subject to strict authorisation and permission regimes. If Large Language Models have access to classified datasets, when prompted they can produce outputs making that information available. Without careful data silos, security and data breaches can easily occur. There has already been a serious security breach in March of this year after the rolling out of Heidi AI into New Zealand’s hospitals. 

There are also concerns with the loss of institutional knowledge in the public sector by outsourcing to AI systems. If AI systems are learning from data and processes, then civil servants themselves are often failing to learn. Judgmental atrophy is the phenomenon whereby individuals lose their ability to make difficult fine-grained judgments and assessments because they outsource those judgments to automated systems. 

Efficiency is not the only value of government. The public do not want government making incorrect decisions about them efficiently. The government also owes different obligations from the private sector, as individuals cannot opt out from interacting with them. Values of open government, transparency, fairness and accountability can be impeded by the delivery of government services using opaque and fast-moving AI models hosted in other parts of the world. These are the conversations we need to be having before the government goes ahead with civil service cuts in favour of AI models.