The review of the 2018 census backs up his experience that Statistics New Zealand is a monster with a small but distant brain, argues Tony Burton, a former senior official at Treasury

After the 2011 Canterbury earthquake Statistics New Zealand’s Christchurch office was red stickered, declared too dangerous to enter. This may not sound important when people are risking their lives to save others, but the reality is there are investors making multimillion-dollar decisions that affect our economy and they start getting nervous when all they hear about New Zealand is ruined buildings and death tolls. Evidence the government is struggling to cope may have real consequences for income and jobs, even in areas not affected by the disaster. Routine tasks such as releasing scheduled economic data on time are crucial.

So it became a priority when it was discovered that vital economic data was on a server in the Christchurch building with no readily accessible way to get it. In what sounds like the beginning of a nerdy joke, an engineer, an IT technician and a manager went into this life-threatening situation to recover the server. It’s possible to see the humour now because they suffered no injury, got what they needed and the data was released.

But the story does not end there. You may recall that in the 2016 Kaikoura earthquake Stats NZ’s building in Wellington was one of the worst hit. So once again, having a plan to get the economic data out on time was crucial. They did have a plan, but they were still late.

Statistics New Zealand’s is legally protected to ensure the data it provides is independent of politics. The cost of that independence is that the parliamentarians whose job is to make the bureaucracy accountable have little power to call Stats NZ to account. In my experience the organisation and its leadership have responded with multiple layers of committees that have meetings upon meetings with little purpose. The result is that Stats NZ is, it seems to me, a bit like a brontosaurus: a colossal lumbering body that’s good at the basics of Jurassic survival but with a brain the size of a walnut.

The basics of being a statistics agency are collecting and checking data. Stats NZ’s field staff, the people who do the painstaking work of knocking on doors and helping people fill in the forms, are very good at what they do. Likewise checking data is a hard, extremely dull job. There are many people, me included, who are indebted to Stats NZ’s data checkers. But in the organisation’s distant but small brain there is dissatisfaction with this lowly role. Chief executives and their deputies do not get promoted to high profile policy agencies by doing the boring stuff only data nerds know about. The result is that Stats NZ repeatedly does what it is not good at and does it badly.

Tuesday’s report on the Census 2018 debacle has shown the brontosaurus at work. So desperate were the reviewers to find nice things to say about Stats NZ that when looking for “what worked well” they came up with “NZ Post was able to deliver invitation letters to almost 80% of households (up from the original target of 70%)”. Stats NZ asked the post office to deliver letters and the letters were delivered. Well I never.

But the reviewers are also honest and thorough. For instance, how were those field staff handled? “Compounding an already aggressive reduction in the field workforce, the workforce target was further reduced to 2,300.

“The decision appears to have been driven by a mathematical model without careful consideration of respondent behaviour. We could not find any evidence that this issue was raised within the governance structure and that options or risks were considered.”

The head of the monster tried biting off its own legs but in its small and distant brain no one noticed this might be a problem because “the programme governance was overly complex and ultimately ineffective in guiding the programme to a successful outcome. The leadership at the programme level lacked strategic direction and effective programme management.”

In itself the review is great. The public service response, not so much. Government statistician and Stats NZ chief executive Liz MacPherson has stepped down, but she was merely unlucky enough to be the boss when the most public of many, many Stats NZ initiatives was mishandled. In my view she has sacrificed herself for the greater complacency of the state services commissioner.

The Stats NZ website camouflages its failure behind a forest of initialisms: the New Zealand Progress Indicators (NZPI); Indicators Aotearoa New Zealand (IANZ); the slow and clumsy introduction of the Integrated Data Infrastructure (IDI); SoFIE. And so on.

The last was nearly a decade ago and gives a flavour of the common or garden Stats NZ debacles that you rarely hear about outside the small community of expert data analysts. SoFIE, the Survey of Family, Income and Employment, was run between 2002 and 2010. It was a longitudinal survey that followed the same people over time and is the only way to properly understand issues like social mobility and retirement income where we need to know what happens in families over many years.

Ministers of all parties have been enthusiastic about these studies and the initial funding for SoFIE was for eight years. Unless you intend to build roads, hospitals and the like, that is as secure as funding gets in government.

Yet in 2011 SoFIE’s funding was not renewed. A comparison with SoFIE’s Australian cousin HILDA makes it easy to understand why. While fewer than 50 studies in total use SoFIE, more than 100 used HILDA in 2018 alone. Remember, this is the data that helps us understand long term poverty, whether education really gives New Zealanders a “fair go”, what matters in the long run for staying healthy, and whether we are saving enough for retirement.

This gap between Australia and New Zealand is a gap in understanding what matters most to New Zealanders. The SoFIE equivalent of under-recruiting census staff was making life difficult for the analysts who turn data into information that makes a difference in people’s lives.

Trinh Le of economic research agency MOTU, one of the superhumanly persistent people who did manage to use SoFIE, believes it was superior to HILDA in some ways, particularly its more inclusive sampling and at times more in-depth coverage.

However in 2010 Stats NZ brought in charges for using SoFIE – $95 for a half-day per terminal in the data lab, and $115 an hour for checkers to check the output. “Research usually takes years, so this added enormous cost. So many projects had to be cancelled due to this,” Le says. One Marsden Fund project had to switch to using HILDA because the data lab costs were unaffordable, she says.

Aside from the de facto research taxes, even the basics of SoFIE were “raw and messy”, as Le diplomatically puts it.

The age data was inconsistent, for example. There was no question on “who is your partner” in a household, so Le was forced to identify couples herself based on cross-person relationships. Little wonder she sometimes ended up getting a teenage male paired with a 50-something female.

In comparison the HILDA data is well organised, comes pre-cleaned, and is available at a trivial cost on a DVD.

The review of the very public failure of the 2018 census is an opportunity. When James Shaw, the current minister of statistics, talks about his role he has a palpable interest and knowledge, and he is just the latest example of the striking interest ministers of all colours have shown for making more use of available information. Unfortunately, the gentle evolution that a minister of statistics can initiate, however enthusiastically, is not enough. It was not gentle evolution that got rid of the dinosaurs.