Which public entities can we trust to manage our information? Not as many as we should, says Samson Samasoni, who finds that many are badly failing to meet their legal obligations.
We trust government organisations to properly manage our information and the materials they create, collect and store. But information management (IM) audits of 39 public entities reveal poor management, outdated processes, under-trained staff and, in a few cases, staggering incompetence.
There’s the polytechnic that can’t detect if students access a staff-only collaboration and data-sharing tool, the government department that has no idea what records are being held in an off-site storage facility, the anti-doping sports agency that was operating on Microsoft Office 2008, and the state-owned broadcaster the chief archivist isn’t confident can transition to the government’s new public media entity without putting records at risk.
But that’s only a sample of the information management non-compliance and damning comments scattered throughout 800-plus pages of audit reports compiled so far.
After an exhaustive review of audits by Deloitte and KPMG, it’s clear most agencies are failing to meet even the basic standards set by Archives NZ or the requirements of the Public Records Act 2005 (PRA).
While some of the obligations might be considered technical, the guard-rails implicit in the PRA are there to protect the information held by public entities on our behalf, to promote transparency and, when needed, hold governments to account.
More than 140 entities are being audited between 2021 and 2025 and, if the 39 completed to date are anything to go by, we have every reason to be worried about the way our individual and collective information is being managed.
Keeper of records
The PRA requires the public sector to create and maintain full, accurate and accessible records of the affairs of government. When it was introduced in 2005, it was largely to reflect changes in technology, legislation and record-keeping practices.
Public organisations were given a five-year grace period to become compliant with the new record-keeping standards and audits by 2010. Twelve years on, many appear no closer to achieving even the minimum standards.
As a legislative troika, the PRA, the Official Information Act (OIA) and the Privacy Act (PA), have become a powerful combo of systems for accessing and managing government information. But of the three, the PRA is the one we probably know least about.
Think of it this way: the OIA is the powerful one journalists work with to uncover errant behaviour, the PA is the guardian that makes sure our personal information is being handled properly and the PRA is like the bow-tied librarian waving at us from a distance – we know he’s there but we don’t really know what he does.
Even the titles of the roles associated with the acts evoke different energies. For the OIA you go to the ombudsman, the PA it’s the privacy commissioner and the PRA it’s the chief archivist.
The record-keeping audit programme that started late 2020 is certainly bringing the chief archivist to the fore, especially for agencies who are falling behind in their obligations.
The chief archivist’s official reports are littered with warnings about New Zealand’s public record-keeping “lagging behind”, showing “little improvement or even regression” since 2010, and that there’s “no excuse for poor record-keeping in the digital age”. The frustration is obvious.
Acting chief archivist Honiana Love, who is also chief executive of Ngā Taonga Sound & Vision, confirmed to The Spinoff that many agencies audited do not meet Archives NZ’s expectations and that it’s “certainly frustrating”.
That frustration was clearly felt during the Royal Commission Inquiry into Abuse in Care, which highlighted why effective public record-keeping is so important.
“For survivors, past government records have often obscured or misrepresented information about their identity and family connections,” said Love. “Records often lack evidence about how people were treated and why. Poor records management and the resulting gaps in records, hamper survivors’ recovery and their ability to seek redress and to hold government to account.”
Measuring mediocrity
The chief archivist independently leads the regulatory framework for the creation, management, disposal and preservation of public sector information and records. In practice, the framework includes an audit of public record-keeping, an Information Management Maturity assessment and an annual survey of the public sector.
The latest survey found that while the number of governance groups established to oversee information management (IM) within agencies had increased, at least seven large organisations had few IM specialists and “were at greater risk of mismanaging their information”.
Thirteen percent of the public entities had not yet identified the “high-value or high-risk information” they held and half were still in the process of completing that work.
A total of 258 agencies were asked to take survey, only 216 responses were valid (others were late or incomplete) and 33 didn’t respond – 15 public offices and 18 local authorities. Callaghan Innovation, Television New Zealand, the NZ Parole Board and Department of Conservation were amongst those that didn’t respond. The NZ Qualifications Authority, Parliamentary Commissioner for the Environment and the Hawkes Bay District Health Board were amongst those deemed invalid due to lateness.
The minister of internal affairs has recently written to ministerial colleagues asking them to encourage their agencies to complete the survey if they had failed to do so last year.
The IM Maturity assessment is a tool Archives NZ uses to measure and audit where organisations are at in terms of 20 topics, such as whether they have an IM strategy or appropriate storage arrangements.
An agency’s IM maturity for each topic is then assessed at five levels: Beginning, Progressing, Managing, Maturing and Optimising. To broadly meet the minimum requirements expected by the PRA and Archives NZ’s mandatory information and records management standard, they have to achieve at least ‘Managing’ for all topics. None of the 39 audited accomplished that, some only achieving Beginning or Progressing levels.
Victoria University information studies senior lecturer Dr Maja Krtalić said it’s concerning that some public agencies are only at the beginning stages of their IM maturity assessment, as the PRA was introduced many years ago.
“It is important for agencies and the public to recognise that this issue matters,” said Krtalić.
“The PRA gives clear reasons on why it matters: among other purposes, to enable the government to be held accountable by creating and maintaining, preserving and providing access to full and accurate records of the affairs of central and local government, especially to records of long-term value that are relevant to the historical and cultural heritage of New Zealand and to New Zealanders’ sense of their national identity,” she said.
“Isn’t that a goal worth pursuing?”
Agency audits: the results
Drug Free Sport New Zealand, which was established in 1994, is one example of an organisation at the beginning stage. The agency, responsible for keeping New Zealand sport clean and free from doping, was at the Beginning level for 16 of the 20 topics in its audit (pictured). It didn’t achieve Managing or better on any.
The March 2021 audit found the anti-doping agency’s records were managed on an outdated, shared network drive using the Microsoft Office 2008 suite, which is no longer supported by Microsoft. At the time, the organisation had no plans to upgrade its software. The chief archivist concluded its overall maturity “is too low and needs to be addressed”.
DFSNZ chief executive Nick Paterson told The Spinoff they’re actively addressing the report’s findings and are intent on building up their IM maturity levels, although Covid had affected their implementation timeline. He says they’ve now recruited a chief information officer who is currently developing an IM strategy and associated policies, and they’ve migrated from Microsoft Office 2008 to Microsoft 365.
*
The chief archivist was more critical of state-owned broadcaster RNZ. In a September 2021 letter, the chief archivist Stephen Clarke wrote to RNZ that its low maturity rating “is disappointing and reflects an apparent lack of commitment to IM improvement”. RNZ had no topics that achieved Managing level or above – 15 were at Beginning and five at Progressing.
RNZ were found to have no IM policy-and-process documents, which is why staff and managers didn’t understand their IM roles and responsibilities beyond what was needed for their day-to-day jobs.
While RNZ performs an annual legislative compliance survey, it doesn’t make any reference to the PRA and there was no other monitoring of compliance with the act or the Archives NZ standard. As a result, the person at RNZ overseeing this area, “doesn’t know if RNZ is complying with these legislative requirements and cannot take corrective actions that may be needed to address possible non-compliance”.
The chief archivist was also concerned about whether RNZ could successfully transition to the government’s new public media entity without putting records at risk.
“Your low IM capacity and maturity does not give assurance that any potential risks to IM will be managed within a complex environment that includes the business case development for a proposed new public media entity,” he wrote to RNZ chief executive Paul Thompson. The government has appointed the establishment board overseeing the new public media entity that will incorporate TVNZ and RNZ, with plans of getting it operational by mid-2023. The TVNZ IM audit isn’t scheduled until 2025.
In response to written questions submitted by The Spinoff, RNZ says it fully understands the value of the public information it holds and the significance of the effective management of this information for Aotearoa New Zealand.
RNZ has prioritised steps to improve its performance and has established several policies and practices which will form part of an organisation-wide information management strategy. An internal public records governance group has been convened. RNZ says high priority has been given to investment in the development of a new digital content management system (Pātaka) which has now been launched.
RNZ did not directly respond to a question about the chief archivist’s concern about the risk of records being lost in the transition to the new public media entity. But it believes its new policies and protocols “will give Archives NZ greater comfort” that RNZ has the essential elements of an IM strategy in place to meet accepted standards.
The Ministry for Pacific Peoples (MPP) was another agency that received a “disappointing” comment from the chief archivist. Its audit had no topics assessed at Managing level or above – 18 were at Beginning and one at Progressing. It was exempt from one topic.
The MPP audit released in June 2021, found that staff were aware of their IM roles and responsibilities but it was from experience at other public sector offices, not guidance or training they’d received at MPP.
Most MPP staff were creating and capturing information in uncontrolled environments, storing material on their own OneDrive rather than the ministry’s enterprise content management system, making it difficult for staff to access information when needed.
The auditors reported that no staff had access to the register of physical information being held by a third-party, off-site storage facility. “It is not known what information is being held in this facility, when it was last accessed or how long it has been there for.”
The ministry reported it was confident its outsourced IT function is taking backups of key digital systems but wasn’t sure how regularly this happens or how long such backups are kept.
Interestingly, MPP was not assessed on Te Tiriti o Waitangi topic, as it was not applicable – unlike every other public entity, it doesn’t hold information of importance to Māori or related to Te Tiriti o Waitangi settlement agreements.
The chief archivist advised the ministry it has “significant work” to do to improve its IM maturity and it was disappointing that MPP has never had “an approved disposal authority, which is a mandatory requirement for all public offices”.
MPP were unable to come back to The Spinoff with a response by the publication deadline. Its now treating the media enquiry as an Official Information Act request, which allows agencies 20 working days to respond.
The NZ Film Commission was another agency that had no topics that achieved Managing level or better when the audit was completed in April 2021.
The audit found that while the commission understands its obligations, it doesn’t have IM staff and lacks the internal capacity and capability to maintain adequate day-to-day IM practices. “NZFC has relied on the best efforts of individuals at the agency to maintain information and records.”
Historically NZFC had a records management assistant role that was vacant at the time of the audit. The commission opted to use specialist professional service firms when it needed help with IM practices.
NZFC staff told auditors there was rarely a handover of documentation or knowledge to new staff when key individuals left; it was up to the new staff member to reverse-engineer the processes or make new processes.
When asked for this article whether the Film Commission had failed to abide by the Public Records Act 2005 and Archives NZ’s mandatory information and records management standard, acting chief executive Mladen Ivancic said the commission “accepts and acknowledges the 2021 Public Records Act Audit Report carried out by Deloittes” and its recommendations.
He said the NZFC is taking “an active approach to hiring staff with record management experience into the ICT team”.
Ivancic said the current CEO, David Strong, started after the report was published in April 2021 but his updated goals and objectives include developing capability and capacity and improving IM practices. (It’s understood Strong is currently on “special leave” pending an internal review, as reported by The Spinoff)
According to Ivancic, the commission is continuing to improve its systems and staff training, and has engaged an external vendor to provide expert guidance with the review and development of SharePoint, disposal authority, governance and an IM strategy.
Ivancic says the commission, “like many others, experienced changing work practices, staff turnover and difficulty in recruitment due to the pandemic and a tight labour market”.
At the Waikato District Health Board, the corporate records team reports to the service excellence manager for information services – but none have information management expertise.
The audit, which was completed in July 2021, noted that the executive sponsor overseeing IM and the corporate records team had raised issues around capacity previously. “However, there is no plan to address the capacity issues due to limited funding.”
Worryingly for a DHB, it didn’t achieve Managing or better on any topic – eight were at Beginning and 11 at Progressing. They weren’t audited on the “information maintenance and accessibility” topic because the auditor’s fieldwork was interrupted by a cyber security event at the DHB. Stuff reported in November that 5% of the DHB’s workstations were running out-of-date software susceptible to viruses and malware, shortly before the cyber-attack paralysed services at its five hospitals in May 2021.
The audit also found that the DHB has no formal management plan for high-value or high-risk information assets. The auditors noted there was “a risk that when staff leave Waikato DHB, knowledge of specific high-value or high-risk information could be lost”.
In response to written questions from The Spinoff, Waikato DHB acting executive director for digital enablement, Garry Johnston, said it was confident its IM practices are aligned and compliant.
Technically that’s correct – the audits measure IM maturity not compliance – but Archives NZ says agencies should achieve at least Managing level to “broadly meet the minimum requirements” expected by the act.
The DHB also said the corporate records team had “significant experience and expertise related to information management”.
Johnston said the DHB had also hired a digitisation and records manager and that records management functions are being consolidated to provide improved focus and support for staff and organisational record management objectives.
*
The New Zealand Growth Capital Partners Limited (NZGCP) is a Crown entity company that partners with other investors to invest in early-stage New Zealand companies, through the Aspire NZ Seed Fund and the Elevate NZ Venture Fund.
NZGCP was another entity that risked high-value or high-risk information being lost when staff depart because it hadn’t formally identified or developed a management plan for those assets, including shareholder agreements, investment funding records and historical investments. In its audit, the Crown entity had three topics at Managing level but the rest were either at Beginning or Progressing.
Wellington-based tertiary education institution Whitireia and WelTec had six topics at Managing level or better, so didn’t require a stern note from the chief archivist. Instead, they were given 43 recommendations that would help improve their IM maturity.
One recommendation from the auditors was in relation to the polytechnic not being able to monitor or detect unauthorised access to any of the sampled systems. So if a student accessed a staff-only SharePoint site – a web-based collaborative platform that integrates with Microsoft Office – it wouldn’t be detected.
Even the powerful Te Kawa Mataaho Public Service Commission, the organisation that provides leadership and oversight of the public service, wasn’t spared. The commission had seven topics that were below Managing. Most information that is over 25 years old is meant to be moved to Archives NZ, but the commission hasn’t transferred physical or digital information since 2014.
Ironically, Te Puni Kōkiri (TPK) Ministry of Māori Development only achieved Progressing for the topic Te Tiriti o Waitangi. As the principal advisor on Māori wellbeing and development to the government, all information held by TPK is of importance to Māori. However, TPK hadn’t identified which information would be considered high-risk or high-value.
Comparatively, the Financial Markets Authority (FMA) was in a league of its own. Only three topics were below the Managing level in its audit, with half assessed at the top Optimising level. The FMA was found to have a comprehensive knowledge management strategy and the chief archivist even wrote to the FMA’s chief executive Rob Everett that he was keen to showcase its successful approach to other public offices.
Ranking the agencies
Acting chief archivist Love told The Spinoff there is a “real risk with agencies that have a low IM maturity” that they don’t have proper control, security or management of our individual records or interactions with public entities.
“To manage this risk, agencies should address our IM concerns alongside other key requirements such as those made by the Privacy Act and the protective security requirements,” said Love.
The record-keeping audits give agencies clear recommendations for improvements. They’re required to develop follow up plans and to keep Archives NZ informed of their progress.
To get a sense of how the agencies compare in their maturity, The Spinoff has created a league table based on points allocated against the levels they achieved (1 = Beginning, 2 = Progressing etc). However, they’re only measured against 17 topics, not 20, as three agencies weren’t assessed against some topics. Nevertheless, we believe the table offers a useful illustration of how the 39 public entities rank.
The agencies that scored the top five marks were: FMA, Public Service Commission, Broadcasting Standards Authority, the Takeovers Panel, Kordia Group and the Ministry of Transport.
The organisations that scored the five lowest marks included: NZGCP, Animal Control Products, NZ Film Commission, Drug Free Sport NZ, RNZ, NZ Fish and Game Council, NZ Symphony Orchestra and the Ministry for Pacific Peoples.
When asked to respond to our ranking system, the acting chief archivist directed The Spinoff to a summary table of agency maturity (page 19 of the chief archivist’s annual report on the State of Government Recordkeeping 2020/21). Love said the table gives the public an immediate visual impression of overall agency maturity but also adds more detail on the range of maturity levels achieved across topics, than would a single grade or ranking.
“Ordering the agencies by maturity level in our table, rather than chronologically by audit date, may enhance the value of this table in the next report,” said Love.
All carrot, no stick
Given many agencies are essentially committing an offence by not meeting their legal requirements, the obvious question is whether the chief archivist has sufficient powers to penalise wayward agencies. The act specifies an individual can be fined up to $5,000 and in other cases, a fine up to $10,000.
Love said it’s “possible that PRA offences” could occur within agencies.
“To date, however, we have not used the offence provisions of the PRA, preferring to focus on other regulatory tools like standards, guidance and audit that support general improvement in IM across agencies,” she said.
“Prosecutions under the PRA need to occur within six months of the alleged offence, which may not be discovered in time,” she continued. “Archives New Zealand also follows the prosecution guidelines for government agencies and our own department. These include criteria of whether prosecution is backed by sufficient evidence, has a reasonable prospect of success and is in the public interest. This is a high bar and, as noted, we consider other regulatory tools are more effective.”
Love said the offence provision was considered in a 2018 case involving the chair of the RNZ board, withholding a voicemail message from a parliamentary select committee and incorrectly asserting that it was a personal record, not a public record.
Dr Krtalić, the information studies lecturer, said the consequences of not complying are not just penalties defined in the PRA, “but also the impact on business continuity, loss of reputation and social accountability”.
“Organisations should comply to enhance public confidence in the integrity of public records,” she said. “The key to compliance is about making information culture part of organisational culture and seeing it as a process rather than a one-time exercise.
“Good information management is often invisible,” she said. “It becomes noticeable when things go wrong.”