Rory Clifford is using VR tech to digitise a marae in Motupōhue (Bluff) and help whānau reconnect with their tūrangawaewae.

Dr Rory Clifford (Kāi Tahu, Kāti Māmoe) remembers taking his computer with him on family holidays, creating a high-tech haven in the confines of the caravan. While other kids were perfecting their manu technique off the wharf near the campground, he was learning new tricks to clock levels and beat bosses. 

“I just really enjoyed playing games most of my life, just getting immersed in the stories. It was almost a bit of an escape,” he says. 

As he grew older and his passion for gaming and tech never diminished, Clifford found a way to make virtual reality (VR) and immersive storytelling his career. His VR work has allowed him to uncover the potential of technology to share, preserve and provide access to mātauranga Māori. It’s allowed him to embed concepts like kaitiakitanga in the emergency services, and on his current project he’s helping hapū stay connected with their marae. 

Clifford’s PhD project in human interface technology, which was created in collaboration with Fire and Emergency New Zealand, used virtual reality simulation to train aerial response firefighters how to deal with life-threatening situations. 

The technology from the HIT Lab NZ (Hangarau Tangata, Tangata Hangarau) at Te Whare Wānanga o Waitaha University of Canterbury, where Clifford’s VR projects are tested and put to use, has been specifically designed to create the multi-sensory experiences that make his work so lifelike. Fans are mounted to a rig that surrounds the space, which can each be spun independently to create atmospheric winds, diffusers are used with different scents to create “smellscapes” such as fuel smells and vibration units attached the seats and floor simulate the feeling of the helicopter rotors spinning, or in other applications, anything from a violent earthquake to the rumble of a concert. 

The effects of this simulated atmosphere means Clifford was able to put the Fire and Emergency Air Attack crew under some of the stresses that they may face when out battling a blaze – without the danger. 

“During a fire they can be under a lot of stress, so [the VR training] really helped them get into that state of mind. I used a heart rate monitor to measure their heart rate variation and it showed that their heart rates were quite similar when they’re using our training system versus out in the real world,” says Clifford.

The PhD study, funded by the Ngāi Tahu Research Centre and a Ngā Pae o te Māramatanga doctoral completion scholarship, was one of the first opportunities for Clifford to learn more about his taha Māori. He says while the research may not seem, at face value, to be based in mātauranga Māori, stopping wildfires is kaitiakitanga in action. 

“In terms of mātauranga Māori and kaitiakitanga, we need to look after the whenua and this is a way to actually show this, training these guys to be able to be more alert and more prepared when they go out and fight the fires, to prevent them from spreading faster.”

Clifford didn’t grow up with frequent access to te ao Māori – the support of organisations like Ngā Pae o te Māramatanga has helped him find his place in te ao Māori. Since completing his PhD in 2020, he’s become even more involved in his iwi and hapū – joining tuna trap-and-release teams to help protect his awa and learning more about tikanga and reo Māori. 

“I didn’t really get that much exposure to te ao Māori or maybe I was a bit too whakamā growing up in a Pākehā space. I just didn’t really get that much opportunity to explore, but going further and further down the track, I thought, ‘No, I should get more into this because it’s who I am’.”

Now, he’s dedicated to using his unique skill set in the VR tech space to help share mātauranga Māori, pūrākau and kōrero in new ways.

“I’m really proficient in computing and that sort of thing and I’m probably one of just a small number of Māori in that space with this kind of knowledge,” he says.

“Where I’m at now is just getting better at my tikanga and my reo and working and just acknowledging what it is to be Māori, understanding those different aspects of kaitiakitanga, manaakitanga and whakawhanaungatanga.”

Clifford’s skills and knowledge have the potential to help others reconnect with their communities. In a new project called Ātea, funded by Science for Technologies and Innovation, National Science Challenge and led by Massey University Professor Dr Hēmi Whaanga, augmented, virtual and mixed realities are being used to share history, knowledge and stories to connect Māori with their language, culture and identity.

Ātea is a joint project between Massey University with Te Pūtahi a Toi, Otago University with the Department of Information Science led by Professor Holger Regenbrecht, Canterbury University with the HIT Lab NZ led by Professor Rob Lindeman, Waikato University with the Department of Computer Science led by Professor David Bainbridge and Associate Professor Te Taka Keegan, and Te Rūnanga o Awarua in Motupōhue.

One component of the Ātea project is working with the Rūnanga and Te Rau Aroha Marae to explore how technology can allow Māori dispersed across Aotearoa, and the world, to connect with their “historical, cultural, linguistic and geographic mātauranga”. 

Te Rau Aroha is the world’s southernmost marae. A Māori hostel was established in the late 1800s to accommodate Māori moving between the islands off the Southland coast. The marae in its current form began its development in 1985, and is now a thriving community space.

In February 2003 the wharenui Tahu Pōtiki was opened, named after their Ngāi Tahu tūpuna. His descendants were pivotal in the expansion of the iwi into Te Wai Pounamu. It was designed by the late Cliff Whiting, an esteemed Māori arts practitioner and teacher (who also designed the wharenui inside Te Papa Tongarewa). Stepping away from the traditional design of a wharenui, Tahu Pōtiki takes its architectural dome-like structure from the wharerau (tītī island huts) used by South Island Māori for mutton-birding. Whiting’s design embraces vibrant colours. 

It’s Clifford’s job as part of the wider project team to help create multi-sensory atmospheric effects for use alongside recorded kaikōrero telling the stories of Te Rau Aroha marae, to further enhance the recorded experiences, and translate them into VR experience that can be enjoyed by people across the motu. The aim is to preserve the stories of the marae, its intricate carved poupou and woven tukutuku panels. 

“It’s got all these amazing carvings, figures and walls that speak to different things about who they are as people, their tūpuna, place, connection and history,” says Clifford.

As well as its contemporary design, the Rūnanga has a contemporary vision for how they want to share the history of Te Rau Aroha with their people. While for a long time for most marae, tikanga has dictated that the online sharing of photos and recordings is prohibited, because Te Rau Aroha is remote and its people are scattered far and wide they have chosen to carve a new kawa and embrace the potential of new technology.

“A lot of people who whakapapa there, no longer live in the region. They’re scattered throughout Aotearoa, Australia and all over the world and we want to be able to connect to them and use new technologies to enable this. The marae saw an opportunity and embraced sharing these technologies with their whānau.”

Clifford says he can understand why a lot of marae have tikanga and rules that stop people from taking photos and recording after seeing the way Māori culture has been appropriated by the entertainment industry. Video game Cyberpunk was criticised for its use of tā moko on characters. 

“Some don’t want you to photograph their tūpuna and all their artefacts, which is fair enough because you don’t know if someone’s going to take advantage of it or monetise it. It happens quite frequently with our culture, even in the games industry.”

Dean Whaanga is Kaiwhakahaere of Te Rūnanga o Awarua, and has worked for years to bring the people of Te Rau Aroha back to their rohe. The recent opening of six new homes on the marae, built to house kaumātua, is one of the projects Whaanga has been working towards for years. He says it’s been an aspiration of the marae to have a papakainga since 2000, and a recent funding from He Kāinga Pai Rawa allowed that to begin.

But for those who don’t have the option to move closer to their marae, the digitisation journey is another way Whaanga is hoping to bring people together to connect with their stories.

“The challenge is how to make it easier for our hapū to engage with their marae if they live outside of our takiwā and how do we make it easier for those that might be whakamā about approaching their Rūnanga?”

Whaanga says the digitisation of the marae achieves multiple outcomes, from engaging young members with the cutting-edge technology, to acting as a storage kete for their mātauranga, and to allow it to be shared easily between generations.

“The use of new technology is extremely important for our people, engaging it for purpose is very desirable. Not one kaumātua I have shared kōrero with has understated the use of technology. Technology and its application grows our Māoritanga.”

Alongside Clifford’s work creating an immersive VR experience of the marae, other researchers have been animating pou so the ancestors depicted can tell their own stories, and explaining the Māori night sky in the VR environment – in te reo Māori and English.

The Ātea project is layered with tikanga and reo expertise and consultation to ensure the mātauranga and imagery that comes from Te Rau Aroha is used authentically and with sensitivity, says Clifford. 

“They want to use it as a way to pull in people who whakapapa there to show them, ‘This is your marae and these are your tūpuna’,” he says. “We’ve recorded some of their kaikōrero so we’ve got kōrero about the eight different walls, and we can show all that in this virtual space.” 

The project aims to deliver to the marae an experience that truly reflects the vision of master carver Cliff Whiting, to share and build on the history, mātauranga and stories it holds, to as many people as possible. It is designed to be an inclusive and inspiring encounter. 

“You can use a VR headset to be able to have a really immersive experience, but you don’t need a VR headset to do it. It’s really to connect with the people who whakapapa there.”

Taking part in this project to preserve the stories of a marae, Clifford has learnt more about his own whakapapa. His personal te ao Māori journey is only just beginning, and he’s grateful for all the support he’s had over the years that’s led him down this path. 

Using technology like VR and immersive video, Clifford is helping to preserve these stories, tikanga and reo intricacies in a way they never have been before. Passing on that knowledge between generations has the potential to create new connections between the past and the future and to help people like Clifford to learn about their taha Māori from their very own tūpuna, from wherever they are.

“Having that exposure is really cool because it solidifies who you are as Māori, and it helps you understand how you should act and the protocols. There’s a lot of things that I missed out on growing up.”