Jesse Austin-Stewart is using the unlikeliest of instruments to make music more accessible to the hard of hearing and Deaf community.

You can make music with just about anything. If it makes a sound, you can make music with it. Turns out, if it vibrates (which is also sound, obviously), you can also make music with it. Yes, that includes the controller to a big, lumbering behemoth of a console called the PlayStation 5.

Jesse Austin-Stewart, who recently completed his PhD at Massey University and is one recipient of the Arts Foundation’s 2022 Springboard Award, has been making music with just that. His recent project, Music for PlayStation, funded by Creative New Zealand, turns the innocuous-looking DualSense controller into an instrument for people not just to experience music, but to make music in their own right.

Austin-Stewart was actually doomscrolling when he came across an article about how the PlayStation DualSense controller was able to be used as an audio output. In games, it’s mostly used to provide secondary or tertiary feedback to the player – a tinkle for completing a task here, or the moan of a zombie there. Austin-Stewart actually happened to have a PS5 (still a hard thing to get your hands on), so he decided to give it a try.

“I plugged it into my computer, set it up to play some music off Spotify and it worked, it was a thing,” he says. It worked fine when he plugged his headphones into the controller, complementing sounds well, but the actual output from the controller was lacking. “A lot of the high frequencies were just really quiet, and any low frequencies every now and then would come through quite strongly. It wasn’t representative of what you might expect when you think you can play your music back.”

He did a few tests with the controller and found that certain frequencies vibrated much more strongly than others. “With those heavier vibrational frequencies, you could do some really cool things, like moving the vibrations across the controller and also varying the volume of those vibrations to make them louder, or softer.”

As it turns out Austin-Stewart had experience with vibration haptics – the other output of the DualSense controller – as he’d been exploring it within his PhD research. The most common form of haptics we experience is probably via our phones – those vibrations that notify us of a call or a text message. Gaming controllers are probably the next most common, having been a common feature since the 90s. 

During his doctorate, Austin-Stewart created a work called Spatial Variations, a musical interface that audience members could rest their arms on and feel it it vibrate along different points. It was a particular success with the hard-of-hearing and Deaf community, removing barriers they’d encountered with spatial audio. However, that interface was expensive to make and required technical expertise to set up, so providing audio access to the Deaf community remained a challenge. 

That focus on accessibility has been key to Austin-Stewart’s PhD. Specifically within the context of electronic art music – generally restricted to universities and art spaces – he wanted to see how artists could make creative works that didn’t exclude people based on factors like education, socio-economics and hearing ability. 

As part of his PhD he engaged with hard-of-hearing and Deaf musicians and composers to learn how to make spatial audio more accessible to them. Looking into haptic works became a big focus. 

Making Music for PlayStation was actually less complex than it sounds, according to Austin-Stewart. In this context, the controller is basically an instrument, so it’s no different to plugging, say, an electric guitar into a computer via USB. “You plug the controller into your computer, do a few clicks to set it up, and then, if you’re familiar with making music in an audio programme like GarageBand or Logic, you can make music for the controller in the same way.”

“You can create whatever you want to go out to the controller and then the controller will play it.”

He admits that it takes a bit of fine-tuning to figure out which frequencies are going to be the most useful, but if somebody has basic knowledge of an audio workstation, they should be able to get started fairly easily. The main barrier to access is, of course, actually getting a DualSense controller – they go for about $90, which is still significantly less expensive than other haptic interfaces for making music. (Perhaps ironically, it’s much easier to get your hands on a DualSense controller than an actual PS5, two years after that console’s release.)

“Often those things are quite customised, novel and require a significant amount of technical expertise to be able to use. This controller allows a whole lot more people to engage with this type of music making.”

With a project like Music for PlayStation, however, Austin-Stewart says it’s about looking at an existing arts practice and figuring out who is currently excluded, and how to include them – not just tokenistically, but in a way that brings everyone onto the same playing field. It fits nicely into the body of work he has been setting out, which includes a project next year called Waterfront Monophony. “There’s going to be a bunch of performers with Bluetooth speakers, and they walk back and forth along a stretch of the waterfront, playing multi-channel works by a bunch of composers.”

“Because there’s no seating position where you’re supposed to listen to it, everybody has a really different experience from each other. There’s less of an intended experience, which means there’s less likelihood of someone feeling left out by not being able to experience something in one way when it’s meant to be experienced in a certain way.”

With Music for PlayStation, however, he hopes that people are just able to have a novel music experience. “I also hope it’s a way that people can see how to engage with music not just as an audience, but as makers,” he says. “I hope people can latch onto it and realise they can make music for this controller.”