We designed this experiment for Meta’s Quest headsets as well as Apple Vision Pro - not only were we keen to develop one of our first apps for the Vision Pro, but this approach also served as a useful exercise in understanding the capabilities of each headset and how we could approach parallel workstreams.

‍

We used room mesh to allow users to anchor the virtual strings to their environment, and hand gestures to enable them to pluck the strings and generate sounds. The audio is procedurally generated based on the length of the strings themselves, as well as how the user chooses to interact with them - a light pluck generates a short note, while pinching and pulling the string back further before letting go generates a larger audible reaction with a more varied note range. We also introduced an open palm gesture to allow the strings to be treated more like a theremin - controlled without physical contact.

‍

It was crucial that we presented our mixed reality instrument as users would expect it to behave in order to have the most realistic experience possible, including making sure it sounded consistent and satisfying throughout the experience - for instance, as with any stringed instrument, it only makes a sound once a string is plucked and released. Our team learnt a lot about music theory in the process. Once again, we used Bezi for early prototyping - even though it doesn’t have specific functionality to integrate sound, it did help us validate that we needed to keep our string interactions close to real world expectations - eg pluck, pull, grab. We created a separate application to test what kind of audio reactions we could generate based on our input parameters in order to get it just right.

‍

Applying spatially mapped VFX to the experience allowed us to visualise the audio outputs as users interacted, with rippling currents generated from each interaction. These ripples are mapped to the room mesh, revealing a user’s room geometry in a fun and immersive way. It also worked as a nice counter to the physical environment impacting the audio - in turn, the mixed reality audio visualisation transformed the appearance of the physical environment.

‍

We loved the idea that we could use the size and complexity of the room mesh to impact the resonance of the tones as well - for instance, a long hallway would generate more echo and reverb compared to a small, cube-shaped room like a living room. We were able to apply this to the Quest build, although the Apple Vision Pro (at the time of prototyping) did not support this feature.

‍