Shifting Soundscapes

“Spatial audio, in its most basic form, emulates how we perceive sound in the real world,” says Brian Yessian, chief creative officer and partner of sound design company Yessian Music. “If we are in a room and someone in front of me is speaking and I turn my head to the left, then my right ear is going to pick up more of the voice and my left ear will [hear] less,” he explains. “Or if something falls behind me, I know to look back because our audio senses pick up on the fact that it was actually behind me.” Spatial audio achieves that effect in VR using software algorithms that manipulate a program’s sound wave frequencies, creating audio levels that become louder or softer depending on the user’s distance from a virtual object. The sound also shifts from one headphone speaker to the other as the person moves their head from side to side or as the virtual objects move on their own.

Makers of VR headsets have begun to embrace immersive soundscapes as a way to improve the quality of the virtual experience. The new Nokia 8 smartphone, for example, includes OZO Audio software that uses the device’s multiple microphones to record video that has surround-sound qualities when heard through headphones. Spatial sound is also a major feature of the Microsoft’s new Windows Mixed Reality head-mounted displays, which go on sale later this month. Mixed reality is essentially a hybrid of VR and augmented reality that relies on cameras and other sensors to integrate digital objects into the real world or real objects to be embedded in a virtual world.

Microsoft already makes a mixed-reality headset with spatial audio called the HoloLens, but that device is targeted at software developers and corporate users. The HoloLens price also starts at $3,000, several times the cost of the consumer-targeted Mixed Reality headset. Microsoft has also made spatial audio part of Windows 10, which means any device running that version of the operating system will be able to play dynamic sound when headphones are used. The new Windows Mixed Reality and HoloLens headsets offer very different user experiences. The HoloLens has a transparent display that projects vivid holograms on top of real-world environments and is a fully self-contained computer that allows the user to move around without being restricted by cables. The new Mixed Reality headset plugs into a PC and has an opaque display similar to that of the Oculus Rift and HTC Vive.

Microsoft already makes a mixed-reality headset with spatial audio called the HoloLens, but that device is targeted at software developers and corporate users
Share on X

Microsoft developed its spatial audio algorithms after studying different ear shapes and the way in which a person’s brain via the inner and outer ears locates the source of a sound in three dimensions. The goal was to better understand how they affect a person’s sound perception. “We’ve also done a lot of work in modeling how audio bounces and reverberates off the environment,” says Noel Cross, Microsoft principal software engineering lead, who has worked on both the HoloLens and the new Mixed Reality devices.

“Different size rooms give you different levels of comfort as a human, and if things don’t match your expectations in terms of [what] they should sound like, you instinctively feel quite uncomfortable.”

Cross and his team took those lessons into account when designing the interface for Microsoft’s new consumer headset. Instead of presenting users with a start menu or screen when they don the device, they begin their experience in a virtual house located on a cliff by the sea, where they are then free to wander its three different floors. Users interact with that setting much the same way they would stare at pictures or TV screens in the rooms of a real house—and for that to make sense, Cross says the developers had to get the sound element just right. Oculus likewise has its users begin in the same location—sort of like a VR home page—each time they use the device.

The Sweet Spot

Earlier efforts at “3-D audio”—a precursor to spatial audio—were very limited in that they were designed for PCs in the 1990s, Cross says. The speakers were typically located on either side of the computer monitor, and the sound was immersive if the users sat in a “sweet-spot zone in the middle” and did not move, he says. “Even with headphones in earlier experiences, as you moved your head, the sounds wouldn’t stay locked in the context of the world but would stay fixed relative to your head,” Cross adds.

Earlier efforts at 3-D audio—a precursor to spatial audio—were very limited in that they were designed for PCs in the 1990s
Share on X

Given that humans are hardwired to pay attention to sound and instinctively use it to map their surroundings, find points of interest and assess potential danger, it is hard to overstate the usefulness of spatial audio for VR. As virtual environments move to the mainstream in education, training and health care—including the treatment of phobias and trauma via virtual reality exposure therapy—users will want to fully engage their senses.

As virtual environments move to the mainstream in education, training and health care users will want to fully engage their senses
Share on X