Mixed Reality creates a magical soundscape at South by Southwest.
By Laura Kobylecky
The couch is covered with a mysterious grass. It sways slightly. It glows, beautiful and ominous. A large bell-shaped plant is growing from a small table. It grows under my grip. Everywhere I turn the world is covered with this plant matter. A sound that is both beautiful and alien hovers all around me, changing slightly as I move.In a side area, there is a substantial line waiting to try out Magic Leap Click To Tweet
It might be my glasses. They are Magic Leap.
Magic Leap is here at SXSW with FLOOD magazine. It’s a concert and a technology experience. Very on brand for the festival.
They have occupied a venue in downtown Austin. It’s walkable from the Convention Center and heart of SXSW, but also within walking distance of the hubbub. On the main FLOOD stage, artist Kari Faux is rapping about how she doesn’t care “if you’re famous on the internet.” An appropriate message for this week, full of braggadocio. Meanwhile, in a side area, there is a substantial line waiting to try out Magic Leap’s Augmented Reality.
I sit down with Rebecca Barkin, Sr. Director of Magic Leap Studios, to learn more about what’s happening here. She’s been with Magic Leap for about three years and explains how much it has changed in that time. She describes “something they used to call ‘The Beast.’” It was “like this giant thing that you kind of put your head into and had to look around… You were attached by this long chord that came down your back.”
Magic Leap is here at SXSW with FLOOD magazine Click To Tweet
Since then, the device has been dramatically scaled down into a small shoulder pack attached to a headset. The website describes it as “Lightwear” a headset device paired with the “Lightpack,” shaped a bit like a small round purse and worn over the shoulder. The Lightwear contains “cameras, sensors, speakers and optical relay” that work together with the “superpowered processor” in the Lightpack.
The device lets in “natural light waves together with softly layered synthetic lightfields.” It also features spatial audio that relays distance and intensity with accuracy and control, so people can experience sound in its actual intended location.
The experience I try is called “Tónandi.” It is a collaboration between Magic Leap and Sigur Rós. Barkin explains more about how this game came to be, saying, “Rony [Abovitz, CEO of Magic Leap] handed me a list. It was like 20 artists long, and said, ‘I love these people and what they do, like make something with them.’” Barkin’s initial response was, “How do you even start to scope and prototype something like this? It was such an abstract concept.”
According to Barkin, Abovitz reached out to Sigur Rós “way before the tech was really in any condition to believe in it, to be honest.” She describes it as “a challenge” but one that could be overcome because “our engineers are brilliant.”The Lightwear contains “cameras, sensors, speakers and optical relay” that work together with the “superpowered processor” in the Lightpack Click To Tweet
The idea for this was to make tech that could actually allow for an emotional connection on the content side Click To Tweet
The idea for this was to make tech that could actually allow for an emotional connection on the content side. This had to be more than a 360 video or linear Virtual Reality music video that had been seen before, because Sigur Rós was really clear that they didn’t want to make another music video. he was really insistent they wanted people to be able to explore and discover their music in a totally different way. They were like, ‘I want you to be able to take the stems out of the music…to stretch and manipulate them.’
Barkin explains that Magic Leap was using a controller for this experience, but then they “decided to drop the controller and actually just use gestures.” This was a “turning point for the project and something that allowed us to really turn up the actual connection, but also “one of the most difficult things that we could have done.”
Using gesture requires that the technology is able to recognize the “intuitive” movements that humans might make while using devices. She describes an incident involving a jellyfish, saying, “People kept, just intuitively trying to do this,” and makes a scooping gesture with her hand. She explains that this gesture would cause the content to “sort of shake and glitch.”
This forced the team to adapt to the user. To Barkin, the absolute best test is that it “performs the way it’s supposed to within that experience. The fact that an experience has eight gestures that are working and everything’s registering is not adequate if it isn’t matching the natural movements that people make when they go to interact with the content.”
Tónandi, with Sigur Rós, was a particular challenge because they were insistent that the result not feel like a video game with traditional rewards. She explains that “when you’re developing things it doesn’t always connect…there are times when it’s really messy to share that work in progress.” Magic Leap was apprehensive when they shared their results with Jónsi of Sigur Rós and gave “his music back to him, in a new format.” However, when he put the headset on he was amazed by the results.1500 music samples were remixed to create dynamic music systems and make virtual instruments that you can interact with Click To Tweet
Tónandi, “which translates to ‘sound spirit’ in Icelandic,” is “an interactive audio-visual exploration of the sounds and spirit of Sigur Rós.” The experience is “a sensory ecosystem that grows and surrounds you.” It is filled with “Tónandi” and each one of them “is a creature with unique sound and personality, inhabiting your environment and inviting interaction.”
Mike Tucker, co-director of the project and Interactive Director at Magic Leap, gave more insights on the experience. He explains that they wanted to inspire artists to “do something that was a little bit different than what we’re currently seeing with just streaming music.” In a typical experience when you’re streaming a track “it’s just a stereo, linear, thing and it’s just exactly the same every time you go through it.”
He wanted this project to be different and include systems they built “so you’ll never hear the same thing twice.” First, Magic Leap developed a “good relationship” with Sigur Rós. Then, “ultimately we chopped up all their music…over 1500 samples.” The samples were “sort of” remixed to create “dynamic music systems” and make “virtual instruments that you can interact with.”Tónandi, with Sigur Rós, was a particular challenge because they were insistent that the result not feel like a video game with traditional rewards Click To Tweet
These samples became the “Tónandi,” the sound spirits, of the experience. Each one “represents a different musical element.” One might be vocals, another guitar, or another could be “a percussive element.” The “spatialized audio” means that the sounds are adjusted according to your position. If you move away from a Tónandi, then you also move away from the sound that Tónandi is making. This allows you to experience the feeling that you are truly walking among the sounds.
Tucker explains the challenges of spatial audio. Normally a music album is “finely tuned for different volume levels at different parts of the track.” This project needed to be “audibly pleasing no matter where you are in the space” but still make people feel like they are moving around the sounds. It was both a “dilemma” and a “tightrope.” The goal was to be “sort of aggressive or uncomfortable” but never push so far that people “have a bad experience and want to leave.”
The visuals, Barkin explains, are the result of an interesting “back and forth” with an illustrator they contracted. They ended up looking to make things that have a “shape and feel” that matches with the sounds. She gives the example of “vocal bulbs” that look like tubas and “you touch them and it’s like swallowing, except in reverse… and they give off this deep horn sound.”As I wander through the space, the luminous plants and Tónandi are generated on the surfaces surrounding me. Their individual sounds are linked to their placement Click To Tweet
The result is truly interesting. I am trying the “Lightwear” and “Lightpack.” There’s a space under the Lightwear goggles. I can still see the “normal” ground beneath me, which is oddly reassuring. The Lighwear is attached to the small Lightpack, which rests lightly on my shoulder. There are no further wires, and beyond these two objects, I am free to roam.
The sounds are alien and soft. A consistent, musical, hum. As I wander through the space, the luminous plants and Tónandi are generated on the surfaces surrounding me. Their individual sounds are linked to their placement. The accuracy is within reason and they seem to have no problem populating on the surfaces of chairs and tables in this restaurant venue. I reach for the Tónandi and they react in various ways. Some make different sounds or move while altering the sound in response to my movements.The experience is intuitive. I seem to be able to make whatever gestures are coming to mind Click To Tweet
Petting these strange, incandescent, plants is an oddly satisfying experience as they glimmer and glow in whatever direction I glance Click To Tweet
The experience is intuitive. I seem to be able to make whatever gestures are coming to mind. Petting these strange, incandescent, plants is an oddly satisfying experience as they glimmer and glow in whatever direction I glance. The return to the real world is a bit dim. Outside, I can hear the performance from The FLOOD Magazine stage. The Tónandi have disappeared but the music continues elsewhere. After all, it is South by Southwest.The return to the real world is a bit dim Click To Tweet
Overall Magic Leap has demonstrated innovative use of the Augmented Reality medium. The music and visuals meld well and it’s a unique way to experience music. This is something worth following in the future.
Tech Trends’ Virtual Reality Consultancy services offer support for companies looking to enhance brand strategy with immersive technologies such as Virtual, Augmented and Mixed Reality
Laura Kobylecky is a writer. She is particularly interested in new and emerging technology and culture. Connect with her on LinkedIn