After experiencing Mixed Reality, it’s really hard to go back.
“Welcome to Life in Holograms – this is my room,” says HoloLens Evangelist Colin Edwards as he welcomes me to my Mixed Reality initiation.With a couple of gestures and voice commands, I can make my virtual TV screen as big as I want Click To Tweet
We start off by looking at a wall, which, in the real world, contains two pictures. Through the HoloLens, however, I can now see 3 additional photographs placed between them. Colin points proudly to one of his son in his football gear before we turn to look at the opposite wall, which is adorned by two abstract holographic images downloaded from the Internet.
“Say I’m looking to buy a picture, but wanted to see how it will look on my wall before I decide, I can do that,” he explains before asking me which of the two is my least favourite. “The red one,” I decide.
Colin then instructs me to look at the red picture and say “remove”. Just like that, the offending image disappears. I then look at the remaining image, say the word “adjust,” and with a pinching motion of my thumb and index finger I drag it so that it is positioned centrally on the wall.
Next up, I sit on the (real) sofa to watch a virtual TV. With a couple of gestures I drag the corner of the screen to make it about 3 times bigger, and play a high resolution video. When I’m done watching I simply tell it to “stop.”It's a good job the HoloLens is not available for the general public to buy yet, or it would be an expensive Christmas Click To Tweet
Then the REAL fun begins. I select a tiny T-Rex hologram from a board on the corner (other options included a unicorn, a zombie, and a rather cute bulldog) Under Colin’s watchful eye, I place it on the floor a few feet away from us and utter the magic spell – sorry, voice command – which makes it go full-size. With a final air-tap flourish, the beast then comes to life and rushes towards us. It really did make me feel like a Harry Potter character to make such things happen just by looking, gesturing, and issuing imperious voice commands.
That was my third HoloLens demo, and it’s a damn good job these aren’t for sale in the shops yet, or this would be one heck of an expensive Christmas.It feels rather strange to be lusting over a Microsoft product like this. They're not supposed to be the cool ones Click To Tweet
Microsoft are looking a lot cooler than Apple right now Click To Tweet
It feels rather strange to be lusting over a Microsoft product like this. But I will actually go out on a limb and risk some serious Twitter flack here by saying that they’re looking a lot cooler than Apple right now.
While the best idea that Apple seems to come up with these days is getting rid of a headphone jack, Microsoft’s Surface book looks good enough that I decided to switch as soon as my MacBook (which has been behaving in a worryingly erratic manner of late) finally gives up the ghost. I’m rather looking forward to it.In less than two years since announcing it, Microsoft has built a seriously impressive piece of kit Click To Tweet
But that’s nothing compared to the innovation the HoloLens brings. In less than two years since announcing it, Microsoft has built a seriously impressive piece of kit that manages to be cool even though it’s still primarily a corporate tool. Unlike with most VR devices, Gaming and Entertainment only form a very small proportion of its content, and Microsoft’s Director of Product Marketing Leila Martine stressed several times that this is not meant to be a consumer device yet, and that this is only the start of a multi-year journey.This is only the start of the journey for the HoloLens Click To Tweet
Mixed Reality really kicks VR's butt in a lot of ways Click To Tweet
Which is a shame, because as much as I’m a fan of VR, Mixed Reality really kicks its virtual butt. Unity Labs Lead Designer Timoni West explained one of the reasons for this when we were discussing the reason why most VR experiences don’t last more than a few minutes.
“There are very few experiences in this world that are worth 100% of your time and attention, and I think that’s one of the reasons why VR experiences do tend to be shorter right now. There’s no way to check your phone in VR, it’s very hard to find your drink if you’ve misplaced it, It demands most of your attention, and there is nothing else out there that does that except maybe a new-born child.”
HoloLens is an entirely different proposition: a fully untethered holographic device. This means that everything it needs to operate (basically a fully functional Windows 10 computer) is contained in the headset itself. I cannot stress how impressive that is. No wires, no huge external processing units, just the glasses. And call me a dork if you will, but I actually think they look good.
At the start of my session Colin measured my IPD (Individual Pupillary Distance) which meant there was very little adjustment required before I could start exploring those Mixed Reality experiences. Everything was in focus, my gaze was automatically and accurately tracked, and at no point did I feel even a hint of any discomfort or eyestrain. The same cannot be said for the times I used VR on various devices.
“You tend to forget that over the last 70 thousand years we’ve only had one reality which has been physical, based on atoms, things you can touch,” Says Martine. “It’s only in the last couple of generations really that you started having this digital world of bits and bytes and logic, but those two realities have still been very separate. This is now a really significant moment across the entire spectrum of what we call Mixed Reality.”
It’s interesting that they now seem fully committed to the MR label rather than AR. Martine explains that what they’re doing is not merely augmenting reality – creating something overlays onto a real-world background – nor is it a separate immersive experience like VR. Mixed Reality represents an entire new spectrum of possibilities.It’s interesting that Microsoft now seem fully committed to the MR label rather than AR Click To Tweet
As cool as it is though, this is about much more than a single device. The hardware is an enabler for Microsoft’s bigger play to become the dominant Mixed Reality platform. At Computex this summer they announced Windows as a holographic platform that would allow anyone who wants to build on this continuum to use it and therefore have the content accessible across any device. Offering a hardware option to go with this is part of that broader strategy, explained Martine:
“The reason why we chose to get involved building first party devices like these is that when you get involved in a category that is emerging you often need to build out as this is something that hasn’t been done before.”
Although they’re focusing on creating new 3D experiences and applications, 2D apps built for regular computers will also run on the HoloLens, as it’s essentially a Windows computer, so that anything that works on a PC – such as Skype, for example, will work with it.The hardware is an enabler for Microsoft’s bigger play to become the dominant Mixed Reality platform Click To Tweet
Dong Yoon Park from BRDY Studios is one such developer. He adapted his existing iOS app for the HoloLens, turning what was originally a learning and teaching toolkit for typography into a visualisation platform that graphic designers such as himself can use to see how different lettering and fonts would look in 3D environments. This is particularly useful when designing signage, for instance.This is not just about getting into the hands of tech enthusiasts Click To Tweet
This technology is bringing deep levels of transformation and we’re seeing that in a number of areas Click To Tweet
“This is not just about getting into the hands of tech enthusiasts – that’s cool and we want to do it – but this is actually solving hard problems, doing really transformational things that you simply could not have done before. This technology is bringing deep levels of transformation and we’re seeing that in a number of areas.”
Take building and architecture as an example. An Architect’s job involves constantly translating 3D concepts into 2D blueprints, then into 3D scale models to help clients visualise and sign off on the plans, to eventually become physical buildings. Yet the early partnership between HoloLens and Trimble shows how those concepts can be rendered into full-scale 3D holograms in real time, allowing professionals from all over the world to visualise and share those plans, just as they would look in the real world, making accurate adjustments before they ever break ground and saving enormous amounts of time, energy and money.When you see these things and you bring them to life, you can learn in an entirely new visual way Click To Tweet
Microsoft are also working with the NASA Kennedy Space Center Click To Tweet
“I always think I could have gone to medical school had it not been for the having to learn through textbooks. When you see these things and you bring them to life, you can learn in an entirely new visual way,” says Leila Martine. After experiencing the demo of the HoloLens anatomy application (a pared-down version of what students at Case Western Reserve University use to do just that, I can certainly see her point.
Microsoft are also working with the NASA Kennedy Space Center on an experience that will be open to the public, as well as applications that will allow for greater collaboration between scientists. Meanwhile, some companies are also starting to roll out consumer trials. DIY retailer Lowe’s – the second largest hardware chain in the US – already offers a service in some of its stores which generates suggestions based on customers’ Pinterest feed. Those customers are then able to select holographic items from Lowe’s catalogue, see how they look in their own home, and interact in real-time with a sales rep who advises them while looking at the same things.
There are two main things which make these experiences work so seamlessly: The first is spatial mapping – the constant scanning of the environment that relays the positioning of objects in the surrounding environment, so the device can accurately and realistically place holograms around them.
The second is persistence, or world-locking. As I place my holograms around a physical environment, they don’t disappear once I remove my headset. If I bring a friend to that room 3 weeks later and she’s wearing a different HoloLens device, she’ll still be able to see the holograms in the exact places I left them.
Another important feature of the HoloLens is spatial sound Click To Tweet
What enables this world locking is the HoloLens’ HPU – Holographic Processing Unit. This, according to Martine, is a significant leap in technology, as current CPUs (Central Processing Units) and GPUs (Graphics Processing Units) were simply not up to the extreme complexity of processing real-time data from a multitude of different camera and infrared depth sensors.
Another important feature of the HoloLens is spatial sound, which emanates from two small red units built in on either side of the headset and is very effective at adding directional audio cues and building a soundscape that makes the holograms come to life. I rather liked the fact that this wasn’t done through headphones, as it ties into the ethos that these experiences are not about shutting out the world around you, only enhancing it.
The HoloLens demos all felt like real products addressing real needs Click To Tweet
Indeed, whereas most VR experiences out there have been pretty fun and/or showed great potential in all sorts of areas, the HoloLens demos all felt like real products addressing real needs, even the one which showcased a luxury watch from an entirely fictitious brand. The objective here was to demonstrate the potential of the technology for sales and marketing, specially in the luxury goods space.The marketing applications for Mixed Reality are tremendous Click To Tweet
At one point in this experience I was shown a deconstructed view of the internal workings of the watch; I could walk around and look at each part closely as it floated in the space in front of me. Later, when I switched to Presenter Mode on the device, I had access to a heat map tool that showed me exactly which parts of that hologram attracted the most attention.Where it comes to Mixed Reality, Microsoft shows that it really is Hip to Be Square. Click To Tweet
The marketing implications here are tremendous, as it’s the kind of thing that would eliminate the guesswork in terms of measuring audience engagement. If you can see where your potential customer’s gaze is drawn to and where it lingers the longest – something the HoloLens does automatically as it relies on eye tracking to work – then you can easily fine-tune that content for maximum impact.
As great as the practical stuff was, however, I still spent most of my fake-watch demo time happily gazing at a group of tiny holographic seagulls, flying over a fleet of tiny holographic sailing boats in a holographic ocean contained within that holographic watch. All of which were sitting on top of a perfectly ordinary table – which (weirdly) happened to be made out of real wood.
Considering that this is such an early-stage product, it’s very difficult not to be impressed, and there were only a couple of small niggles I could pick on: While gaze and voice worked seamlessly, the gesture commands took slightly more getting used to. Still, I mostly got it right on the first try, and under the helpful guidance of people like Colin I was able to make the necessary adjustments– such as remembering not to over-extend my arm or keep all but two fingers folded down as I did the quick air-tap motion that works as the equivalent of a mouse click. I can see how, as motion tracking technology improves, you will be able to use increasingly natural gestures to shape the world around you, and I’m really looking forward to doing some more flamboyant holographic spell-casting.The HoloLens manages to be at once utterly practical, incredibly fun and more than a little bit sexy Click To Tweet
The other issue is that your holographic Field of View (FOV) is not wrap-around, but restricted to a rectangular area in front of your eye, which means that for larger holograms you have to move your head up and down to get the full picture, which breaks immersion slightly. That, however, feels very much like a bridging compromise in favour of texture and quality. I have a feeling that an expanded FOV will be on the cards for later iterations
The HoloLens manages to be at once utterly practical, incredibly fun and more than a little bit sexy, so my My biggest niggle with it is simply that I can’t have one just yet.
And since I started this post with a 60’s song reference (those who failed to pick up on that should seriously brush up on Janis Joplin classics ) so I’ll finish with an 80s one: Where it comes to Mixed Reality, Microsoft shows that it really is Hip to Be Square.
Microsoft started shipping the HoloLens developer suite and commercial edition to the US and Canada back in March, and last week they opened pre-orders in the UK, France Germany, Australia, New Zealand and Ireland, which are expected to start getting their kits by the end of November.
For companies looking to get into VR/AR/MR our Virtual Reality Consultancy services offer guidance on how these technologies can enhance and support your brand strategy.
Alice Bonasio is a VR Consultant and Tech Trends’ Editor in Chief. She also regularly writes for Fast Company, Ars Technica, Quartz, Wired and others. Connect with her on LinkedIn and follow @alicebonasio and @techtrends_tech on Twitter.