Mark Pickavance examines what we know about Microsoft HoloLens and questions where it fits in among the many competing devices
The words ‘Revolutionary’ and ‘Microsoft’ haven’t made it into the same sentence for some time, but the arrival last year of something quite unexpected from the Redmond-based company did find many combining them.
After the retreat that Kinect ultimately became, few were expecting Microsoft to launch something quite so radical again anytime soon. Yet the appearance of HoloLens provided an interesting counterpoint to what Google and others are doing in the virtual reality marketplace.
So what is this new device, and should we get excited about it?
Project Baraboo
According to various sources, the concept of the HoloLens was first pitched at exactly the same meeting when the Kinect was presented. In fact, they’re a fork of a single original idea, about making computers and games consoles spatially aware.
Three years later, in 2010, Microsoft started working in earnest on what became ‘Project Baraboo’, a system of holographic lenses designed to deliver an augmented reality OS to the wearer through a headset. Interaction with the computing power in the headset would be through gesture, eye tracking and voice. Though, Microsoft hasn’t ruled out the use of wireless keyboards, mice or other controllers.
From the outset the intention was to make it ultimately self-contained and, unlike the Kinect, independent of an external computing platform.
It took five years for Microsoft to have technology it felt comfortable enough to reveal publicly, which it did at the Windows 10 announcement on 21st January this year.
The versions of the HoloLens headset shown to the press after that announcement weren’t the finished item and required lots of extra equipment connected to work. Subsequently at the Microsoft Build conference in May, a much more polished version of the hardware was shown that didn’t require the users to mount batteries and other paraphernalia to their bodies.
While Microsoft was seemingly happy to demonstrate its equipment to a select few on both occasions, it seemed less keen to have the experience filmed or provide any great details about the hardware and capabilities of this device. That is probably because the hardware development team is at a critical point where they’re trying to finalise the technology, so not everything is set in stone.
However, based on the experiences of those lucky enough to try it, here is what we do know about Microsoft’s virtual reality experience.
HoloLens Revealed
At the heart of the headset is a Windows 10 PC, one that’s small enough to fit inside and be powered by batteries. While neither the CPU nor GPU have been discussed, given what calculations need to be performed for the system to work they must be reasonably powerful.
To avoid exhausting the batteries very quickly (although we still have no idea of operating life), Microsoft has designed some specialist support chips, including a holographic processing unit (HPU). This is tuned to handle the maths involved in converting the piles of data coming from the many sensors on board into the visuals that the wearer experiences. These sensors include an accelerometer, gyroscope, magnetometer, depth camera, photographic camera and a microphone array. In many respects, this is a headmounted Kinect, of sorts.
3D stereoscopic images created by the system are projected onto the inside of the lens, and these are positioned correctly by means of a calibration process. Each user must go through calibration to compute the exact distance between the pupils of each eye, so the stereo visuals are aligned with real-world objects. By using the sensor data, objects can also be made to interact with the real environment, and the user can direct with them using their hands, eye or voice commands.
The number of calculations needed to make this work believably is disturbingly high. HoloLens chief inventor Alex Kipman is quoted as saying that the HPU processes “terabytes of information” from the HoloLens’s sensors in real time to achieve the effect.
That this is possible at all is impressive, but as HoloLens is a wireless device that recharges via USB, the achievement seems almost magical.
In many respects, this is very like some other VR technology we’ve been seeing in recent years, including Oculus Rift. But where HoloLens diverges is that instead of supplanting reality entirely, it chooses instead to overlay things on the real world.
While this solution doesn’t entirely meet the definition of ‘virtual reality’, what it does is avoid some of the very obvious pitfalls that those solutions have encountered, like spatially induced headaches and nausea.
When the HoloLens gets much closer to actually being available, hopefully we’ll get more details about the computing power, sensors, battery life and cost.
At this time, we have no hard information on any of those things or even when the headset will be able to be ordered or bought. What we do know is that Nasa has had it for a while, and it seems very impressed with what can be achieved using it.
Walking On Mars
One of the first users of HoloLens was Nasa’s Jet Propulsion Laboratory in Pasadena, California, which developed a software tool called ‘OnSight’, for working with its Mars Rovers. By using the headsets in conjunction with image data they’ve previously sourced from the rovers, mission specialists can walk around the surrounding terrain and decide where to move the vehicle next and by what route.
What’s fascinating about their solution is that they realised early on that if the Mars graphics overlaid reality, then they’d need to create markers for where other team members are located in the work area or risk bumping into them.
Jeff Norris, JPL’s OnSight project manager, explained why the HoloLens was critical to the solution: “Previously, our Mars explorers have been stuck on one side of a computer screen. This tool gives them the ability to explore the rover’s surroundings much as an Earth geologist would do field work here on our planet.”
While all the work with HoloLens up to this point has been research, it’s expected that later this year OnSight will be used in anger on forthcoming Curiosity missions. And future applications may include Mars 2020 rover mission operations and in support of Nasa’s journey to Mars. If successful, OnSight might be seen as a planning and prototyping tool that allows a mission to be conducted on a much more interactive basis than Nasa has previously been allowed to work.
But HoloLens developers aren’t all involved in hard core scientific research. Some are just interested in the interactive play potential that augmented reality offers.
Holobuilder And HoloStudio
Microsoft’s recent purchase of Minecraft creator Mojang is seen in an entirely different light when viewed in the context of what HoloLens can do.
Microsoft already has a prototype clone of Minecraft running on the HoloLens platform, called HoloBuilder. Using this application it’s possible to create Minecraft structures in your home that either conform to the space and surfaces already there or that totally subvert them. Worlds can be viewed through virtual windows and holes, presenting scenery that extends above and below your room, as well as beyond solid walls.
While Microsoft has yet officially confirm that Minecraft is coming to HoloLens, given how well the demo was received by those who tested it, this might well be one of the critical default software components that it will use to generate sales.
A less compelling, but equally imaginative demo tool is HoloStudio. This software is focused on providing a means to sculpt 3D objects inside the environment offered by HoloLens. The images and video of HoloStudio reveal something that’s reminiscent of the old PC classic TrueSpace, where you can dynamically scale and distort primitive shapes to make more complex compound objects.
That you have the freedom to move your viewpoint easily rather than needing multiple fixed viewpoints could make for very rapid prototyping, even if professional modellers might want finer control than this method currently enjoys.
As an interesting aside to HoloStudio, Microsoft has created an interface within the tool that allows the finished models to be outputted on a 3D printer, allowing the designer to progress from virtual 3D to generating real objects seamlessly.
Nobody working a professional CAD station to create film effects models or structural engineering is likely to drop what they’re doing and switch to HoloStudio soon, but as a proof of concept, it seems an interesting development that could evolve into something more practical.
Along with these tools, Microsoft has also shown a wider selection of conceptual work, including some educational apps for teaching biology, architectural engineering, a virtual movie studio and even a holographic version of Skype.
Clearly, HoloLens isn’t being tailored specifically for serious or entertainment use but for as wide an application as possible. It will be the software that ultimately sells this product and the work of developers that will keep Microsoft’s customers happy in the future.
Competing Products
A bit like 3D movies, VR has gone through a number of cycles, where everyone is told it’s the ‘next big thing’ only for it to fall flat on its virtual face.
Infamously, Nintendo launched the Virtual Boy on an unsuspecting public in 1995, only to discontinue it the following year, after a generally poor reception and disappointing sales. Reviewers complained the headset made them feel decidedly unwell, to the point of vomiting.
The ideas of both full VR, as in the Virtual Boy, and augmented reality solutions have gained traction, both in popular culture and with military hardware designers, but the re-emergence of this technology into the realms of retail product has been much slower. However, in the past few years, two major branded products have been propelling forward the idea that we all might join the few who enjoy VR on a regular basis: Google Glass and Oculus Rift.
The Google Glass project has never been a commercial product, yet Google sold a fair number of ‘Glass Explorer’ headsets to those interested in either experiencing or developing for this unfinished product.
In terms of the VT experience, Glass was designed to provide a hands free means to interact with Google services, while providing some minor degree of augmented reality. When it was first released, many thought that Google had stolen a march on its competitors, like Apple, and soon we’d all be wearing these curious glasses that you can talk to and that provide useful information in the corner of your eye.
Unfortunately, life is rarely that simple. Almost at launch, some people expressed privacy concerns about how Glass could be used to record others. Cinema chains also entirely misunderstood their purpose and assumed that those wearing them were trying to pirate movies.
Social issues, safety concerns and the quite limited functionality all pushed Google away from a commercial launch and in the direction of a rethink.
At this time, you can no longer buy the Explorer edition, and Google doesn’t have a timescale as to when a new Glass solution will be beta tested or commercially available.
If Google Glass is at one end of the spectrum, then the other is Oculus Rift, developed by Oculus VR. This is a full VR headset that requires a connected PC to perform the 3D calculations necessary to drive the visuals. Originally funded by a Kickstarter campaign, the project has been driven by fans of FPS games, who wanted a much more immersive experience.
The headset provides 90 degrees horizontal and 110 degrees vertical stereoscopic 3D perspective, and in the games that have been modified to use it so far you can actually look down and see your hands holding a weapon.
After several prototypes and numerous demonstrations, a consumer version with spatial audio, higher resolutions displays, enhanced head tracking and even a wireless mode is promised in Q1 2016. Pre-ordering will begin later this year, when we also get to find out how much Oculus Rift will cost.
Being a full VR solution, Oculus Rift’s biggest challenge has been to reduce the nausea that the immersive images can generate. The reason they do this in many people is that the head positioning sensors drift out of sync with the real positions, making the images appear to lag the motion sensors in the user’s own head. This is the same problem as those who experience seasickness get, where their internal sensors tell them one thing, but their eyes say something subtly different. Much work at Oculus VR has gone into making the tracking lag so low that people won’t be violently sick after extended use.
The second developer kit model called DK2 was sent to over 100,000 users, so Oculus VR has a substantial amount of feedback about what works well and those areas that need addressing.
The questions that haven’t yet been answered about Oculus Rift are how much it’s going to cost and whether it will arrive before or after HoloLens.
Microsoft would undoubtedly like it to arrive after, but that isn’t its only concern, because the reaction to the Build 2015 demos has been mixed, at best.
The HoloLens Cracked
Over the past few years, I’ve given various TV makers a kicking in these very pages for their portrayal of 3D TV. Where it really annoys me is that they show 3D imagery breaking the frame of the panel, in a way that it’s impossible for the images to do.
It isn’t a mistake on their part, as they know full well that 3D doesn’t work like that. They’re intentionally misleading those who have never experienced it, and amazingly in this instance, those that regulate advertising appeared to notionally allow it.
The excuse (and I’ve heard it first hand) is that it’s difficult to use a 2D image to convey 3D without doing these things. How else can they get around this limitation? It’s a tough one, but showing it doing something it patently can’t do seems the worst possible solution to the problem.
But sadly, in the Build presentations, Microsoft actually went further than those indiscretions, by presenting a version of the HoloLens experience that wasn’t the one users of the prototype equipment got.
I wasn’t at BUILD, but based on what those journalists who did attend said, the presentation was at best “misleading” and at worst, as one attendee put it, “a lie”.
The point of contention was the field of view that those using the device would experience and how it differed significantly from the version shown in January and used by Nasa.
In the earlier version, the field of view was sufficiently wide that wearers described the experience as “immersive”, to the extent that in the part of the demo where they used the OnSight software, they actually felt like they were walking around on Mars.
Unfortunately, the Build users got an entirely different experience, as the field of view had been drastically pared down, to the point where everything seemed viewed through a virtual letterbox.
This was so different that a number of journalists who had attended the original demo thought their HoloLens equipment was initially broken, until Microsoft staff convinced them that it was working as intended. Based on the descriptions provided by a number of attendees, the field of view is only about 40 degrees horizontally and even less vertically.
The typical field of view of an average human adult is about 180 degrees horizontally and 135 degrees vertically. Even if you discount those parts of our vision that are considered peripheral or even fully stereoscopic (just 114 degrees), the area covered by HoloLens isn’t comparable. No explanation of the discrepancy between this and the earlier prototype was provided by Microsoft, and none has been forthcoming since.
What irked many is that in the stage presentation, Microsoft said that it had mounted a mounted a HoloLens on a camera so those attending might understand what the wearer was experiencing. However, that viewpoint had no small viewport, so it wasn’t at all representative of the HoloLens experience.
One attendee, Paul Thurott (thurott.com), commented that using the Build version of HoloLens he could only see objects directly in front of him. And in the stage demonstration, when the presenter asks for a video to follow him, he’d be totally unaware if it was following, until he turned back to look at it.
As you might expect, lots of journalists weren’t happy with the impressions that the stage presentation gave and how they didn’t gel with those experiences they had with the real equipment, leaving something of a cloud over HoloLens.
As we still don’t have a launch timetable, these things might be addressed before HoloLens is commercially available, but given Microsoft’s lack of transparency on the subject, they might not.
Final Thoughts
I so want to be entirely wrong about HoloLens, because conceptually I can see the augmented reality path as potentially a much more fruitful direction than the pure VR one.
It negates the motion sickness issues and allows you to interact with the real environment around you, in addition to the virtual one.
I can imagine a working environment where I don’t generally use monitors, where I can spread my web pages and documents around me and interact with them in a virtual space. It’s the future depicted by Minority Report, but actually better in some respects and tons more fun.
But undeniably, there was an issue with the original prototype platform either in battery life or cost, which caused Microsoft to step back from its original concept in their its demonstration versions.
It’s also very disconcerting that the camera hooked up to the technology at the Build presentation isn’t remotely delivering the graphics that testers experienced hours later. There’s a horrible mismatch between the marketing dream and the reality, or virtual reality in this case, which Microsoft appears in no great hurry to address. And based on technology companies in general, and Microsoft in particular, if they don’t want to talk about something, it’s for commercial reasons.
If the finished HoloLens that Microsoft actually launches is like the one it showed at Build, a lot fewer people will be interested in having one, I suspect.
The problem is that nerds these days are more social creatures, so given the information void that’s been created, there are plenty of people on the internet ready to fill that space and trash HoloLens before a finished product is ready for release. That’s a little depressing, but the way that expectations have been dampened, it was inevitable given the relative drought of factual information about this project.
Until the finished product arrives with a horribly small FOV, costing a fortune and with software that’s very limited, then we probably shouldn’t judge it. Mind you, if it does arrive and Microsoft has somehow managed to snatch defeat from the jaws of victory once more, then it will deservedly get both barrels. If that’s not the case, we can just put this down to a corporation with hundreds of thousands of employees but not a single one who can communicate coherently.
Managing expectations is always part of any technology product’s launch, and at this point Microsoft seems content for many to believe this won’t be the product that makes the Microsoft brand exciting again. However, it could all be misdirection.
In that case, if HoloLens does what the original demo version did and isn’t stupidly expensive (like the Surface Pro 3), then I’ll be at the head of the queue to buy one, almost regardless of what it costs.
Based on past experience, though, I’m confident that my cash is likely to remain unsquandered in the coming year – or at least until HoloLens 2.0.