New HoloLens 2 gives Microsoft the edge in the next generation of computing

From flat screen to hologram

Until now, most people have experienced computing through a flat piece of glass: laptop, computer monitor, phone, tablet, video games on a TV. Microsoft’s mixed reality offerings draw digital information out of rectangular screens and allow people to interact with holograms in physical space. These can exist independently, like a three-dimensional rendering of a human heart that medical students can grab, resize and rotate to see all the structures clearly. Or they can relate to physical objects in the real world, like instructions superimposed on a furnace that show you how to change a filter.

Microsoft says the new HoloLens 2 provides a far more immersive, instinctual and comfortable experience for first-line workers whose hands are occupied by physical tasks. It can help them diagnose a problem with a jet engine or access step-by-step holographic instructions to assemble an electric bus battery. The person using it can go backwards to double-check a step with a nod of her head or a voice command. The person is able to see in three dimensions — on the physical equipment she is working on — precisely where each screw needs to go, or what direction to turn a ratchet.

Compared to the first generation of HoloLens, HoloLens 2 also offers new features like the ability to grab and rotate holograms as you would a real object rather than having to learn specific gestures. Eye tracking can sense when someone’s eyes land on a particular part of a machine and call up useful digital information about it. Words automatically scroll as you read. The end result is like going from watching a cartoon flip book to the truly immersive experience of actual cinema.

The business opportunities HoloLens 2 presents are powered by expanding capabilities at the intelligent edge. That’s comprised of an exploding array of devices — ranging from baby monitors and refrigerator sensors to something as sophisticated as HoloLens — that can process information quickly and locally.
These increasingly perceptive devices can tell a self-driving car when to turn, a piece of factory equipment when to shut off or store workers to clean up a puddle where customers could slip in instances where split-second decision making is necessary, and even in locations where connectivity is limited.
But as edge devices become more intelligent, they also need to become more secure. Apps that previously needed to run in the cloud and on local servers now need to work across a rapidly expanding collection of devices, with more developed each day for specific industrial processes. Azure now delivers secure computing power to and from the edge with solutions like Azure Sphere and Azure IoT, and it allows developers to design products that run seamlessly across diverse computing environments.

Technology for first-line workers

Information workers — people who do much of their work at a computer to produce words, design things, respond to emails, manage people or make business decisions — have benefited from an explosion of technologies allowing them to learn nearly anything with the touch of a mouse and communicate instantaneously. That’s driven unprecedented productivity gains for individuals and businesses.
But most people don’t work that way. People who use their hands to assemble, care for, repair, troubleshoot or interact with things have too often been an afterthought in this technological revolution. That’s left an enormous opportunity to give first-line workers the technology they need to make similar productivity gains, said Lorraine Bardeen, Microsoft general manager for Dynamics 365 Mixed Reality at Work.
“The first-line workforce in so many companies is vital not only to day-to-day operations, but also in the way they craft their products. And they’re often the majority of employees,” Bardeen said. “And yet they’ve experienced very little of the empowerment that technology has brought to people who work in offices or more traditionally compute-friendly environments.”
When Microsoft started asking companies about how mixed reality could benefit them, the same needs bubbled up across industries. They wanted help connecting workers in remote locations or disparate workplaces with experts to troubleshoot problems, in envisioning how equipment or furniture or physical objects will actually fit in three-dimensional spaces and in training new employees who need their hands free to perform work.
To deliver more out-of-the-box value — in the way that programs like Microsoft Word and Excel helped people find value in a new operating system called Windows — Microsoft has created mixed reality applications for Microsoft Dynamics 365. These allow companies to almost immediately use HoloLens 2 to meet workplace needs without needing to hire a small army of developers.
Dynamics 365 Guides now joins Dynamics 365 Remote Assist and Dynamics 365 Layout as Microsoft’s pioneering mixed reality applications for business. Guides allows companies to move training materials from flat paper and screens into an immersive three-dimensional experience.
“Everyone here wants to be successful, and the ability to get new employees in and make them productive quickly is invaluable,” said Rob Branson, senior director of global technologies and operations for PACCAR ITD, which has worked with an early version of Dynamics 365 Guides.
“If you think about the way adults learn, it’s very visual. And the ability to see step-by-step instructions overlaid on an actual physical object will really accelerate how an employee learns a new task,” he said.
HoloLens 2 was designed with these workers in mind — it’s lighter and far more comfortable than the previous generation of HoloLens, with a more balanced center of gravity, so a person can wear it all day. A new flip-up visor lets workers switch easily between physical and holographic worlds.

Intelligent conversations between people and things

Mixed reality is even more powerful when it takes advantage of the intelligent edge and intelligent cloud’s different capabilities.

In some scenarios, you want to quickly process information on the intelligent edge without sending that data to the cloud, such as in cameras that can alert you to imminent safety risks or algorithms that control braking systems. On a factory floor, not all the data from each sensor on each piece of equipment is relevant at any given time. So running less-complicated AI services on the edge can help filter out irrelevant information or perform tasks that don’t require the power of the cloud.

If you need a hologram to help potential customers envision how a new car or potential remodel will look with different options, a single HoloLens 2 using on-board capabilities in a showroom or living room will offer plenty of computing power and resolution.

But connecting that device to the new Azure Remote Rendering mixed reality cloud service can quickly produce intricate, three-dimensional digital models that begin to rival the sculpted clay or detailed architectural models that a company might spend days or months building today. That simply wouldn’t be possible without the graphics processing power of the cloud.
“Suddenly mixed reality goes from something that’s a novel way to augment what you’re already doing to being able to replace an entire business process — for example, using full digital construction in a way that just couldn’t happen before,’” said White.
When overhauling an urban train station or building a new soccer stadium with lots of moving parts and heavy equipment, looking down to access information on a phone or tablet can be dangerous, said Noah Eckhouse, Bentley senior vice president for project delivery. HoloLens headsets allow workers to access digital information while remaining aware of their physical surroundings.
Through HoloLens, the company’s SYNCHRO software allows workers to zoom in on a particular location on the construction site and access important digital information, like safety guidelines or installation instructions, for that particular job or area. Managers can see in three dimensions what the project is expected to look like two days or three weeks from now — based on constantly changing realities and projections — and anticipate any scheduling conflicts.
“A construction site is like a giant ballet — it’s a very highly choreographed operation with movements of materials and people that all have to exist within a certain space,” he said. “And the plan changes the first day you’re on the job.”

Bringing powerful perception tools to the edge

Two defining achievements in computer vision and AI contribute to HoloLens 2’s immersive experience. The ability to interpret physical spaces with semantic understanding allows the device to differentiate between walls and windows or a couch and coffee table. Natural hand-tracking now allows people to grasp, rotate and expand the holograms more instinctively, rather than having to learn gestures that mimic mouse movements.

Those advances are enabled by the fourth generation of Kinect, combined with AI tools that operate on the edge. That depth- and motion-sensing technology was originally developed nearly a decade ago to create a gesture-recognition accessory for Xbox. But the ability to sense depth accurately and pinpoint how human bodies are moving in space turned out to have far broader applications than gaming.