How holograms can help NASA explore Mars

Windows Holographic and HoloLens made up one of the most exciting announcements at Microsoft’s big event this week. During and after the event, the demonstrations ranged from using the augmented reality goggles to play a living room-sized version Minecraft to working with a CAD-like software. But the most audacious demo was of OnSight, a program developed for HoloLens by NASA’s Ops Lab. It lets scientists explore a virtual Mars using data collected by the Curiosity rover. It's collaborative, too: scientists in different locations can join the session remotely, appearing in each other's HoloLens as humanoid avatars. Our own Tom Warren tried it out in Redmond and said it was jaw-dropping.

Ops Lab is in charge of building systems for controlling robots and spacecraft at NASA’s Jet Propulsion. Some of their projects include controlling articulating arms, building immersive environments, and controlling robots like Robonaut 2 and Athlete using technology like the Oculus Rift and a Kinect sensor. The Ops Lab team is already integrating OnSight into the Curiosity team’s software — so how did they get there?

The partnership began over five years ago. During the Project Natal days of Microsoft’s Kinect, Ops Lab’s project manager Jeff Norris was introduced to Kinect (and HoloLens) creator Alex Kipman. The two hit it off. "Our teams started talking about how we could use technologies that were being developed in his incubation group to better control robots and spacecraft," Norris tells me.

A partnership five years in the making

Not long after that, Kipman showed Norris an early version of what would eventually become Windows Holographic and HoloLens. At that moment Norris says he knew there was potential, and they started figuring out how they could use it in space exploration. That collaboration resulted in a program that will expand the capabilities of the Curiosity team scientists.

OnSight takes the data and images from the Curiosity rover and uses HoloLens to make a room appear to be the surface of Mars. Mission scientists will virtually step onto the surface and move around with an extra sense of perspective and presence not afforded by two-dimensional images, and that will improve the quality of conclusions drawn from things like shape and layout of geological features. HoloLens will also be aware of where a scientist's computer is, cutting it out of the virtual scene and letting a user control a mouse seamlessly between the desktop and the surface.

Curiosity scientists will mostly interact with OnSight by using the global interactions built into Holographic. There are gesture and voice controls, and each scientist’s avatar has a "gaze ray" that draws a line to what they’re looking at, making it easier for other participants to follow along. Being able to use a computer while wearing HoloLens — another part that impressed us in Redmond — will let scientists take advantage of the raw data available in MSLICE, a program that provides the raw data from Curiosity that has been fully integrated with OnSight.

The Ops Lab team views presence — whether real or virtual — as a critical tool for the explorer. "What a geologist is doing when they’re looking at a scene is they’re trying to understand a story that this environment has to tell them," Norris says. "One of the chapters of that story is the shape of the environment — the way that the rocks are worn, the way that they form lines and curves. That information is one of the ways that they divine what’s happening."

He cites a study the Ops Lab team performed over a year ago: 17 rover scientists were given standard images of a Martian scene provided by MSLICE. They then viewed the same scene through a head-mounted display. In both cases, the participants were asked to draw a map of the shape of the environment and flag the location of certain points of interest. The scientists’ estimation of distances was more than twice as accurate while wearing the head-mounted displays, and their estimation of the angles of objects was more than three times as accurate.

More surprisingly, Norris says the scientists were accomplishing this increased accuracy without any basic VR or AR training — only one of the people in the admittedly few person study had ever even worn a head-mounted display before. "That’s what really put the wind in our sails to go further with this," Norris says.

While the Curiosity team has already been able to do great work with two- and three-dimensional images of the surface, OnSight is the next evolved step in how they analyze the data. It’s a better option than viewing a 3D-model (or stereoscopic images) because of the power of proprioception, or the body’s sense of self. "Your body, as you’re walking around in a place on Earth, knows where it is. And your eyes are presenting to your brain images of what you see at that position," says Norris. The integration of those things is essential to building a mental model of your environment, and it’s why the Ops Lab team is so excited about the work they’ve done with OnSight.

Ops Lab was already working on immersive interaction, using an Oculus Rift. But the Rift presented its limitations for NASA, like its need for wires and how it closes users off from their surroundings. HoloLens relieves both of those problems, so the Ops Lab team started an intense development period about a year ago to create OnSight — so intense that a portion of Norris’s team actually moved to Redmond to live and work shoulder to shoulder with the HoloLens team.

Ultimately, the goal is to test OnSight in actual operations later this year, though Norris couldn’t say when. But Curiosity scientists already professed excitement in a video released this week by JPL. Geologist Fred Calef says not only is OnSight "like teleportation," but it will also save time. Katie Stack Morgan says the enhanced visualization will help the team make better decisions about operating the rover in general — an important issue considering the problems the team has had with things like wheel degradation.

That doesn’t mean OnSight is close to being a finished product. "There’s a lot of work to do," Norris says. "It’s no trivial undertaking to bring a new capability into mission operations." And beyond Curiosity, nothing is set in stone — but Norris says there is vision to take this past Mars to other planets in the solar system. "This is a new tool in the toolbox of explorers."