Owlchemy Lab Report: Mobile Spectator - AR Spectator Camera Experiment
As the saying goes— science never sleeps! Today, we’re excited to share an R&D project fresh from the Labs.
After months of mixing realities, rendering ridiculous interactions, and concocting crafty code, we think we’ve created a truly novel way to bridge the gap between the physical world and what’s inside the headset— something we call Mobile Spectator!
Here’s a brief glimpse of it in action:
Mobile Spectator Camera— The latest R&D experiment from Owlchemy Labs!
Mobile Spectator is a fun way to share a VR experience with players outside of the HMD, utilizing a tool everyone already has in their pockets— a smart phone!
We are always looking for new and unique ways to explain the magic of VR at a glance. Our Mixed Reality R&D experiments were a first step, but we recently set out to see if we could create a different way to view VR— one without the prohibitive space and hardware requirements of mixed reality.
How It Works
The Mobile Spectator mobile app uses ARCore to track the phone’s position and planes in the physical space. To calibrate the phone’s position in VR, simply point the phone’s camera at the headset, approximately a foot away, and tap “Sync”. This process synchronizes both the VR position and AR position of the mobile device in the same play space.
To allow for this unique third-person view, we tried a different approach from a traditional multiplayer networking solution. The phone and PC communicate directly via WiFi. Once the phone is calibrated, its AR-calculated position is sent to the PC, placing an additional camera in the VR scene to represent the Mobile Spectator. The PC renders a frame from this camera, encodes the frame, sends it back to the phone, decodes it there, and… viola! You have a live video feed on both the phone and PC.
There are two approaches we identified when tackling this experiment: using the phone to render frames natively; and using the PC to render frames— the latter being the method we decided to implement in this experiment. This method requires almost no networking code, does not require the application to run natively on the phone, and allows for high quality video capture. Of course, the tradeoff here is that rendering on the host VR machine is resource intensive, has bandwidth limitations, and might not be feasible on all hardware.
The Final Result
Unlike mixed reality, Mobile Spectator requires an in-game avatar to be visually legible and interesting. Luckily, we already had an amazing avatar system implemented in Vacation Simulator, so our final result was immediately satisfying! The results are ridiculously fun for both the person playing in VR and with Mobile Spectator.
Viola! A live feed of the Mobile Spectator camera view.
From inside the HMD, the phone is shown floating in space with a screen attached to it, displaying the live view from the mobile phone to the VR player. Similar to the cameras in previous versions of Spectator Mode, this live view allows the player in VR to frame themselves within a shot, as well as see where their friends are and interact more directly.
On the mobile side, the player in VR is represented by their avatar on the phone screen. Mobile Spectator players can walk around the physical play space alongside the person in VR, using the phone screen as a window into the virtual world. You are free to look around wherever you choose and experience the game world in a more active way outside of VR!
Interaction
Once we found a solution for displaying the Mobile Spectator in the VR world, we knew immediately that we had to find ways to add more interactive elements.
Take photos and save them directly to the mobile device!
Everyone loved to pose at the floating Mobile Spectator camera, so adding a camera capture button to the app was an easy choice. By tapping the camera button on the phone, the device tells the PC to capture a high resolution frame. This image is then sent back to the phone and saved directly into the phone’s camera roll, ready for editing or sharing on social media. It’s so quick and natural it feels like taking a photo of the real world!
Interact with players in VR by bombarding them with beach balls!
Interaction from the Mobile Spectator app seemed compelling, so we also added a button to the mobile app that, when pressed, will toss beach balls at the player in VR! Keep in mind that, while this is essentially a multiplayer component, we didn’t have to build a custom network to support this. This especially feels promising— the interaction between phone and VR users with Mobile Spectator is unlike any solution we’ve encountered before.
Considerations
Mobile Spectator is a truly unique experience that allows you to get a glimpse into the VR world— straight from your mobile phone.
Mobile Spectator— a ridiculously fun, experimental way to share VR!
There are various considerations that make Mobile Spectator a formidable undertaking, including some significant challenges we encountered. First, the feature adds quite a bit of performance overhead to the PC with additional rendering and video encoding. The mobile device is also heavily taxed by simultaneously running ARCore and decoding video. Additionally, latency is noticeable due to the encoding/decoding process and round-trip delay time. These are just a few of the challenges we’ve noted, but we think it’s just as important to share these findings as part of the experimentation process.
We are very excited to be sharing our experiments with Mobile Spectator. The “Labs” in our name is a statement of purpose and, as such, we place a great importance on being open about our latest experiments and findings.
Thanks for reading— now, back to science!!
-Owls
Credit to the entire team for their feedback on this project, and to the Owls that worked directly on Mobile Spectator: Devin Reimer, Ben Hopkins, and Ryan Dawson!