How to Hands: A Developer Deep Dive on Hand Tracking in Vacation Simulator
Bringing Hand Tracking support to Vacation Simulator was quite the journey—one we didn’t even think was possible at first! What started as an R&D project to enumerate why hand tracking wouldn’t (or wouldn’t yet) work ended up becoming a viable, shippable project—one that translated the entirety of Vacation Simulator into something you could play, end to end, with only your hands. In fact, Vacation Simulator wound up being the perfect proving ground for hand tracking: it revolves around direct hand interaction and contains almost every different type of grabbing, manipulating, and gesturing that you can think of. If you haven’t yet tried hand tracking in Vacation Simulator it’s available now for Oculus Quest and Quest 2.
We could fill a small novel with all the lessons learned, but wanted to discuss a few of the most interesting aspects of development.
Grabbing Is All About Intention
The vast majority of hand tracking applications to date use pinches and laser pointers to solve the problem of “grab”. For us it needed to feel natural and require no special gesture. Our first discovery was a good one, Tomato Presence (not seeing your hand when holding an item because the item acts as a stand-in) seemed to work just as well with hands as controllers. After that, things got way more complicated.
As for grabbing itself, our first approach was to map grab and release to a binary style input. That is, you either are grabbing or you aren’t grabbing—there’s no in between. While it technically “worked”, it definitely didn’t universally feel good or work well for all types of objects or interactions.
A big problem with hand tracking is that your input is muddy. Like, very muddy. Not only is the finger tracking data with current technology at times very poor, the player’s intention is also not clearcut. You can’t just say finger curl is X therefore intention is Y. Instead, you have to think about what the players’s intentions are as a whole and then determine what other data you can utilize to make a better guess on what the desired outcome should be.
For example, if you are grabbing a small object you expect to have to close your hand more than if you are grabbing a big object (and even that is over simplified; it also has a lot to do with shape and general grab orientation). But if you close your hand too tightly on an object and we don’t successfully register a grab, your finger colliders might actually push the item out of the way, as you feel no resistance.
Our solution was to create a custom “grab threshold data”—basically, how much do you expect to have to close your hand—for every single object and interactable item in the game (which is many hundreds) and then playtest. Again. And again. And again.
Even with all this effort, it wasn’t the full solution. In controller-based grabbing, if you hold the grab button and then come near an object it won’t grab (which makes total sense: you don’t want a closed hand to grab!). With hand tracking you want the same thing, except without real world object resistance you don’t know if your hand is already in a grabbed state when nearing an object.
Our solution was to introduce “retry grab intervals”. Let’s say your hand entered the grab area of an object and was just passed the threshold for grabbing it (that is, slightly too closed to grab the object). We store how closed your hand was and re-check once it closes by a certain additional percentage. So, if you keep trying to grab the object by closing your hand more, you can succeed. This fixed a lot of the difficulties in grabbing without real world resistance.
Releasing? Way Harder Than Grabbing
The penalty for releasing at the wrong time is high, even in a relaxed game like Vacation Simulator. Accidentally dropping an item you’ve grabbed is definitely frustrating, but so is trying to release repeatedly and finding an object still ‘stuck’ on your hand. We had to use as much data as we could to determine overall intention.
Working on release gave us the clearest picture of just how challenging current finger tracking data is. It’s very easy to occlude fingers, for hands to leave the tracking area, and for velocity to be high enough that no accurate data is available. Over time we determined that we would just use the index finger as our data point for grab and release. As long as people try to grab items naturally, this gives us just enough data to work with. It isn’t the perfect solution, and there might be a way to use the thumb in certain situations, it’s just not something we’ve solved yet in a way that didn’t introduce different problems.
Lack of resistance is a problem in grabbing, and it turned out to be a much bigger problem in releasing. At its core, release timing is critical, but without physical feedback to a player it’s unclear exactly when they grab, so it’s even harder to tell when they release.
To solve this, we came up with the concept of “overgrab release”. For each grab threshold there is also a larger overgrab threshold. Once the user has grabbed beyond that threshold, we then reduce the amount they need to open their hand to release. This required some tuning, but instantly made the game feel more responsive.
Haptics, We Miss You
Going into this project we knew we would miss haptics, but the adage holds true: You don’t know how much you miss it til it’s gone.
Haptics provide so much data to a user that’s integral to gameplay. To make up for it, we found every single place that had a haptic and replaced it with a spatialized sound effect at that location. This instantly improved gamefeel. Players better understood when they grabbed and released items, which made the game feel more responsive in addition to helping with our grab release work. While not a direct replacement, it was at least helpful in filling the haptic gap.
Throwing…Oh My
We knew this one would be hard, but yikes! Everything is working against you! While we don’t think we solved this problem, given the quality of data we’re happy and impressed that we made it work enough for players to complete the game.
There are so many problems with throwing:
#1 Hand Speed - When you throw, often your hand is moving too fast to get finger data or even any hand tracking data.
#2 Camera View - Often (especially with overhand throws) your hand completely leaves the camera tracking area.
#3 Tracking Delay - When your hand re-enters the tracking area and/or slows enough to be tracked there is often a long delay on re-establishing tracking.
#4 Hand Angles - Some hand angles in front of you are very bad for hand and finger tracking and a lot of those angles happen when throwing.
#5 Release Timing - As mentioned above release timing is already a huge problem and the concept of throwing is all about release timing.
Much of our work on releasing revolved around throwing. It was a balancing act of improving overall releasing while not making throwing worse and vice versa. This topic is big enough to be its own blog, but one piece we want to talk about is high velocity throw intention.
A really cool thing we came up with was the “invisible throw”. If you were holding an object, lost all tracking, and then regained tracking and your hand was moving the direction you are looking and outside a bubble around you, we would make an educated guess you wanted to overhand throw. We take the current placement of your hand, ignore your finger positions, and then use your head position and rotation to estimate the shoulder position of the corresponding hand. With that, we build a vector from your shoulder to your hand, apply some tuned numbers, and launch the object in that direction. Even this explanation is somewhat simplified, but the end result is that players can perform overhand throws when we have almost no data at all to work with.
Waving: The Best Gesture
In Vacation Simulator you can wave at bots to start conversations, and it’s fun and intuitive. This was one place where hand tracking made the gameplay even better. With waving there is no abstraction at all in what you are doing. The game feel is unmatched.
Teleporting: The Necessary Gesture
Non-natural gestures have many problems in hand tracking. Unlike using a controller, you can’t spam buttons to figure it out. Instead, it’s about teaching a player to do something that may feel unnatural or unclear. To use our teleport gesture, simply face the direction you want to teleport, hold your hand with your palm up and curl your fingers. This gesture ended up being easy to teach players—it almost feels like drawing the space towards you. It’s also one-handed and comfortable for repeated use.
Overall, we’re very happy with where our teleport gesture ended up. We can’t say if this would work for every game, but for Vacation Simulator it’s a great fit.
Tracking Types
We ended up implementing a few different tracking types when holding objects depending on expectations. For most objects, we continue to use low quality tracking data when holding items for a short period of time, as it feels far superior to locking in place, even if it’s less accurate. This isn’t universal, though.
For example, we only use high quality tracking data for moving the ice pick, as a mistake has a high cost to the player. The handheld camera uses something completely different. We smooth all data coming in because slight hand tracking jiggles are very noticeable when looking at a camera screen.
Overall we discovered that once again it comes down to trade offs, and each interaction type required tuning for the optimal outcome.
Bringing Out The Backpack
When starting this project we were confident that the Backpack would be the showstopper. We assumed there was simply no way to grab something off your back given the camera view for hand tracking.
But working on the backpack taught us a very valuable lesson about hand tracking: the data you don’t have is sometimes just as important as the data you do.
Since there are only cameras on the front of the headset, and unlike a controller with an accelerometer and buttons to press, when you reach over your shoulder with hand tracking we have nothing to work with.
We’re so very excited that we found a solution, and it’s been wonderful to see folks (particularly those who have some hand tracking knowledge) give it a try and be amazed that it works.
In solving this problem our breakthrough was realizing we could keep track of the last tracked hand position before hand tracking was lost. If your hand was empty we could determine based on its location relative to the head if the intention was to reach over the shoulder. Then when you regained hand tracking, if that hand was closed and it was within a certain timing window we would then logically assume you meant to grab your backpack. It turns out putting your hand over your shoulder open and bringing it back in a closed is something you rarely do, so it ends up working out great.
The Twists and Turns with Dials
Dials are a tricky thing even with controllers, hand tracking only amplifies those problems. Most dial movement is based on finger movement (which we can’t accurately handle at present), and hand rotation with your fingers closed has very poor accuracy. Something as simple as releasing a knob could instead register as a large turn.
While this isn’t solved, our solution to improve it was to dampen rotation as your hand moves away from the knob. We noticed that as someone releases a dial, their hand naturally moves away from it. This dampening improved accuracy immensely.
Additionally, on dials with only two states we again assumed a lot more intentionality. If you start rotating a knob, even if you weren’t able to get it half-way, odds are you meant to turn it to a new state so we follow through. This is just the tip of the iceberg of everything we needed to do to make this work.
No Re-Making Menus
Menus with hand tracking are very difficult to pull off and generally are implemented with a completely different type of interaction like laser point and pinch. Fortunately for us, our goal in all our games is to make ‘menus’ in-game: that is, as actual interactable objects within the world as opposed to screens or overlays. When it comes to hand tracking, this decision benefited us immensely. No need to teach a player a new and often far more complicated way of interacting. Want to switch subtitles, recenter, or enable shorter human mode? Simply grab your backpack, zip the zipper and then directly interact with the switch, knobs, or drawers exactly how you interact with the world of the game itself.
Death by a Million…Finger Cuts
We had to make changes to pretty much every part of the project to deal with the never ending list of edge cases. Here are a few amusing examples:
Having to add a playspace recentering knob to the backpack because with hand tracking you no longer have access to a long press of the Oculus button to recenter
Dealing with every edge case that arises from having each finger being a unique physics item, including performance impacts. Like hitting the water with both hands and having it kick off 10+ sound effects for splash impact.
Detecting which hand is behind the other in cases of overlap to heavily dampen movement enough to allow a user to take off and put on a wristwatch.
Repositioning the grab volumes for glasses when on your face, as hand tracking doesn’t allow your hands to get as close to your face as controllers do.
Discovering that handles to adjust the height of tables are in our game by design at the height of your hands when relaxed at your sides. Which means often your natural finger curl will unintentionally grab the table and change its height. Solution was for those handles to get a custom grab type that requires the player to be looking at the handle for grabs to work.
The list goes on and on and on, but we hope you enjoyed that small taste!
Summary
While we’re still in the early days of hand tracking, we’re sold on it being an interesting and promising way to interact in VR and are confident it will only get better. Is hand tracking going to replace controllers for all VR games? Definitely not. Controllers are awesome, buttons are awesome, haptics and IMUs are awesome. But for certain games and content hand tracking will offer users a different, and in some cases, better and more natural experience.
-Owls