Hand tracking in colocated multi-user virtual reality

Abstract

To enhance immersion in virtual reality applications, this dissertation addresses the challenges and opportunities of implementing natural hand interactions through hardware- based hand tracking, particularly in the context of multi-user colocated VR environments. The research introduces ’EasyHand’, a hand-tracking framework that unifies detection mapping, visualization, interaction, and networking for several tracking systems to facilitate intuitive hand tracking for visualization and interaction while addressing the limitations of tracking range dead spots in colocated scenarios. The first experiment describes a novel approach for creating colocated multi-user VR scenarios for SLAM-tracked (Simultaneous Localization and Mapping) VR headsets without the need for external tracking cameras to continuously track all users. By leveraging the hand recognition capabilities of the VR headset, the system synchronizes the virtual space for colocated users. This method is compared to alternative approaches such as initial positioning and ArUco marker recognition, with a comprehensive evaluation of accuracy, consistency, and simplicity, demonstrating the superior performance of the proposed hand tracking-based calibration method. The dissertation proceeds with an experiment that demonstrates a method for trans- forming hand data detected via an RGB camera and the MediaPipe framework into 3D space. This technique includes user-specific hand length estimation to determine 3D hand positions, enabling the detection of multiple hands simultaneously. This allows a hand detection system to track the hands of other users and thus also support their detection systems in the event that they are not able to see these hands. Comparative analysis with Oculus Quest and Leap Motion, conducted under different conditions (static & dynamic) and distances from the tracking device, confirms the effectiveness of the proposed method, with significantly extended tracking ranges for colocated scenarios. Finally, the dissertation explores methods for assigning tracked hands to colocated virtual users, introducing an algorithm that leverages past assignments to enhance future assignments’ robustness and effectiveness. Multiple assignment algorithms are evaluated, highlighting the precision of the proposed algorithm. Overall, this work provides initial insights into calibrating multi-user environments, compensating for tracking loss and assignment of hands to users for colocated VR scenarios, while maintaining user-specific interactions, representing a substantial advancement in VR technology.