Learning dexterous robot hand control by imitating human hands

Abstract

This paper presents an unsupervised deep-learning method for controlling dexterous robotic hands by mimicking human hand motions. We introduce a cross-domain similarity metric to capture the spatial and kinematic relationships between human and robot hands. Using this metric, our approach learns a shared latent space that aligns motion features across the two embodiments. The framework consists of two separate encoders that map human and robot hand data into the latent space, along with a robot decoder that generates feasible robot hand motions. During inference, only the human hand encoder and the robot hand decoder are needed to seamlessly retarget human hand movements to the robot hand, enabling scalable and flexible motion retargeting without requiring paired human-robot data. To demonstrate real-world applicability, we integrate our motion retargeting system with Mediapipe, a human hand pose estimator, enabling real-time robotic hand control from RGB video input.