Enhancing depth-based human action recognition through zero-shot depth estimation

Abstract

Diese Masterarbeit befasst sich mit der Herausforderung der begrenzten Verfügbarkeit von Tiefendaten in der menschlichen Bewegungserkennung (Human Action Recognition, HAR), indem synthetische Tiefenbilder aus RGB-Videos mittels Zero-Shot- Tiefenschätzung erzeugt werden. Obwohl Tiefendaten für HAR Vorteile wie Robustheit gegenüber Lichtverhältnissen und verbesserten Datenschutz bieten, limitiert die begrenzte Anzahl und der Umfang verfügbarer Tiefendatensätze im Vergleich zu RGB-Datensätzen die Modellentwicklung. Die Forschung schlägt ein Framework vor, das RGB-Datensätze menschlicher Bewegungen mithilfe von Zero-Shot-Tiefenschätzungsmethoden in synthetische Tiefenbilder umwandelt, um die Trainingsdaten für tiefenbasierte HAR-Systeme zu erweitern. Durch vergleichende Experimente zwischen realen und synthetischen Tiefenbildern sowie die Untersuchung von Domänenadaptationstechniken.

Human Action Recognition (HAR) is a computer vision task that identifies human activities from visual data, with applications ranging from healthcare to robotics. While RGB video is the standard input modality, depth data, which measures distance from a sensor, offers advantages such as robustness to lighting changes and enhanced privacy. However, progress in depth-based HAR is hindered by the scarcity of large, publicly available depth datasets compared to RGB. This data limitation restricts model performance and generalization. To address this, we show that high-quality synthetic depth data, generated from RGB videos using off-the-shelf zero-shot depth estimation models, can effectively augment limited real depth data for training robust HAR systems. While previous works have used estimated depth as a complementary modality to RGB or evaluated it only on synthetic depth data, our work is the first to demonstrate that synthetic-to-real transfer is effective for depth-only HAR on real sensor-captured datasets. We find that augmenting a small fraction of real data with synthetic data can recover performance lost due to data scarcity, and can even surpass the accuracy of models trained on the full real dataset, when using recent, robust video understanding architectures for HAR. This framework provides a scalable and cost-effective solution to the data acquisition bottleneck in depth-based HAR. By enabling the use of large existing RGB datasets, our approach supports the development of privacy-preserving, reliable, and practical depth-only HAR systems for real-world applications.