Hands, Objects, Action! Egocentric 2D Hand-Based Action Recognition

Abstract

Action recognition is at the core of egocentric camera-based assistive technologies, as it enables automatic and continuous monitoring of Activities of Daily Living (ADLs) without any conscious effort on the part of the user. This study explores the feasibility of using 2D hand and object pose information for egocentric action recognition. While current literature focuses on 3D hand pose information, our work shows that using 2D skeleton data is a promising approach for hand-based action classification and potentially allows for reduced computational power. The study implements a state-of-the-art transformer-based method to recognise actions. Our approach achieves an accuracy of 95% in validation and 88% in test subsets on the publicly available benchmark, outperforming other existing solutions by 9% and proving that the presented technique offers a successful alternative to 3D-based approaches. Finally, the ablation study shows the significance of each network input and explores potential ways to improve the presented methodology in future research.