Constraint-Informed Temporal Action Segmentation

Abstract

Temporal action segmentation is essential for learning robotic skills from demonstration, particularly in contact-rich manipulation tasks composed of multiple sequential sub-actions. While prior methods use visual and proprioceptive signals, they typically encode motion data implicitly. In this paper, we propose a constraint-informed temporal action segmentation framework that explicitly integrates motion constraint information derived from end-effector velocity and force/torque profiles into a multistage prediction model. The key idea is to use constrained motion directions as signals for sub-task transitions and to jointly predict both action labels and constraint states at each stage. We validate our approach on a simulated peg-in-hole insertion task and compare it with several baselines. Results show that incorporating constraint cues improves segmentation accuracy and temporal alignment, highlighting the value of structured physical interaction signals in segmenting complex robotic tasks.