Evaluating Pose Awareness and 3D Consistency in Semantic Matching

Abstract

Semantic matching is increasingly adopted in robotics as a flexible approach for generalizing point transfer across different manipulation tasks, creating a need for suitable benchmark datasets and a reassessment of existing evaluation metrics. Point correspondences used for manipulation require not only pixel-level precision but also accuracy in 3D space. Current methods and evaluation metrics suffer from a bias toward the pixel position relative to the image at the expense of accurate 3D localization. Objects’ orientation differences between source and target images severely affect matching results. In this paper, we introduce a novel evaluation procedure for assessing pose-aware and 3D-consistent semantic matching, supported by a synthetic dataset, SemanticHouseCat3D, that includes rich annotations of household objects. Our evaluation features a new orientation-based assessment that bins point matching metrics according to changes in object orientation and precisely measures the 3D accuracy of estimated image points after their re-projection. Using SemanticHouseCat3D, we conduct an exhaustive evaluation of state-of-the-art methods and investigate the influence of full-image context on performance compared to techniques like object cropping and segmentation. Our results indicate that while foundation models are reliable and adaptable across different domains, there is a critical need for improving semantic matching methods in terms of pose awareness and feature localization.