Objective and Subjective Evaluation of a Multi-Stereo 3D Reconstruction System

Abstract

In recent years, 3D reconstruction systems comprising multiple depth sensors have received increasing interest for dynamic scene reconstruction and related applications. Publicly available ground truth data are of limited usefulness when dealing with quality assessment of self-recorded data delivered by customized stereo configurations. In this paper, we propose a framework that incorporates versatile strategies for quantitative and qualitative evaluation of a multi-stereo reconstruction system and its intermediate products. Besides the design of suitable calibration objects for quantitative measurements, the framework exploits multiview data redundancy and generated novel views for objective quality assessment and to obtain subjective ratings from users. We demonstrate the applicability of our evaluation system in experiments with several stereo matching algorithms and view fusion approaches along with a pair-comparison based user study. We believe that our proposed evaluation framework is beneficial for the assessment of 3D products derived from self-recorded dynamic data of comparable set-ups, for example, in the context of subsequent augmented reality applications.