In-Time 3D Reconstruction and Instance Segmentation from Monocular Sensor Data

Abstract

In most implementations of 3D reconstruction, depth information is provided by RGB-D sensors recording RGB and depth for each pixel. However, these sensors are still considerably expensive, characterized by high consumption of resources on the hardware itself, and have failed to reach mass markets of everyday mobile devices. With this paper, we aim to demonstrate the possibility of leveraging devices lacking 3D sensory data, in combination with external processing, to enrich the AR experience. We propose a client-server architecture that, in combination with a mobile client, allows for scanning of indoor environments as 3D semantically labeled sceneries while providing the user with instant feedback about the scanning results. At its core, the system is composed of the following expendable and exchangeable modules: (1) Depth Prediction from 2D images (2) Semantic Instance Segmentation of 2D images and (3) 3D Projection and Reconstruction. The result is a continuously updated mesh of the scenery with instance level segmentations in 3D. In comparison to the state-of-the-art, we are not only independent of the RGB-D input, but offer an architecture that complements and enhances the current AR client frameworks while demanding little extra computation from the mobile device.