Informed patch sampling for 3D medical image reconstruction

Abstract

Efficient analysis and processing of 3D volumes are crucial in clinical practice. A common task in 3D medical image processing is object segmentation, where objects of interest are delineated. However, segmenting large volumetric scans requires substantial memory and computational power, making end-to-end segmentation both memory- and computationally intensive. A potential solution to reduce these costs is to select only a subset of the input, segment this subset, and estimate the values of the remaining regions by reconstructing them from the segmented subset. Our hypothesis is that the choice of subset influences reconstruction performance, with some regions of the input volume being more informative for reconstruction than others. To test this hypothesis, we propose a neural network capable of identifying subsets that contribute most to accurate reconstruction. To simplify the process and focus on the core task, we assume that binary segmentations of the objects of interest are provided and select subsets directly from them, reconstructing the original binary segmentations afterwards. We build our neural network upon an existing point cloud-based network that learns to select representative points, and integrate it into a novel end-to-end pipeline for reconstructing full volumes from limited input data. We modify the original point cloud–based loss function to operate on voxel grid data and introduce conversion and extraction mechanisms that enable transitions between voxel grid and point cloud representations. Our proposed pipeline first converts the input voxel grid into a point cloud representation to enable efficient geometric processing. A neural network architecture then processes the point cloud and predicts a set of candidate centers for volumetric patches. These predicted centers are subsequently used to extract the output patch set, which is then fed to the downstream reconstruction network. We evaluate our pipeline on two datasets of medical shape segmentations with varying geometrical complexity. Our experiments show that the proposed learned sampler identifies informative regions, which support reconstruction performance, especially for complex shapes and limited spatial context. We further evaluate the effect of reconstruction network quality across different input configurations, varying patch size and number of patches, and show that our approach is effective when reconstruction accuracy is poor or when the input shape has complex geometry. Finally, we analyze the computations and memory demands of the proposed pipeline, showing that the additional overhead remains under 1 GB of memory and 0.5 s of extra computation, making the method suitable for deployment in resource-limited environments.