ArchComplete: Autoregressive 3D architectural design generation with hierarchical diffusion-based upsampling

Abstract

Recent advances in 3D generative models have shown promising results but often fall short in capturing the complexity of architectural geometries and topologies. To tackle this, we present ArchComplete, a two-stage voxel-based 3D generative pipeline consisting of a vector-quantized model, whose composition is modeled with an autoregressive transformer for generating coarse shapes, followed by a set of multiscale diffusion models for augmenting with fine geometric details. Key to our pipeline is (i) learning a contextually rich codebook of local patch embeddings, optimized alongside a 2.5D perceptual loss that captures global spatial correspondence of projections onto three axis-aligned orthogonal planes, and (ii) redefining upsampling as a set of multiscale conditional diffusion models learning over a hierarchy of coarse-to-fine local volumetric patches, with a guided denoising process using 3D Gaussian windows that smooths noise estimates across overlapping patches during inference. Trained on our introduced dataset of 3D house models, ArchComplete autoregressively generates models at the resolution of (Formula presented) and progressively refines them up to (Formula presented), with voxel sizes as small as (Formula presented). ArchComplete solves a variety of tasks, including genetic interpolation and variation, unconditional synthesis, shape and plan-drawing completion, as well as geometric detailization, while achieving state-of-the-art performance.