Accurate and realistic dense 3D reconstruction from multiple images

Abstract

3D modeling is an important task to preserve and visualize real-world scenes by computer. The applications include but are not limited to heritage preserving, city-scale surveys and AR/VR applications. Typical image-based 3D reconstruction methods include SfM, MVS, meshing and texturing. This thesis aims at improving the last three steps with neural network techniques so that the reconstruction pipeline is capable to produce a 3D model with high accuracy and realistic appearance. First, we introduce visibility handling into learning-based stereo matching systems to improve the quality of estimated depth maps. We propose to explicitly detect the visibility and recover the erroneous pixels by neighborhood or other views. Second, we adopt differentiable rendering in neural implicit surface optimization to simultaneously obtain accurate geometry and realistic appearance. We investigate effective geometric prior and critical regularizations to improve the ability of generalization and robustness of the system. Third, we decompose the appearance into environmental lighting and physical-based material to support efficient rendering in arbitrary novel environments. We discuss possible techniques to reduce the ambiguity between environment and material, and provide an approximated indirect illumination handling method to improve the estimation quality in complex scenes. The proposed modules are extensively evaluated on multiple datasets, including both synthetic and real-world data.

Date of Award	2022
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Long QUAN (Supervisor)