Towards robust 3D object generation and understanding : resolving structural, perceptual, and vulnerability challenges

Abstract

As applications in autonomous systems, robotics, and virtual environments continue to grow, the need for robust and accurate 3D object generation and understanding systems becomes increasingly critical. These systems must interpret and manipulate three-dimensional environments with precision and resilience, particularly in the face of real-world challenges such as occlusions, sparse data, and diverse viewpoints. This thesis addresses these challenges by investigating the vulnerabilities of 3D models and proposing innovative techniques to enhance their robustness and applicability. The first focus of this thesis is on point cloud inversion and editing, addressing the unique challenges posed by the unordered and discrete nature of point cloud data. This research introduces a novel framework that maps point clouds into latent spaces, enabling flexible and precise manipulation of 3D objects while preserving geometric integrity and feature disentanglement. The second contribution tackles the issue of occlusion handling in 3D generation, proposing a bi-level Gaussian splatting framework to reconstruct missing or occluded regions with high fidelity. This approach significantly enhances the realism and structural consistency of 3D models generated from sparse or incomplete views. The third focus is on improving multi-view consistency in 3D generation. This research leverages pretrained generative models and introduces a self-calibrated latent refinement process to achieve scalable and efficient multi-view synthesis. By minimizing reliance on extensive 3D datasets and computational resources, the proposed framework ensures geometric alignment across multiple perspectives. Finally, this thesis investigates adversarial vulnerabilities in 3D classification models, exploring how minimal perturbations can expose critical weaknesses in robustness. By analyzing shared representations between generation and classification tasks, this work underscores the importance of robust evaluation frameworks and provides insights to guide the development of more resilient 3D systems. In summary, this thesis advances the field of robust 3D object generation and understanding through contributions in point cloud editing, occlusion-aware generation, multi-view consistency, and adversarial evaluation. Together, these studies establish a comprehensive framework for enhancing the reliability, scalability, and robustness of 3D technologies in real-world applications.

Date of Award	2024
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Sai Kit YEUNG (Supervisor)