Advancing Deep Learning for 3D Medical Reconstruction from Sparse-View X-Rays

Abstract

3D imaging techniques, such as computed tomography (CT) and magnetic resonance imaging (MRI), are essential in the medical field, enabling the visualization of internal anatomical structures to support a wide range of clinical scenarios, including diagnosis, surgical planning, and intraoperative navigation. In orthopedics, 3D volume data (e.g., CT/MRI) can be further processed to extract underlying surfaces of bones for applications such as surgical guide design and orthopedic surgery simulation. However, acquiring 3D medical data for clinical use comes with certain limitations — CT scans involve high radiation exposure, posing long-term health risks; MRI requires lengthy scanning times, is impractical for emergencies or uncooperative patients, and is unsuitable for those with metallic implants; Extracting bone surface models is time-consuming and requires specialized expertise. In this thesis, we explore an alternative approach to obtain 3D medical data — 3D reconstruction from sparse-view X-ray, as X-ray can penetrate the human body and multi-view imaging can provide rich spatial information, thus helping reveal internal 3D structures. To tackle challenges in solving the inverse problem, this thesis presents several novel deep learning frameworks for cone-beam CT (CBCT) reconstruction from extremely sparse X-rays, as well as bone model reconstruction from biplanar X-rays.

In the first four parts of this thesis, we develop a series of innovative frameworks for the reconstruction of CBCT from fewer than ten projections. We represent CT as a continuous implicit function that defines the attenuation coefficient of any point in 3D space and reformulate the reconstruction task as a conditioned implicit function, where the sparse-view inputs serve as the condition. Based on this formulation, we developed DIF-Net, the first data-driven reconstruction framework based on implicit neural representations. Specifically, a 2D encoder is applied to extract multi-view features from input projections, and DIF-Net leverages projection parameters to interpolate view-specific features for estimating attenuation coefficients. We then extended DIF-Net to DIF-Gaussian by introducing 3D Gaussian representations to incorporate additional spatial information. Furthermore, we proposed C²RV, which leverages multi-scale volumetric representations and a scale-view cross-attention module to enable efficient crossregional feature learning and adaptive multi-view aggregation, achieving superior reconstruction quality. Finally, we introduced DeepSparse, the first foundation model for sparse-view CBCT reconstruction pretrained on large-scale datasets, designed to improve adaptability and reconstruction quality across diverse target datasets. The effectiveness and efficiency of these reconstruction methods were validated through extensive experiments on various datasets, including knee, chest, and dental CBCT.

The final part of this thesis focuses on 3D reconstruction from biplanar X-rays, shifting the reconstruction target from CT to bone surface models. Although two X-ray projections lack sufficient information to reconstruct internal details, they clearly visualize bone boundaries due to the much higher density of bones compared to surrounding soft tissues. Then, we introduce SSR-KD, a fast and accurate framework for reconstructing high-quality bone models from biplanar X-rays. We represent the bone model as an occupancy field, and the network takes X-ray projections as input and predicts the occupancy value of each point in 3D space, forming an occupancy field. Notably, high tibial osteotomy simulations performed by experts demonstrated that bone models generated from biplanar X-rays exhibit clinical applicability comparable to those derived from CT scans. Overall, SSR-KD accelerates the reconstruction process, reduces radiation exposure, enables intraoperative guidance, and significantly enhances the practicality of bone models, offering transformative applications in orthopedics.

Date of Award	2025
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Xiaomeng LI (Supervisor)