Neural Visual Representation and Compression

Abstract

With the growing prevalence of visual data, efficient and compact representation methods have become essential. Recent advancements in implicit neural representations (INRs) and Gaussian Splatting techniques have garnered significant attention for their ability to offer high-fidelity reconstruction, efficient storage, and real-time rendering for a variety of visual data, including images, videos, and static and dynamic 3D scenes. This thesis explores three novel methods that enhance the efficiency and effectiveness of neural visual representation and compression.

Firstly, we propose a universal boosting framework for implicit video representation methods by incorporating a conditional decoder to align intermediate features with target frames. This framework significantly improves the reconstruction quality and convergence speed of multiple baseline INRs in video regression tasks, while also providing superior inpainting and interpolation results. Moreover, we introduce a consistent entropy minimization technique and develop competitive video codecs based on these enhanced INRs.

Secondly, to address the challenges of excessive GPU memory consumption and slow rendering speeds in INRs, we present GaussianImage, a groundbreaking image representation and compression paradigm based on 2D Gaussian Splatting. By utilizing a compact set of 2D Gaussians and a novel accumulated blending-based rasterization algorithm, this method greatly enhances memory efficiency and achieves rendering speeds of 1500-2000 FPS, independent of parameter size. Furthermore, we integrate vector quantization techniques to create the first ultra-fast neural image codec.

Finally, we introduce MEGA, a memory-efficient framework for dynamic 3D scene reconstruction using 4D Gaussian Splatting. By decomposing the color representation into per-Gaussian direct color components and a lightweight alternating current predictor, and by incorporating an entropy-constrained Gaussian deformation technique, MEGA significantly reduces storage requirements while maintaining photorealistic rendering quality and ahigh rendering speeds.

Date of Award	2025
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Jun ZHANG (Supervisor)