Benchmark and application of 3D vision tasks for ultra-wide FoV fisheye images

Abstract

This study introduce a new benchmark dataset seeking to facilitate the 3D vision research on ultra-wide Field-of-View (FoV) fisheye images, covering Two-View Stereo (TVS), Multi-View Stereo (MVS), and Novel View Synthesis (NVS) tasks. To collect the data and ensure precise evaluations, this study built up a portable capture device composed of well-calibrated ultra-wide FoV fisheye cameras and high-resolution lidar. The dataset is composed of 23 scenes covering indoor and outdoor environments, captured at daytime and nighttime. In addition, this study provides two forms of ground truth, spherical disparity maps, and fisheye depth maps, for two-and multi-view stereo tasks. By benchmarking the state-of-the-art methods, this study points out the limitations of existing approaches in dealing with ultra-wide FoV fisheye distortion. Finally, this study propose a practical depth-supervised spherical-grid-based neural radiance field (NeRF) that enhances novel ultra-wide FoV fisheye-view synthesis to state-of-the-art performance. Moreover, this research also introduces another application example of fisheye cameras in underwater robotics vision: UWA360CAM, demonstrating its efficiency and compatibility.

Date of Award	2024
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Sai Kit YEUNG (Supervisor)