Robust depth estimation and its applications to HDR imaging and free view generation

Abstract

Stereo matching is one of the most active research topics in computer vision. The generated depth maps are widely used in many applications such as 3D reconstruction, 3DTV, object recognition, autonomous systems, etc. Stereo matching generally performs four steps: 1. matching cost computation; 2. cost aggregation; 3. disparity computation; and 4. disparity refinement. In this thesis, we propose some approaches to improve the performance of each step. Meanwhile, specially designed matching algorithms are proposed for the applications to HDR imaging and free view generation. Firstly, we propose a novel matching cost computation algorithm named Optimal Local Adaptive Radiometric Compensation (LARAC), which is robust to the radiometric differences caused by illumination and exposure variations. In LARAC, we approximate the spatially varying Pixel Value Correspondence Function (PVCF) as a locally consistent polynomial with an optimal window size, where the polynomial coefficients are estimated optimally with a closed-form solution. The simulation results suggest that the proposed LARAC outperforms other state-of-the-art stereo algorithms. Secondly, we improve the performance of the cost aggregation step in stereo matching. While many existing cost aggregation algorithms trade off between accuracy and computational complexity, we propose a novel algorithm named Stereo Matching by Adaptive and Recursive Technologies (SMART) which has low complexity while maintaining high accuracy. In SMART, a novel cost aggregation algorithm is proposed to increase the accuracy and a recursive cost aggregation algorithm is invented to reduce the aggregation complexity from O(n²) to O(1). Thirdly, a novel disparity refinement algorithm, named hybrid plane fitting, with low complexity and high accuracy is proposed. We propose a novel robust cross checking algorithm to exclude outliers in the disparity maps. According to the outlier percentage of each plane, we propose a hybrid method, either RANSAC based plane fitting, or the proposed weighted LSE based plane fitting, to estimate the plane parameters and refine the disparity maps accordingly. Fourthly, a multi-exposed stereo camera system for high dynamic range imaging is proposed. The stereo setup captures two images simultaneously and allows for HDR image composition for dynamic scenes and HDR videos. A specially designed algorithm named Multi-exposed stereo matching (MEX) is proposed to generate accurate disparity maps from two images with different exposure times. Image warping is performed afterwards to generate two aligned images, with the warping errors detected by the proposed warping error detection algorithm. A HDR image is recovered by the two aligned images, with the unreliable pixels interpolated by the proposed radiance consistent hole filling algorithm. Finally, a novel ray-space based free view generation algorithm based on Radon transform is proposed to generate virtual views in-between adjacent real views to realize free viewpoint television (FTV) applications. In the proposed method, the correspondence searching problem in the two-view configuration is extended to a ray space correspondence search problem in the multi-view configuration. The corresponding pixels in the neighboring real views for each pixel are detected by the proposed robust block matching algorithm, in which the smoothness property of the disparity field and the correlation among the neighboring views are explored.

Date of Award	2014
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology