Object-based rendering and 3D reconstruction using a moveable image-based rendering system

Image-based rendering (IBR) is a promising technology for rendering photo-realistic views of scenes from a collection of densely sampled images or videos, and for developing revolutionary immersive viewing systems. However, most multiple camera systems are designed to be stationary and hence their ability to cope with moving objects and dynamic environment is somewhat limited. This paper studies the design and development of a movable IBR (M-IBR) system for object-based rendering and 3D reconstruction in large environment. To reduce shaky motion due to the movable system, we develop video stabilization algorithm based on scale invariant feature transformation. Also, we propose a new segmentation-based mutual-information matching algorithm to estimate the dense depth maps and preserve discontinuities for better quality of rendering. Using these depth maps and sequential structure from motion technique, we estimate the location of the M-IBR system to obtain a set of fairly reliable 3D point cloud. After detecting and removing possible outliers using Kalman-based tracking, we propose a new robust RBF-based modeling of the 3D surfaces to recover high quality 3D geometry. The effectiveness of the proposed system and algorithms is demonstrated using multi-view videos in an open environment.