Single portrait image matting and bokeh effect synthesis via multiplane images

Abstract

High-quality portrait photography has become an essential function in our daily lives. However, due to the limited aperture and focal length of a smartphone camera, images captured by a smartphone could not match the same level of bokeh effect by a digital single-lens reflex camera. A common solution on a smartphone is to simulate out-of-focus effects from an all-in-focus image, where the key is robust depth estimation and portrait matting. To achieve this, we propose a multi-stage, multi-branch matting network to estimate a strand-level portrait alpha mask, which is then used to refine the coarse depth map from the pre-trained model. By integrating the input portrait image with the estimated depth map and alpha mask, we propose a multi-plane image (MPI) representation that includes multiple layers of disk-blurred images, where the kernel size is proportional to the absolute depth distance to the focus layer. Then a depth-aware blurring process is applied to enforce the bokeh effect. Besides, each MPI layer has an alpha channel controlling the visibility according to the corresponding depth. Finally, an image with bokeh is rendered by compositing all MPI layers. We conduct comprehensive experiments to evaluate our method, which demonstrates that our method can generate more accurate alpha masks and more realistic images with bokeh, compared to prior work.

Date of Award	2023
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Qifeng CHEN (Supervisor)