Towards temporally consistent video editing

Abstract

The ubiquity of video content in our daily lives has catalyzed a corresponding surge in demand for advanced video editing techniques. The main goal of video editing is the transformation of visual data into outputs that are not only visually pleasing but also temporally consistent. Such algorithms hold significant potential for a myriad of practical applications, enabling users to seamlessly remove unwanted objects, rectify facial anomalies, and generate aesthetically pleasing outcomes. Despite considerable advancements in video editing , persistent issues remain, notably the emergence of abrupt color artifacts and severe flickering, which detract from the realism of edited videos. This thesis is committed to enhancing video editing performance with a special focus on temporal consistency. Our approach is tripartite: firstly, we employ deep internal learning techniques to delve into the intrinsic structures of video data. By harnessing the power of neural networks, we aim to uncover and utilize the complex patterns and relationships embedded within the video content. As a result, we are able to achieve better video inpainting results, where missing or corrupted parts of the video are filled in seamlessly, maintaining the overall quality and integrity of the visual content. Secondly, we integrate 3D information to further stabilize temporal elements within the video. We use Multi-Plane Image (MPI) representation to formulate the visual data, allowing us to disentangle the different layers of the scene and provide a more accurate representation of the 3D structure. This integration of 3D information significantly enhances the stability and robustness of the video processing pipeline. Lastly, we refine video representation through the application of a content deformation field. We propose a novel approach to video representation that utilizes a 2D hash-based image field coupled with a 3D hash-based temporal deformation field. It can adapt existing image algorithms for application to video sequences with great consistency. The proposed video editing strategies, tested across various tasks, have proven highly effective in enhancing the fidelity and coherence of the editing process. By applying these algorithms and techniques, we preserve the original quality of the video while ensuring a smooth flow of content.

Date of Award	2024
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Qifeng CHEN (Supervisor)