Learning-based video super-resolution

Abstract

The goal of video super-resolution (VSR) is to generate the high-resolution (HR) sequence based on the low-resolution (LR) input. In the past several decades, multi-image super-resolution (MISR) methods have dominated VSR. However, MISR, which generates each HR frame independently, does not consider the temporal correlations among reconstructed HR frames, causing the artifact in the visual consistency. In the meantime, traditional MISR methods cannot handle situations with complex motions because of requiring highly accurate motion estimation. In our work, we propose a sequential model, Bidirectional Convolutional Long Short Term Memory (Bi-ConvLSTM), to explore temporal dependencies and spatial coherence through forward and backward directions for VSR. We further take each input and output of Bi-ConvLSTM as two successive LR and HR frames to avoid motion estimation and preserve output temporal correlations. Taking advantage of the temporal information provided in the successive frames through Bi-ConvLSTM makes the output temporally coherent and better visualized. To further improve the VSR performance, we propose a novel adaptation framework which utilizes test data's self-information for specic model refinement. Experiments illustrate that our Bi-ConvLSTM outperforms the state-of- the-art VSR methods and the adaptation framework can further enhance the VSR performance.

Date of Award	2016
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology