Abstract:Motion compensation is one of the most essential methods for any video compression algorithm. Video frame prediction is a task analogous to motion compensation. In recent years, the task of frame prediction is undertaken by deep neural networks (DNNs). In this thesis we create a DNN to perform learned frame prediction and additionally implement a codec that contains our DNN. We train our network using two methods for two different goals. Firstly we train our network based on mean square error (MSE) only, aiming to obtain highest PSNR values at frame prediction and video compression. Secondly we use adversarial training to produce visually more realistic frame predictions. For frame prediction, we compare our method with the baseline methods of frame difference and 16x16 block motion compensation. For video compression we further include x264 video codec in the comparison. We show that in frame prediction, adversarial training produces frames that look sharper and more realistic, compared MSE based training, but in video compression it consistently performs worse. This proves that even though adversarial training is useful for generating video frames that are more pleasing to the human eye, they should not be employed for video compression. Moreover, our network trained with MSE produces accurate frame predictions, and in quantitative results, for both tasks, it produces comparable results in all videos and outperforms other methods on average. More specifically, learned frame prediction outperforms other methods in terms of rate-distortion performance in case of high motion video, while the rate-distortion performance of our method is competitive with that of x264 in low motion video.

Frame Prediction Using Recurrent Convolutional Encoder with Residual Learning

An End-to-End Future Frame Prediction Method for Vehicle-Centric Driving Videos

One-Step Time-Dependent Future Video Frame Prediction with a Convolutional Encoder-Decoder Neural Network

Next frame prediction using ConvLSTM

Adaptive Recurrent Frame Prediction with Learnable Motion Vectors.

Stacked residual blocks based encoder-decoder framework for human motion prediction

Video Frame Prediction by Deep Multi-Branch Mask Network

Inception-inspired LSTM for Next-frame Video Prediction

Exploring and Exploiting High-Order Spatial-Temporal Dynamics for Long-Term Frame Prediction

A Dynamic Frame Selection Framework for Fast Video Recognition.

Deep Learned Frame Prediction for Video Compression

Video Frame Prediction with Dual-Stream Deep Network Emphasizing Motions and Content Details.

Folded Recurrent Neural Networks for Future Video Prediction

Looking-Ahead: Neural Future Video Frame Prediction

Deep Frame Prediction for Video Coding

Unit Frame 3 : T + 2 Frame 1 : T Frame 2 : T + 1 FrameT + 1 FrameT + 2 FrameT + 3

Pyramidal Predictive Network: A Model for Visual-Frame Prediction Based on Predictive Coding Theory

Z-Order Recurrent Neural Networks For Video Prediction

Predictive Coding Based Multiscale Network with Encoder-Decoder LSTM for Video Prediction

Recurrent Residual Module for Fast Inference in Videos

Efficient Traffic Accident Warning Based on Unsupervised Prediction Framework