Abstract:The success of deep learning-based techniques in solving various computer vision problems motivated the researchers to apply deep learning to predict the optical flow of a video in the next frame. However, the problem of predicting the motion of an object in the next few frames remains an unsolved and less explored problem. Given a sequence of frames, predicting the motion in the next few frames of the video becomes difficult in cases where the displacement of optical flow vector across frames is large. Traditional CNNs often fail to learn the dynamics of the objects across frames in case of large displacements of objects in consecutive frames. In this paper, we present an efficient CNN based on the concept of feature pyramid for extracting the spatial features from a few consecutive frames. The spatial features extracted from consecutive frames by a modified PWC-Net architecture are fed into a bidirectional LSTM for obtaining the temporal features. The proposed spatiotemporal feature pyramid is able to capture the abrupt motion of the moving objects in video, especially when displacement of the object is large across the consecutive frames. Further, the proposed spatiotemporal pyramidal feature can effectively predict the optical flow in next few frames, instead of predicting only the next frame. The proposed method of predicting optical flow outperforms the state of the art when applied on challenging datasets such as "MPI Sintel Final Pass," "Monkaa" and "Flying Chairs" where abrupt and large displacement of the moving objects in consecutive frames is the main challenge.

Pyramid Structured Optical Flow Learning with Motion Cues

RAPIDFlow: Recurrent Adaptable Pyramids with Iterative Decoding for Efficient Optical Flow Estimation

Unsupervised Learning for Optical Flow Estimation Using Pyramid Convolution LSTM

Particle Image Velocimetry Based on a Deep Learning Motion Estimator.

Learnable spatiotemporal feature pyramid for prediction of future optical flow in videos

Frame Interpolation Using Phase and Amplitude Feature Pyramids

A Dense Optical Flow Registration Algorithm Based on Deep Learning

FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation

Unsupervised Optical Flow Estimation Based on Improved Feature Pyramid

Optical Flow Estimation Using Dual Self-Attention Pyramid Networks

ASFlow: Unsupervised Optical Flow Learning with Adaptive Pyramid Sampling

Recurrent Spatial Pyramid CNN for Optical Flow Estimation

A Unified Pyramid Recurrent Network for Video Frame Interpolation

FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

PRAFlow_RVC: Pyramid Recurrent All-Pairs Field Transforms for Optical Flow Estimation in Robust Vision Challenge 2020

LCIF-Net: Local criss-cross attention based optical flow method using multi-scale image features and feature pyramid

OFR-Net: Optical Flow Refinement with a Pyramid Dense Residual Network.

FPCR-Net: Feature pyramidal correlation and residual reconstruction for optical flow estimation

ReFlowNet: Revisiting Coarse-to-fine Learning of Optical Flow

Deep Networks for Image Motion Estimation