On-the-fly total variation minimization for low-latency video stabilization method
Tian Liu,Lei Zhang,Hua Huang
DOI: https://doi.org/10.11834/jig.170377
2018-01-01
Journal of Image and Graphics
Abstract:Objective A video captured with a hand-held device (e.g.,cell-phone or tablet computer) often appears remarkably shaky.It has a great negative impact to people's visual perception and is difficult for the following processings of the video.Therefore,video stabilization has always been one of the focuses in the field of image and video processing for a long time.Many of the previous methods focused primarily on the stability of the final results,while paying little attention to the processing delays.These methods spend too much time and they can only stabilize videos after obtaining most of the vid eo frames.These methods are difficult to handle the scenario that requires low latency processing.To solve this problem,a video stabilization method based on the on-the-fly total variation optimization is proposed.Method The first step is motion estimation.In order to meet the requirements of low latency.The algorithm first gets the feature points by detecting FAST features that are very fast.It is observed that a point is considered as a feature point when the difference of the pixel value between this point and other points in the neighborhood is large enough.Then it tracks the feature points by KLT to the adjacent frame.After that,it calculates the inter frame homography matrix through the feature points.Homography matrix can accurately describe the translation,rotation and other transformations between the video frames.And then we can get the camera path of the shaky video.The second step is motion smoothing.This is usually the time-consuming step of traditional methods.In this step,we should not only remove the shake of the camera path,but also need to try to avoid that the clipping and distortion rate of video is too high.In order to obtain the better effect of video stabilization and meet the requirements of low latency,the on-the-fly total variation minimization method is used to smooth the shaky path so as to obtain a stable path.This method greatly improves the computation speed.And it has a regularization parameter,so we can control the smoothness of the camera path by adjusting it.It can help us get better results.If we want a smoother camera path,we can increase this regularization parameter.However,this may make the optimization path more different from the original path,and results in excessive loss of image information.The visual result is that the video shows larger black edges.When the parameter is reduced,the smoothing effect may not be obvious and the shake may not be removed effectively.So we need to set the parameter according to the size of the camera path.After that,we can compute the motion compensation matrix through the stable camera path and shaky video.The shaky frames are then deformed according to the transformation matrix.Finally,a stable video can be generated.Result To test the effectiveness of this algorithm,a public dataset containing multiple categories of videos is used.This algorithm is compared with several video stabilization algorithms and commercial softwares that can provide good video stabilization results.Among these methods,there are three offline methods.They are the bundled camera paths method (BCP),the deformation stabilizer of Adobe After Effects CC 2015 (AE),and the online video stabilizer provided by YouTube.The two low latency video stabilization methods are video stabilization method based on Kalman filtering and the MeshFlow method.We compare these methods in two ways and implement these methods on the same computer.First,we count the average consumption time per frame of different methods.At the same time,we count the delay time and the number of delay frames in processing the video.Second,we calculate the video distortion rate and cropping rate of different methods.These two rates are derived from the inter-frame affine transformation of videos that are processed by these methods.In addition,we invite 50 non-professional volunteers to make subjective judgments about the stability of these methods.The eventual experimental results show that,different from these offline methods that need to get all or most of the video frames,our algorithm can obtain the stable video with only one frame latency.This result is similar to the Kalman filtering method.Compared with the MeshFlow method,there are about 15％ speed improvements.In terms of distortion,this algorithm is only a little worse than the Kalman filtering method,better than any other methods.Similarly,our algorithm is only a little worse than the BCP method in cropping.But it is better than the others.Generally,our algorithm can produce comparable quality with these offline approaches that are recognized generally as the best ones of all the algorithms in stability.This result is similar to the MeshFlow method and almost completely better than the Kalman filtering method.Conclusion A video stabilization method based on the on-the-fly total variation optimization is proposed,which can get the stable videos with only one frame latency.The results show that the proposed algorithm can generate pleasing results both in delay performance and stability.Compared with the traditional methods,our algorithm is more suitable for the scenario that needs to get the stabilization results with low latency.However,our algorithm estimate inter-frame motion by detecting and matching feature points,it may fail to obtain good stabilization results for those videos that have minor feature points.