Abstract:Due to the sudden movement during the camera shoot, the videos retrieved from the hand-held mobile devices often suffer from undesired frame jitters, leading to the loss of video quality. In this paper, we present a video stabilization solution in mobile devices via inertial-visual state tracking. Specifically, during the video shoot, we use the gyroscope to estimate the rotation of camera, and use the structure-from-motion among the image frames to estimate the translation of camera. We build a camera projection model by considering the rotation and translation of the camera, and the camera motion model to depict the relationship between the inertial-visual state and the camera's 3D motion. By fusing the inertial measurement (IMU)-based method and the computer vision (CV)-based method, our solution is robust to the fast movement and violent jitters, moreover, it greatly reduces the computation overhead in video stabilization. In comparison to the IMU-based solution, our solution can estimate the translation in a more accurate manner, since we use the feature point pairs in adjacent image frames, rather than the error-prone accelerometers, to estimate the translation. In comparison to the CV-based solution, our solution can estimate the translation with less number of feature point pairs, since the number of undetermined degrees of freedom in the 3D motion directly reduces from 6 to 3. We implemented a prototype system on smart glasses and smart phones, and evaluated the performance under real scenarios, i.e., the human subjects used mobile devices to shoot videos while they were walking, climbing or riding. The experiment results show that our solution achieves 32 percent better performance than the state-of-art solutions in regard to video stabilization. Moreover, the average processing time latency is 32.6ms, which is lower than the conventional inter-frame time interval, i.e., 33ms, and thus meets the real-ti-e requirement for online processing.

Inertial Sensors Aided Image Alignment and Stitching for Panorama on Mobile Phones

Gyro-aided Camera-Odometer Online Calibration and Localization

INSPORAMA: INS-Aided Misalignment Correction in Feature-Based Panoramic Image Stitching.

An Indoor Positioning Framework Based on Panoramic Visual Odometry for Visually Impaired People

Design of an Enhanced Visual Odometry by Building and Matching Compressive Panoramic Landmarks Online

Tightly-Coupled Aided Inertial Navigation with Point and Plane Features

Stereo Calibration and Rectification for Omnidirectional Multi-Camera Systems

Unconstrained Self-Calibration of Stereo Camera on Visually Impaired Assistance Devices.

Automatic Sequential Stitching of High-Resolution Panorama for Android Devices Using Precapture Feature Detection and the Orientation Sensor

A Visual-Inertial Fusion Based Tracking System For Mobile Augmented Reality

Inertia Sensor Aided Alignment for Burst Pipeline in Low Light Conditions

Minimal Solutions for Panoramic Stitching Given Gravity Prior

Inavigation: an Image Based Indoor Navigation System

A Monocular Odometer for a Quadrotor Using a Homography Model and Inertial Cues

Video Stabilization for Camera Shoot in Mobile Devices via Inertial-Visual State Tracking

Visual Gyroscope: Combination of Deep Learning Features and Direct Alignment for Panoramic Stabilization

Efficient relative camera orientation detection for mobile applications

Nonlinear Complementary Filter for Attitude Estimation by Fusing Inertial Sensors and a Camera

Research on panoramie vision system of mobile robot

Improvement of Image Stitching Using Binocular Camera Calibration Model

Panoramic Scanned Page Using Mobile Phone Camera