Abstract:This paper proposes a new multi-view human body tracking and model-based rendering system with a textured deformable human body model and explores its practical application to view transfer in image-based rendering (IBR). The proposed approach first reconstructs an initial 3-D model of the human subject offline using Kinect depth cameras or from a general 3-D model if such information is unavailable in the original data. The human pose of the subject is then tracked with multi-view videos using an annealed particle filter (APF)-based tracker with a new color-based likelihood function and a Bayesian-Kalman filter smoother. The previous captured model can then be deformed to the new position for model-based rendering. An immediate application of the proposed approach is to support fly-over effects for view transfer in IBR systems with limited cameras. It avoids the reconstruction of the complete dynamic 3-D model where a large number of cameras may be required. Moreover, for static background, the background can be rendered using IBR with precaptured depth maps to further enhance the user's experience. To reduce the artifacts during fly-over caused by tracking errors and model deformation, a novel morphing technique utilizing a new free-form deformation-based artifacts suppression (FFD-AS) method and other user interface design techniques are also proposed. It allows smooth transition between the original view and the model-rendered views. The performance of the proposed algorithm is evaluated using the publicly available HumanEva dataset and our captured RGB-D multi-view dataset. Experimental results show that the proposed APF-based tracker offered improved tracking performance compared with the conventional bidirectional silhouette likelihood criterion. The proposed morphing approach is also shown to be effective in mitigating the rendering artifacts during view transfer.

Multi-View Articulated Human Body Tracking With Textured Deformable Mesh Model

Human Pose Tracking Algorithm Based on Skeleton-Texture Model

Multi-view Human Motion Capture with an Improved Deformation Skin Model

A New Model-Based Method for Multi-View Human Body Tracking and Its Application to View Transfer in Image-Based Rendering.

A New Multi-View Articulated Human Motion Tracking Algorithm with Improved Silhouette Extraction and View Adaptive Fusion

Dynamic Human Body Reconstruction and Motion Tracking with Low-Cost Depth Cameras

Reconstruction of 3D Human Motion Using Multiple View L1 Trackers

A Robust Multi‐View System for High‐Fidelity Human Body Shape Reconstruction

New multi-view human motion capture framework

Dynamic High Resolution Deformable Articulated Tracking

Long-Range Motion Trajectories Extraction of Articulated Human Using Mesh Evolution

Human model adaptation for multiview markerless motion capture

Markerless 3D Human Motion Tracking for Monocular Video Sequences

MODEL-BASEDMARKERLESS HUMAN BODY MOTION CAPTURE USINGMULTIPLE CAMERAS

3D Human Body Shape and Motion Tracking by LBS and Snake

Optimization and Soft Constraints for Human Shape and Pose Estimation Based on a 3D Morphable Model

Track Articulated Body Motion by Pose-Model Based Stochastic Search

Geometrical kinematic modeling on human motion using method of multi-sensor fusion

Model-Based Markerless Human Body Motion Capture using Multiple Cameras

Tracking Articulated Body by Dynamic Markov Network

Model-based markerless human body motion capture using active contour