Abstract:Unlike quasi-static robotic manipulation tasks like pick-and-place, dynamic tasks such as non-prehensile manipulation pose greater challenges, especially for vision-based control. Successful control requires the extraction of features relevant to the target task. In visual imitation learning settings, these features can be learnt by backpropagating the policy loss through the vision backbone. Yet, this approach tends to learn task-specific features with limited generalizability. Alternatively, learning world models can realize more generalizable vision backbones. Utilizing the learnt features, task-specific policies are subsequently trained. Commonly, these models are trained solely to predict the next RGB state from the current state and action taken. But only-RGB prediction might not fully-capture the task-relevant dynamics. In this work, we hypothesize that direct supervision of target dynamic states (Dynamics Mapping) can learn better dynamics-informed world models. Beside the next RGB reconstruction, the world model is also trained to directly predict position, velocity, and acceleration of environment rigid bodies. To verify our hypothesis, we designed a non-prehensile 2D environment tailored to two tasks: "Balance-Reaching" and "Bin-Dropping". When trained on the first task, dynamics mapping enhanced the task performance under different training configurations (Decoupled, Joint, End-to-End) and policy architectures (Feedforward, Recurrent). Notably, its most significant impact was for world model pretraining boosting the success rate from 21% to 85%. Although frozen dynamics-informed world models could generalize well to a task with in-domain dynamics, but poorly to a one with out-of-domain dynamics.

A Dynamics Observation Model for Moving Object Based on Reinforcement Learning: Experimental Demonstration on Stereo Vision

Stereo Visual Odometry from Disparity Space in Dynamic Environments

Continuous-Time Stereo Visual Odometry Based on Dynamics Model

On Observability of a Relative Attitude Determination System Based on Stereo Vision

Image Gradient-based Joint Direct Visual Odometry for Stereo Camera

Unsupervised Dynamics Prediction with Object-Centric Kinematics

Stereo SLAM in Dynamic Environments Using Semantic Segmentation

DynPL-SVO: A New Method Using Point and Line Features for Stereo Visual Odometry in Dynamic Scenes

DyStSLAM: An Efficient Stereo Vision SLAM System in Dynamic Environment

An Active Stereo Vision-Based Learning Approach for Robotic Tracking, Fixating and Grasping Control

D&D: Learning Human Dynamics from Dynamic Camera

Pose estimation in industrial machine vision systems under sensing dynamics: A statistical learning approach

DynPL-SVO: A Robust Stereo Visual Odometry for Dynamic Scenes

A Robust Multi-Object Tracking SLAM System in Dynamic Scenes for Stereo and RGB-D Cameras

Stereo Visual Odometry for Object Segmentation Based on Sparse Optical Flow in Dynamic Scene

A Stereo Vision SLAM with Moving Vehicles Tracking in Outdoor Environment

Learning Physical Dynamics for Object-centric Visual Prediction

Dynamical Textures Modeling Via Joint Video Dictionary Learning

Robust Stereo Visual Odometry Based on Probabilistic Decoupling Ego-Motion Estimation and 3D SSC

Dynamics Learning with Object-Centric Interaction Networks for Robot Manipulation

Visual Imitation Learning of Non-Prehensile Manipulation Tasks with Dynamics-Supervised Models