Abstract:Internal computational models of physical bodies are fundamental to the ability of robots and animals alike to plan and control their actions. These "self-models" allow robots to consider outcomes of multiple possible future actions, without trying them out in physical reality. Recent progress in fully data-driven self-modeling has enabled machines to learn their own forward kinematics directly from task-agnostic interaction data. However, forward-kinema\-tics models can only predict limited aspects of the morphology, such as the position of end effectors or velocity of joints and masses. A key challenge is to model the entire morphology and kinematics, without prior knowledge of what aspects of the morphology will be relevant to future tasks. Here, we propose that instead of directly modeling forward-kinematics, a more useful form of self-modeling is one that could answer space occupancy queries, conditioned on the robot's state. Such query-driven self models are continuous in the spatial domain, memory efficient, fully differentiable and kinematic aware. In physical experiments, we demonstrate how a visual self-model is accurate to about one percent of the workspace, enabling the robot to perform various motion planning and control tasks. Visual self-modeling can also allow the robot to detect, localize and recover from real-world damage, leading to improved machine resiliency. Our project website is at: <a class="link-external link-https" href="https://robot-morphology.cs.columbia.edu/" rel="external noopener nofollow">this https URL</a>

Learning an internal representation of the end-effector configuration space

A LiDAR Based End to End Controller for Robot Navigation Using Deep Neural Network

Autonomous Learning of the Robot Kinematic Model

Self-developing Proprioception-Based Robot Internal Models

High-Degrees-of-Freedom Dynamic Neural Fields for Robot Self-Modeling and Motion Planning

A Learning-Based Approach for Estimating Inertial Properties of Unknown Objects from Encoder Discrepancies

Towards Orientation Learning and Adaptation in Cartesian Space

Learning State Conditioned Linear Mappings for Low-Dimensional Control of Robotic Manipulators

Learning Kinematic Feasibility for Mobile Manipulation through Deep Reinforcement Learning

Augmented Neural Network for Full Robot Kinematic Modelling in SE(3)

Efficient Intrinsically Motivated Robotic Grasping with Learning-Adaptive Imagination in Latent Space

Robot Body Schema Learning from Full-body Extero/Proprioception Sensors

Learning Articulated Motions From Visual Demonstration

The homunculus for proprioception: Toward learning the representation of a humanoid robot's joint space using self-organizing maps

Robot Arm Reaching Based on Inner Rehearsal

Robot Arm Configuration Self-Awareness Via Interacting with Environment

Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

An Experimental Autonomous Articulated Robot that can Learn

Learning Low-Level Causal Relations using a Simulated Robotic Arm

Learning Efficient Robot Arm Reaching

Full-Body Visual Self-Modeling of Robot Morphologies