Abstract:Imitation learning has demonstrated significant potential in performing high-precision manipulation tasks using visual feedback from cameras. However, it is common practice in imitation learning for cameras to be fixed in place, resulting in issues like occlusion and limited field of view. Furthermore, cameras are often placed in broad, general locations, without an effective viewpoint specific to the robot's task. In this work, we investigate the utility of active vision (AV) for imitation learning and manipulation, in which, in addition to the manipulation policy, the robot learns an AV policy from human demonstrations to dynamically change the robot's camera viewpoint to obtain better information about its environment and the given task. We introduce AV-ALOHA, a new bimanual teleoperation robot system with AV, an extension of the ALOHA 2 robot system, incorporating an additional 7-DoF robot arm that only carries a stereo camera and is solely tasked with finding the best viewpoint. This camera streams stereo video to an operator wearing a virtual reality (VR) headset, allowing the operator to control the camera pose using head and body movements. The system provides an immersive teleoperation experience, with bimanual first-person control, enabling the operator to dynamically explore and search the scene and simultaneously interact with the environment. We conduct imitation learning experiments of our system both in real-world and in simulation, across a variety of tasks that emphasize viewpoint planning. Our results demonstrate the effectiveness of human-guided AV for imitation learning, showing significant improvements over fixed cameras in tasks with limited visibility. Project website: <a class="link-external link-https" href="https://soltanilara.github.io/av-aloha/" rel="external noopener nofollow">this https URL</a>

Active Perception and Representation for Robotic Manipulation

InterRep: A Visual Interaction Representation for Robotic Grasping

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation

Q-Attention: Enabling Efficient Learning for Vision-Based Robotic Manipulation

RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation

Vision-Based Categorical Object Pose Estimation and Manipulation.

Manipulator-Independent Representations for Visual Imitation

Active Exploration for Robotic Manipulation

Learning Robotic Manipulation through Visual Planning and Acting

Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance

Interactive Perception for Deformable Object Manipulation

Vision-Based Manipulators Need to Also See from Their Hands

Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition

AP-VLM: Active Perception Enabled by Vision-Language Models

Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations

Distributed Reinforcement Learning of Targeted Grasping with Active Vision for Mobile Manipulators

Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation

Vision-based reinforcement learning control of soft robot manipulators