Abstract:Imitation learning with human data has demonstrated remarkable success in teaching robots in a wide range of skills. However, the inherent diversity in human behavior leads to the emergence of multi-modal data distributions, thereby presenting a formidable challenge for existing imitation learning algorithms. Quantifying a model's capacity to capture and replicate this diversity effectively is still an open problem. In this work, we introduce simulation benchmark environments and the corresponding Datasets with Diverse human Demonstrations for Imitation Learning (D3IL), designed explicitly to evaluate a model's ability to learn multi-modal behavior. Our environments are designed to involve multiple sub-tasks that need to be solved, consider manipulation of multiple objects which increases the diversity of the behavior and can only be solved by policies that rely on closed loop sensory feedback. Other available datasets are missing at least one of these challenging properties. To address the challenge of diversity quantification, we introduce tractable metrics that provide valuable insights into a model's ability to acquire and reproduce diverse behaviors. These metrics offer a practical means to assess the robustness and versatility of imitation learning algorithms. Furthermore, we conduct a thorough evaluation of state-of-the-art methods on the proposed task suite. This evaluation serves as a benchmark for assessing their capability to learn diverse behaviors. Our findings shed light on the effectiveness of these methods in tackling the intricate problem of capturing and generalizing multi-modal human behaviors, offering a valuable reference for the design of future imitation learning algorithms.

Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

Human Demonstration Trajectory Refinement for Redundant Manipulators.

Learning Skills from Demonstrations: A Trend from Motion Primitives to Experience Abstraction

BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmark

RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control

Learning Generalizable Robot Skills from Demonstrations in Cluttered Environments

Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations

Learning from Successful and Failed Demonstrations via Optimization

How Can Everyday Users Efficiently Teach Robots by Demonstrations?

Digital Robot Judge: Building a Task-centric Performance Database of Real-World Manipulation With Electronic Task Boards

Comparing Skill Transfer Between Full Demonstrations and Segmented Sub-Tasks for Neural Dynamic Motion Primitives

Demonstration Learning and Generalization of Robotic Motor Skills Based on Wearable Motion Tracking Sensors

Comparative Analysis of Programming by Demonstration Methods: Kinesthetic Teaching vs Human Demonstration

Learning Multi-Step Manipulation Tasks from A Single Human Demonstration

Benchmarking Simulated Robotic Manipulation through a Real World Dataset

Quantifying Teaching Behaviour in Robot Learning from Demonstration

Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrations

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

A Survey of Demonstration Learning

Versatile Demonstration Interface: Toward More Flexible Robot Demonstration Collection

HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation