Abstract:The International Journal of Robotics Research, Ahead of Print. In this paper, we propose a real-world benchmark for studying robotic learning in the context of functional manipulation: a robot needs to accomplish complex long-horizon behaviors by composing individual manipulation skills in functionally relevant ways. The core design principles of our Functional Manipulation Benchmark (FMB) emphasize a harmonious balance between complexity and accessibility. Tasks are deliberately scoped to be narrow, ensuring that models and datasets of manageable scale can be utilized effectively to track progress. Simultaneously, they are diverse enough to pose a significant generalization challenge. Furthermore, the benchmark is designed to be easily replicable, encompassing all essential hardware and software components. To achieve this goal, FMB consists of a variety of 3D-printed objects designed for easy and accurate replication by other researchers. The objects are procedurally generated, providing a principled framework to study generalization in a controlled fashion. We focus on fundamental manipulation skills, including grasping, repositioning, and a range of assembly behaviors. The FMB can be used to evaluate methods for acquiring individual skills, as well as methods for effectively combining and ordering such skills in order to solve complex, multi-stage manipulation tasks. We also offer an imitation learning framework that includes a suite of policies trained to solve the proposed tasks. This enables researchers to utilize our tasks as a versatile toolkit for examining various parts of the pipeline. For example, researchers could propose a better design for a grasping controller and evaluate it in combination with our baseline reorientation and assembly policies as part of a pipeline for solving multi-stage tasks. Our dataset, object CAD files, code, and evaluation videos can be found on our project website: https://functional-manipulation-benchmark.github.io.

Benchmarking Simulated Robotic Manipulation through a Real World Dataset

RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarios

Quantifying the Reality Gap in Robotic Manipulation Tasks

Benchmarking Robot Manipulation with the Rubik's Cube

PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks

BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots

THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation

BulletArm: An Open-Source Robotic Manipulation Benchmark and Learning Framework

RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control

FMB: A functional manipulation benchmark for generalizable robotic learning

Benchmarking Reinforcement Learning Algorithms on Real-World Robots

Benchmarking the Sim-to-Real Gap in Cloth Manipulation

ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks

Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulation

Evaluating Real-World Robot Manipulation Policies in Simulation

SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation

ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills

LaNMP: A Language-Conditioned Mobile Manipulation Benchmark for Autonomous Robots

FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation

Benchmarking Reinforcement Learning Methods for Dexterous Robotic Manipulation with a Three-Fingered Gripper