Abstract:The International Journal of Robotics Research, Ahead of Print. In this paper, we propose a real-world benchmark for studying robotic learning in the context of functional manipulation: a robot needs to accomplish complex long-horizon behaviors by composing individual manipulation skills in functionally relevant ways. The core design principles of our Functional Manipulation Benchmark (FMB) emphasize a harmonious balance between complexity and accessibility. Tasks are deliberately scoped to be narrow, ensuring that models and datasets of manageable scale can be utilized effectively to track progress. Simultaneously, they are diverse enough to pose a significant generalization challenge. Furthermore, the benchmark is designed to be easily replicable, encompassing all essential hardware and software components. To achieve this goal, FMB consists of a variety of 3D-printed objects designed for easy and accurate replication by other researchers. The objects are procedurally generated, providing a principled framework to study generalization in a controlled fashion. We focus on fundamental manipulation skills, including grasping, repositioning, and a range of assembly behaviors. The FMB can be used to evaluate methods for acquiring individual skills, as well as methods for effectively combining and ordering such skills in order to solve complex, multi-stage manipulation tasks. We also offer an imitation learning framework that includes a suite of policies trained to solve the proposed tasks. This enables researchers to utilize our tasks as a versatile toolkit for examining various parts of the pipeline. For example, researchers could propose a better design for a grasping controller and evaluate it in combination with our baseline reorientation and assembly policies as part of a pipeline for solving multi-stage tasks. Our dataset, object CAD files, code, and evaluation videos can be found on our project website: https://functional-manipulation-benchmark.github.io.

Digital Robot Judge: Building a Task-centric Performance Database of Real-World Manipulation With Electronic Task Boards

Towards Remote Robotic Competitions: An Internet-Connected Task Board and Dashboard

Perception of Demonstration for Automatic Programing of Robotic Assembly: Framework, Algorithm, and Validation

Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

Benchmarking Simulated Robotic Manipulation through a Real World Dataset

BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robots

Benchmarking Robot Manipulation with the Rubik's Cube

Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulation

Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks

Demonstrating Mobile Manipulation in the Wild: A Metrics-Driven Approach

DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation

Quantifying the Reality Gap in Robotic Manipulation Tasks

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

FMB: A functional manipulation benchmark for generalizable robotic learning

Crowd against the machine: A simulation-based benchmark tool to evaluate and compare robot capabilities to navigate a human crowd

Assistive arm and hand manipulation: How does current research intersect with actual healthcare needs?

RoboCAS: A Benchmark for Robotic Manipulation in Complex Object Arrangement Scenarios

Impact of Tactile Sensor Quantities and Placements on Learning-based Dexterous Manipulation

PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks

RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control

On Hand-Held Grippers and the Morphological Gap in Human Manipulation Demonstration