Abstract:We would like industrial robots to handle unstructured environments with cameras and perception pipelines. In contrast to traditional industrial robots that replay offline-crafted trajectories, online behavior planning is required for these perception-guided industrial applications. Aside from perception and planning algorithms, deploying perception-guided manipulators also requires substantial effort in integration. One approach is writing scripts in a traditional language (such as Python) to construct the planning problem and perform integration with other algorithmic modules & external devices. While scripting in Python is feasible for a handful of robots and applications, deploying perception-guided manipulation at scale (e.g., more than 10000 robot workstations in over 2000 customer sites) becomes intractable. To resolve this challenge, we propose a Domain-Specific Language (DSL) for perception-guided manipulation applications. To scale up the deployment,our DSL provides: 1) an easily accessible interface to construct & solve a sub-class of Task and Motion Planning (TAMP) problems that are important in practical applications; and 2) a mechanism to implement flexible control flow to perform integration and address customized requirements of distinct industrial application. Combined with an intuitive graphical programming frontend, our DSL is mainly used by machine operators without coding experience in traditional programming languages. Within hours of training, operators are capable of orchestrating interesting sophisticated manipulation behaviors with our DSL. Extensive practical deployments demonstrate the efficacy of our method.

What problem does this paper attempt to address?

The main problem this paper attempts to address is: how to reduce integration workload and improve deployment efficiency when deploying perception-guided industrial robotic arms on a large scale. Specifically, traditional industrial robots typically execute fixed trajectories through offline programming, but online behavior planning is required when dealing with unstructured environments. This involves the integration of perception, task, and motion planning (TAMP) algorithms, as well as coordination with other modules and external devices. However, traditional scripting methods (such as Python scripts) become impractical for large-scale deployment because the specific needs of each application differ, making the code difficult to reuse and requiring close collaboration between programmers and on-site application experts, which is time-consuming and costly. To address this challenge, the authors propose a domain-specific language (DSL) designed to provide a user-friendly interface for machine operators without programming experience to construct and solve a class of important task and motion planning problems. This DSL has the following features: 1. **Easy-to-use graphical programming frontend**: Users can construct control flow diagrams through drag-and-drop operations without writing traditional programming language code. 2. **Flexible control flow mechanism**: It can implement branching and looping structures to meet the customized needs of different industrial applications. 3. **Seamless integration with TAMP planners**: Users only need to define the "skeleton" of behaviors, which includes some undetermined parameters, and the planner will generate these parameters at runtime. 4. **Pre-planning mechanism**: By interweaving robot motion with planning through a pre-planning mechanism, the throughput of the robotic arm is significantly improved. Through these designs, this DSL can simplify the process of large-scale deployment of perception-guided industrial robotic arms, reduce integration costs, and improve the system's scalability and flexibility.

KiloBot: A Programming Language for Deploying Perception-Guided Industrial Manipulators at Scale

On reverse-engineering the KUKA Robot Language

Towards Robot-independent Manipulation Behavior Description

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

Improving the lifecycle of robotics components using Domain-Specific Languages

No-code robotic programming for agile production: A new markerless-approach for multimodal natural interaction in a human-robot collaboration context

A Robotic Manipulation Framework for Industrial Human–robot Collaboration Based on Continual Knowledge Graph Embedding

Creative Robot Tool Use with Large Language Models

Grounding Language Models in Autonomous Loco-manipulation Tasks

Programmatically Grounded, Compositionally Generalizable Robotic Manipulation

RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation

Programming Manipulators by Instructions

Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming

Time is on my sight: scene graph filtering for dynamic environment perception in an LLM-driven robot

Non-Prehensile Tool-Object Manipulation by Integrating LLM-Based Planning and Manoeuvrability-Driven Controls

One to rule them all: natural language to bind communication, perception and action

Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

Towards Python-based Domain-specific Languages for Self-reconfigurable Modular Robotics Research

Language, Camera, Autonomy! Prompt-engineered Robot Control for Rapidly Evolving Deployment

Versatile multicontact planning and control for legged loco-manipulation