Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design

Aoran Jiao,Tanmay P. Patel,Sanjmi Khurana,Anna-Mariya Korol,Lukas Brunke,Vivek K. Adajania,Utku Culha,Siqi Zhou,Angela P. Schoellig

2023-12-02

Abstract:This paper presents Swarm-GPT, a system that integrates large language models (LLMs) with safe swarm motion planning - offering an automated and novel approach to deployable drone swarm choreography. Swarm-GPT enables users to automatically generate synchronized drone performances through natural language instructions. With an emphasis on safety and creativity, Swarm-GPT addresses a critical gap in the field of drone choreography by integrating the creative power of generative models with the effectiveness and safety of model-based planning algorithms. This goal is achieved by prompting the LLM to generate a unique set of waypoints based on extracted audio data. A trajectory planner processes these waypoints to guarantee collision-free and feasible motion. Results can be viewed in simulation prior to execution and modified through dynamic re-prompting. Sim-to-real transfer experiments demonstrate Swarm-GPT's ability to accurately replicate simulated drone trajectories, with a mean sim-to-real root mean square error (RMSE) of 28.7 mm. To date, Swarm-GPT has been successfully showcased at three live events, exemplifying safe real-world deployment of pre-trained models.

Robotics

What problem does this paper attempt to address?

The paper aims to address the issues of automation and safety in drone swarm choreography, specifically:

- **Automated Drone Swarm Choreography**: By combining large language models (LLM) with safe motion planning algorithms, the system achieves automatic choreography of drone swarms based on natural language instructions. This addresses the labor-intensive problem of manually choreographing drone performances, especially as the number of drones increases, making choreography design and safety analysis difficult to manage.

- **Enhancing Interactivity and Safety**: The paper proposes a new method that allows non-expert users to generate complex drone behaviors through simple natural language input. These behaviors can be enhanced by built-in safety control and planning algorithms to ensure safety during actual deployment.

- **Seamless Integration of LLM and Safety Filters**: By seamlessly integrating LLM with model-based safety filters, the system ensures safe execution while preserving the creativity of LLM-generated actions.

- **Real-time Modifications**: Users can modify the generated trajectories in real-time through re-prompting, such as asking the drones to "fly faster" or "maintain a greater distance."

In summary, the main goal of this research is to develop an automated, intuitive, and safe method for the design and execution of drone swarm choreography, with a particular emphasis on balancing safety and creativity.