ChatSim: Underwater Simulation with Natural Language Prompting

Aadi Palnitkar,Rashmi Kapu,Xiaomin Lin,Cheng Liu,Nare Karapetyan,Yiannis Aloimonos

2023-08-09

Abstract:Robots are becoming an essential part of many operations including marine exploration or environmental monitoring. However, the underwater environment presents many challenges, including high pressure, limited visibility, and harsh conditions that can damage equipment. Real-world experimentation can be expensive and difficult to execute. Therefore, it is essential to simulate the performance of underwater robots in comparable environments to ensure their optimal functionality within practical real-world contexts.OysterSim generates photo-realistic images and segmentation masks of objects in marine environments, providing valuable training data for underwater computer vision applications. By integrating ChatGPT into underwater simulations, users can convey their thoughts effortlessly and intuitively create desired underwater environments without intricate coding. \invis{Moreover, researchers can realize substantial time and cost savings by evaluating their algorithms across diverse underwater conditions in the simulation.} The objective of ChatSim is to integrate Large Language Models (LLM) with a simulation environment~(OysterSim), enabling direct control of the simulated environment via natural language input. This advancement can greatly enhance the capabilities of underwater simulation, with far-reaching benefits for marine exploration and broader scientific research endeavors.

Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How to integrate large - language models (LLMs) with underwater simulation environments so that users can easily create and modify underwater scenes using natural language without writing code or having complex programming knowledge. Specifically, the paper introduces a platform named ChatSim, which combines the OysterSim underwater simulation environment and large - language models such as ChatGPT, allowing users to directly control the behavior of objects and robots in the simulation environment through natural - language input.

### Main Objectives
- **Enhance User Experience**: By introducing natural - language - processing capabilities, non - professional users can also conveniently operate complex underwater simulation environments.
- **Reduce Programming Requirements**: Users no longer need to write complex code to create or modify underwater scenes; they can achieve this simply by describing in natural language.
- **Improve Research Efficiency**: Provide a more efficient and intuitive tool for ocean exploration and broader scientific research.

### Solutions
The ChatSim platform achieves the above objectives in the following ways:
1. **Function Library Design**: Define a series of functions that can be called by natural - language instructions to operate the simulation environment. For example, `setbotposition(points)` is used to set the position of the robot, and `putobject(name, (x, y, z), (yaw, pitch, roll))` is used to place an object at a specified position.
2. **System Prompt**: Before each natural - language interaction, use a system prompt to guide how the LLM responds to user instructions and restrict it to only call functions in the predefined function library.
3. **Execution Pipeline**: The entire execution process includes importing necessary libraries and models, adding underwater elements, generating Python scripts according to user instructions and executing them, and finally reflecting the results in the simulation environment.

### Experimental Verification
The paper demonstrates the effectiveness of ChatSim through multiple experiments:
- **Experiment 1**: Users move the robot from its initial position to a specified position through natural - language instructions.
- **Experiment 2**: Users can add new objects to the simulation environment, for example, place multiple oysters at specific positions.
- **Experiment 3**: Users can delete objects within a specified area.
- **Experiment 4**: Users can control the robot to move along a circular trajectory and take pictures regularly.

### Conclusions and Future Work
- **Conclusions**: ChatSim successfully combines natural - language - processing capabilities with underwater simulation environments, providing an efficient simulation tool without coding, which is suitable for ocean exploration and scientific research.
- **Future Work**: Further improve the authenticity and accuracy of the simulation, for example, integrate more real - data sets, introduce more sensors (such as multi - beam sonar), and optimize the user interface and interaction experience.

Through these methods, ChatSim not only simplifies the creation and modification process of underwater simulations but also provides researchers with a more powerful tool to support ocean exploration and scientific research in related fields.