AI Assistants for Spaceflight Procedures: Combining Generative Pre-Trained Transformer and Retrieval-Augmented Generation on Knowledge Graphs With Augmented Reality Cues

Oliver Bensch,Leonie Bensch,Tommy Nilsson,Florian Saling,Bernd Bewer,Sophie Jentzsch,Tobias Hecking,J. Nathan Kutz

2024-09-22

Abstract:This paper describes the capabilities and potential of the intelligent personal assistant (IPA) CORE (Checklist Organizer for Research and Exploration), designed to support astronauts during procedures onboard the International Space Station (ISS), the Lunar Gateway station, and beyond. We reflect on the importance of a reliable and flexible assistant capable of offline operation and highlight the usefulness of audiovisual interaction using augmented reality elements to intuitively display checklist information. We argue that current approaches to the design of IPAs in space operations fall short of meeting these criteria. Therefore, we propose CORE as an assistant that combines Knowledge Graphs (KGs), Retrieval-Augmented Generation (RAG) for a Generative Pre-Trained Transformer (GPT), and Augmented Reality (AR) elements to ensure an intuitive understanding of procedure steps, reliability, offline availability, and flexibility in terms of response style and procedure updates.

Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to provide astronauts with a reliable, flexible, and offline - available intelligent personal assistant (IPA) in future space missions to support their operational processes in the International Space Station (ISS), the Lunar Gateway, and further missions. Specifically, the paper points out that current IPAs have the following deficiencies:

1. **Reliability**: Existing IPAs may generate misleading information or errors, especially when the generated information is beyond the scope of their training data. This may lead to serious consequences in space missions.

2. **Flexibility**: Most existing IPAs are rule - based and can only respond to predefined sets of questions, lacking the ability to adapt to new situations and update procedures. In addition, different astronauts have different interaction preferences, and existing IPAs are difficult to meet these diverse needs.

3. **Offline availability**: Due to the risk of communication delays and radio outages, future space missions require IPAs that can operate offline. However, many advanced AI models rely on online resources and cannot be used in an offline environment.

4. **Intuitive visual information display**: Existing IPAs mainly provide information in text or voice form, lacking intuitive three - dimensional visual aids, which increases the cognitive burden on astronauts.

To solve these problems, the paper proposes a new IPA named CORE (Checklist Organizer for Research and Exploration). CORE combines the following technologies:

- **Generative Pretrained Transformer (GPT)**: Used to generate natural - language responses.
- **Retrieval - Augmented Generation (RAG)**: Retrieves relevant information from the knowledge graph (KG) to enhance the accuracy and reliability of the generated content.
- **Augmented Reality (AR) elements**: Provides intuitive three - dimensional visual information through the head - up display (HUD) to help astronauts better understand the operation steps.

Through the combination of these technologies, CORE aims to improve the efficiency and safety of astronauts when performing complex tasks, while ensuring the reliability, flexibility, and offline availability of the system.