Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

Chenfei Wu,Shengming Yin,Weizhen Qi,Xiaodong Wang,Zecheng Tang,Nan Duan

2023-03-08

Abstract:ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called \textbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{<a class="link-external link-https" href="https://github.com/microsoft/visual-chatgpt" rel="external noopener nofollow">this https URL</a>}.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to combine natural language processing with computer vision in order to implement a dialogue system capable of understanding and generating images. Specifically, although existing large - scale language models such as ChatGPT perform excellently in text processing and multi - round dialogue, they are unable to process or generate images. Meanwhile, although Visual Foundation Models (VFMs) perform excellently in image understanding and generation, they can usually only handle specific tasks, and their input - output formats are fixed and lack flexibility.

To overcome these limitations, the paper proposes the **Visual ChatGPT** system. This system enables users to interact with ChatGPT not only in text but also in images by integrating different VFMs. The main objectives of Visual ChatGPT include:

1. **Multimodal Interaction**: Users can send and receive not only language but also images.
2. **Complex Visual Tasks**: Users can provide complex visual questions or editing instructions, and these tasks require the collaboration of multiple AI models and multi - step processing.
3. **Feedback and Correction**: Users can provide feedback and request correction of results.

To achieve these goals, the paper designs a **Prompt Manager**, which is responsible for the following functions:

- **Clearly inform the capabilities and input - output formats of each VFM**.
- **Convert different visual information into a language format** so that ChatGPT can understand.
- **Manage the history, priority, and conflicts of different VFMs**.

Through these mechanisms, Visual ChatGPT can call different VFMs in multi - round dialogues and gradually complete users' complex visual tasks. Experimental results show that Visual ChatGPT successfully combines the dialogue capabilities of ChatGPT and the visual processing capabilities of VFMs, providing new possibilities for multimodal interaction.