Abstract:Event cameras record visual information as asynchronous pixel change streams, excelling at scene perception under unsatisfactory lighting or high-dynamic conditions. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, to the best of our knowledge, marking a pioneering attempt to integrate large language models (LLMs) with event stream comprehension. To mitigate the huge domain gaps, we develop a three-stage optimization paradigm to gradually equip a pre-trained LLM with the capability of understanding event-based scenes. Our EventGPT comprises an event encoder, followed by a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, RGB image-text pairs generated by GPT are leveraged to warm up the linear projector, referring to LLaVA, as the gap between natural image and language modalities is relatively smaller. Secondly, we construct a synthetic yet large dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, Event-Chat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the insufficient ability of existing large - scale multimodal language models (MLLMs) in processing event - stream data. Specifically, existing MLLMs mainly focus on natural RGB images and have poor understanding and application effects on data captured by event cameras (such as asynchronous pixel - change streams). This is especially obvious in low - light conditions or high - dynamic scenes, which are precisely the advantages of event cameras. Therefore, the paper proposes EventGPT, which is the first multimodal large - language model specifically designed for event - stream understanding, aiming to fill this gap and improve the performance of the model in these specific scenarios. ### Main contributions of the paper: 1. **Proposing EventGPT**: This is the first multimodal large - language model specifically designed for event - stream understanding and can perform excellently in scene summarization, reasoning, and question - answering tasks. 2. **Three - stage training paradigm**: In order to gradually bridge the large domain gap between event data and language, the paper proposes a three - stage optimization method to gradually endow the pre - trained language model with the ability to understand event scenes. 3. **Constructing large - scale event - text datasets**: The paper constructs two large - scale event - text datasets, N - ImageNet - Chat and Event - Chat, which contain 1 million synthetic samples and 120,000 instruction samples respectively, for pre - training and fine - tuning the model. ### Three - stage training paradigm: 1. **Vision - language alignment**: Use LLaVA's image - text pairs to warm up the linear projector and establish an initial scene - language association. 2. **Event - language alignment**: Construct a large - scale synthetic dataset N - ImageNet - Chat, train the spatio - temporal aggregator and event - language adapter, and align event features with the language space. 3. **Instruction fine - tuning**: Use the self - collected high - quality dataset Event - Chat for instruction fine - tuning to enhance the generalization ability of the model in complex real - scene scenarios. ### Experimental results: - **Quantitative evaluation**: The experimental results on the N - ImageNet - Chat and Event - Chat datasets show that EventGPT outperforms the existing state - of - the - art MLLMs in three core metrics: detailed description, complex reasoning, and visual question - answering. - **Qualitative analysis**: The qualitative evaluation in extreme lighting conditions and high - speed object - dynamic scenes shows that EventGPT can identify and understand scenes more accurately, especially outstanding in high - dynamic - range and high - speed - motion scenes. ### Conclusion: Through the three - stage training paradigm and the construction of large - scale event - text datasets, EventGPT has successfully improved the performance of multimodal language models in event - stream data processing, providing a solid foundation for future research.

EventGPT: Event Stream Understanding with Multimodal Large Language Models

Can Large Language Models Grasp Event Signals? Exploring Pure Zero-Shot Event-based Recognition

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

Towards Event-oriented Long Video Understanding

MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens

VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework

GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation

VideoLLM: Modeling Video Sequence with Large Language Models

A Survey on Multimodal Large Language Models

PerceptionGPT: Effectively Fusing Visual Perception into LLM

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain

InfMLLM: A Unified Framework for Visual-Language Tasks.

SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding

What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

LLMGA: Multimodal Large Language Model based Generation Assistant

StoryGPT-V: Large Language Models as Consistent Story Visualizers

GPT4Graph: Can Large Language Models Understand Graph Structured Data ? an Empirical Evaluation and Benchmarking.

GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding

A Comprehensive Survey of Multimodal Large Language Models: Concept, Application and Safety