Abstract:This paper presents the development and evaluation of a Large Language Model (LLM), also known as foundation models, based multi-agent system framework for complex event processing (CEP) with a focus on video query processing use cases. The primary goal is to create a proof-of-concept (POC) that integrates state-of-the-art LLM orchestration frameworks with publish/subscribe (pub/sub) tools to address the integration of LLMs with current CEP systems. Utilizing the Autogen framework in conjunction with Kafka message brokers, the system demonstrates an autonomous CEP pipeline capable of handling complex workflows. Extensive experiments evaluate the system's performance across varying configurations, complexities, and video resolutions, revealing the trade-offs between functionality and latency. The results show that while higher agent count and video complexities increase latency, the system maintains high consistency in narrative coherence. This research builds upon and contributes to, existing novel approaches to distributed AI systems, offering detailed insights into integrating such systems into existing infrastructures.

What problem does this paper attempt to address?

The problems that this paper attempts to solve are: How to use large - language models (LLMs) to enhance existing complex event processing (CEP) systems, especially for multimedia Internet of Things (IoT) application scenarios such as video query processing. Specifically, the research aims to: 1. **Create a proof - of - concept (POC) framework**: This framework will integrate the state - of - the - art LLM orchestration framework with publish/subscribe (pub/sub) tools to address the challenges of current CEP systems in integrating with dynamic multi - source data. 2. **Explore the application potential of LLM in CEP**: By introducing an LLM - based multi - agent system (MAS), improve the flexibility and adaptability of the CEP system, enabling it to better handle complex, dynamically changing workflows. 3. **Evaluate system performance**: Through extensive experiments, evaluate the performance of the system under different configurations, complexity levels, and video resolutions, and analyze the trade - off between functionality and latency. ### Key Research Questions The paper revolves around two main research questions: - **RQ1**: What is the current research status of single - agent or multi - agent systems based on LLM? Through a comprehensive review of existing literature, clarify the application areas and development trends of LLM agents. - **RQ2**: How can large - language models (LLMs) be combined with multi - agent systems to enhance the complex event processing (CEP) pipeline and meet the quality - of - service (QoS) indicators of a given use case? Especially when integrating with existing pub/sub infrastructures (such as Kafka), how can a seamless transition and efficient operation be ensured? ### Main Contributions 1. **Comprehensive survey and gap analysis**: Conducted a detailed literature review and future trend analysis of LLM - based autonomous agent applications. 2. **Proof - of - concept system**: Designed and implemented a CEP pipeline based on LLM - MAS, which can be tested, evaluated, and extended. 3. **Summary of deployment experiences**: Discussed the experiences and lessons in deploying multi - agent systems in CEP use cases. ### Technical Background The technical background involved in the paper includes distributed AI systems, the computing continuum, the publish/subscribe paradigm, large - language models (LLMs), and the Autogen framework. These technologies jointly support the development and evaluation of the multi - agent system proposed in the paper. Through these studies, the paper provides new insights into the development of distributed AI systems and shows how to use LLM to enhance the functionality and performance of existing CEP systems.

Large Language Model Based Multi-Agent System Augmented Complex Event Processing Pipeline for Internet of Multimedia Things

Large Language Model based Multi-Agents: A Survey of Progress and Challenges

OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquer

Self-Adaptive Large Language Model (LLM)-Based Multiagent Systems

Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

LLM Harmony: Multi-Agent Communication for Problem Solving

MindAgent: Emergent Gaming Interaction

Synergistic Simulations: Multi-Agent Problem Solving with Large Language Models

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

Large Language Model-Driven Cross-Domain Orchestration Using Multi-Agent Workflow

Building Cooperative Embodied Agents Modularly with Large Language Models

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

Exploring Autonomous Agents through the Lens of Large Language Models: A Review

Large Multimodal Agents: A Survey

Navigating Complexity: Orchestrated Problem Solving with Multi-Agent LLMs

MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents

Large Language Model Evaluation Via Multi AI Agents: Preliminary results

LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation

Chain of Agents: Large Language Models Collaborating on Long-Context Tasks

VideoAgent: Long-form Video Understanding with Large Language Model as Agent

TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage