A Blueprint Architecture of Compound AI Systems for Enterprise

Eser Kandogan,Sajjadur Rahman,Nikita Bhutani,Dan Zhang,Rafael Li Chen,Kushan Mitra,Sairam Gurajada,Pouya Pezeshkpour,Hayate Iso,Yanlin Feng,Hannah Kim,Chen Shen,Jin Wang,Estevam Hruschka

2024-06-02

Abstract:Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goal, there is a notable shift to building compound AI systems, wherein LLMs are integrated into an expansive software infrastructure with many components like models, retrievers, databases and tools. In this paper, we introduce a blueprint architecture for compound AI systems to operate in enterprise settings cost-effectively and feasibly. Our proposed architecture aims for seamless integration with existing compute and data infrastructure, with ``stream'' serving as the key orchestration concept to coordinate data and instructions among agents and other components. Task and data planners, respectively, break down, map, and optimize tasks and data to available agents and data sources defined in respective registries, given production constraints such as accuracy and latency.

Databases,Artificial Intelligence

What problem does this paper attempt to address?

This paper mainly discusses how to build a Compound AI (Artificial Intelligence) system suitable for enterprise environments to address the challenges encountered when integrating large language models (LLMs) into production environments effectively. In existing methods, LLMs play a central role in task planning and data retrieval, but considerations such as latency, accuracy, and cost need to be taken into account during actual deployment. The paper proposes a blueprint architecture aimed at seamlessly integrating the Compound AI system with existing computational and data infrastructure in an economically viable manner. Key design factors include: 1. Ensuring smooth integration with existing infrastructure through appropriate touchpoints and interfaces. 2. Efficient coordination of internal and external components through proper resource allocation and workflow coordination. 3. Maximizing system utilization while reducing costs. The key components of this architecture include: 1. Agents: Computational entities that perform tasks and can interact with service APIs, LLMs, and other tools. 2. Agent and data registries: Store and organize metadata of deployed models, APIs, databases, and tools for integration purposes. 3. Streams: The primary concept for coordinating data and instructions across components. 4. Task and data schedulers: Optimize task execution and data retrieval based on cost and quality constraints. Event-driven orchestration is achieved through the concepts of streams and sessions, while task and data schedulers are used to optimize the execution of tasks and data operations based on production constraints. The paper believes that this blueprint architecture helps in developing reliable, efficient, and user-friendly artificial intelligence applications, and encourages interdisciplinary research.

A Blueprint Architecture of Compound AI Systems for Enterprise

SoK: A Systems Perspective on Compound AI Threats and Countermeasures

Towards Effective GenAI Multi-Agent Collaboration: Design and Evaluation for Enterprise Applications

Synergistic Integration of Large Language Models and Cognitive Architectures for Robust AI: An Exploratory Analysis

Building AI Agents for Autonomous Clouds: Challenges and Design Principles

Large Language Model-Driven Cross-Domain Orchestration Using Multi-Agent Workflow

Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

Enhancing AI Systems with Agentic Workflows Patterns in Large Language Model

APT: Architectural Planning and Text-to-Blueprint Construction Using Large Language Models for Open-World Agents

Incorporating Large Language Models into Production Systems for Enhanced Task Automation and Flexibility

Composition of Experts: A Modular Compound AI System Leveraging Large Language Models

Exploiting Language Models as a Source of Knowledge for Cognitive Agents

On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

CompeteAI: Understanding the Competition Dynamics of Large Language Model-based Agents

Cognitive Architectures for Language Agents

Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation

Collaborative AI in Sentiment Analysis: System Architecture, Data Prediction and Deployment Strategies

LLM-based Optimization of Compound AI Systems: A Survey

CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agents

Enhancing Sentiment Analysis with Collaborative AI: Architecture, Predictions, and Deployment Strategies

Exploring Autonomous Agents through the Lens of Large Language Models: A Review