Abstract:Large Language Models (LLMs) excel in diverse applications including generation of code snippets, but often struggle with generating code for complex Machine Learning (ML) tasks. Although existing LLM single-agent based systems give varying performance depending on the task complexity, they purely rely on larger and expensive models such as GPT-4. Our investigation reveals that no-cost and low-cost models such as Gemini-Pro, Mixtral and CodeLlama perform far worse than GPT-4 in a single-agent setting. With the motivation of developing a cost-efficient LLM based solution for solving ML tasks, we propose an LLM Multi-Agent based system which leverages combination of experts using profiling, efficient retrieval of past observations, LLM cascades, and ask-the-expert calls. Through empirical analysis on ML engineering tasks in the MLAgentBench benchmark, we demonstrate the effectiveness of our system, using no-cost models, namely Gemini as the base LLM, paired with GPT-4 in cascade and expert to serve occasional ask-the-expert calls for planning. With 94.2\% reduction in the cost (from \$0.931 per run cost averaged over all tasks for GPT-4 single agent system to \$0.054), our system is able to yield better average success rate of 32.95\% as compared to GPT-4 single-agent system yielding 22.72\% success rate averaged over all the tasks of MLAgentBench.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How to build an efficient and economical multi - agent system using low - cost or free large - scale language models (LLMs) in automated machine learning tasks. Specifically, the paper focuses on: 1. **The high - cost problem**: - Although existing systems based on a single large - scale language model (such as GPT - 4) have better performance, their running costs are very high (about $0.931 on average per task). This is a significant obstacle for large - scale experiments and practical applications. 2. **The insufficient performance of low - cost models**: - Low - cost or free models (such as Gemini - Pro, Mixtral, CodeLlama) perform poorly when handling complex machine learning tasks and cannot achieve the same success rate as expensive models. 3. **The lack of an effective multi - agent cooperation mechanism**: - A single - agent system has difficulty dealing with complex machine learning tasks, and existing multi - agent frameworks have not fully utilized open - source LLMs to solve these problems. To solve the above problems, the author proposes a multi - agent system named BudgetMLAgent, which improves performance and reduces costs in the following ways: - **Combination of multiple LLM agents**: Combine multiple LLM experts, and each agent has a different role (such as planner and executor) to complete tasks collaboratively. - **LLM cascading**: Adopt the method of gradually invoking LLMs with different costs. First, try low - cost models, and if unsuccessful, invoke more expensive models (such as GPT - 4). - **Efficient retrieval of past observations**: Agents can retrieve and summarize relevant information from long - term memory, thereby improving decision - making efficiency. - **Occasionally seek help from experts**: When necessary, invoke a higher - level LLM (such as GPT - 4) for planning to ensure the success rate of tasks. Through these methods, BudgetMLAgent not only significantly reduces costs (up to 94.2%) but also improves the success rate of tasks (on average, from 22.72% to 32.95%). This makes it possible to use low - cost or free LLMs to handle complex machine learning tasks.

BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks

MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation

How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments

DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning

AgentBench: Evaluating LLMs as Agents

Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration

Towards Efficient Multi-Agent Learning Systems

MegaAgent: A Practical Framework for Autonomous Cooperation in Large-Scale LLM Agent Systems

Affordable Generative Agents

Small LLMs Are Weak Tool Learners: A Multi-LLM Agent

MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMs

AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML

Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent System

Enhancing LLMs for Power System Simulations: A Feedback-driven Multi-agent Framework

TrainerAgent: Customizable and Efficient Model Training Through LLM-Powered Multi-Agent System.

Training Language Model Agents without Modifying Language Models

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents