Abstract:Large Language Models (LLMs), such as ChatGPT, Phi3 and Llama-3, are leading a significant leap in AI, as they can generalize knowledge from their training to new tasks without fine-tuning. However, their application in the financial domain remains relatively limited. The financial field is inherently complex, requiring a deep understanding across various perspectives, from macro, micro economic trend to quantitative analysis. Motivated by this complexity, a mixture of expert LLMs tailored to specific financial domains could offer a more comprehensive understanding for intricate financial tasks. In this paper, we present the FinTeamExperts, a role-specialized LLM framework structured as a Mixture of Experts (MOEs) for financial analysis. The framework simulates a collaborative team setting by training each model to specialize in distinct roles: Macro Analysts, Micro analysts, and Quantitative Analysts. This role-specific specialization enhances the model's ability to integrate their domain-specific expertise. We achieve this by training three 8-billion parameter models on different corpus, each dedicated to excelling in specific finance-related roles. We then instruct-tune FinTeamExperts on downstream tasks to align with practical financial tasks. The experimental results show that FinTeamExperts outperform all models of the same size and larger on three out of four datasets. On the fourth dataset, which presents a more complex task, FinTeamExperts still surpass all models of the same size. This highlights the success of our role-based specialization approach and the continued training approach for FinTeamExperts.

What problem does this paper attempt to address?

The paper attempts to address the challenges faced when applying large language models (LLMs) in the financial domain. Although existing large language models perform excellently on many tasks, their performance remains limited in the complex and highly specialized field of financial analysis. Tasks in the financial domain not only require a broad knowledge base but also an in-depth understanding of different perspectives such as macroeconomic trends, microeconomic analysis, and quantitative analysis. To this end, the paper proposes **FinTeamExperts**, a role-based specialized Mixture of Experts (MOEs) framework designed to improve the performance of financial analysis tasks. Specifically, FinTeamExperts simulates a collaborative team setting, training each model to specialize in different roles: macro analyst, micro analyst, and quantitative analyst. This role-specific expertise enhances the model's ability to integrate domain-specific knowledge. The main contributions of the paper include: 1. **Role Specialization Framework**: FinTeamExperts is the first large language model framework to apply role specialization in the financial domain. 2. **Improved MOEs Architecture**: Introduces techniques such as dynamic routing, role-specific training, and hierarchical expertise, significantly enhancing the model's performance on downstream financial tasks. 3. **Experimental Validation**: Through extensive experiments, it is demonstrated that FinTeamExperts outperforms existing large language models on multiple financial datasets. Overall, by proposing the FinTeamExperts framework, the paper demonstrates how role specialization and mixture of experts models can be utilized to enhance the performance of financial analysis tasks, providing new insights and tools for AI applications in the financial domain.

FinTeamExperts: Role Specialized MOEs For Financial Analysis

TradExpert: Revolutionizing Trading with Mixture of Expert LLMs

Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance

DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning

FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models

The FinBen: an Holistic Financial Benchmark for Large Language Models

Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing

Data-centric financial large language models

Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications

FinBen: A Holistic Financial Benchmark for Large Language Models

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

Equipping Language Models with Tool Use Capability for Tabular Data Analysis in Finance

FinGPT: Democratizing Internet-scale Data for Financial Large Language Models

Revolutionizing Finance with LLMs: An Overview of Applications and Insights

A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification

FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models

A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist