CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications

Yupeng Cao,Zhiyuan Yao,Zhi Chen,Zhiyang Deng

2024-07-02

Abstract:The integration of Large Language Models (LLMs) into financial analysis has garnered significant attention in the NLP community. This paper presents our solution to IJCAI-2024 FinLLM challenge, investigating the capabilities of LLMs within three critical areas of financial tasks: financial classification, financial text summarization, and single stock trading. We adopted Llama3-8B and Mistral-7B as base models, fine-tuning them through Parameter Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA) approaches. To enhance model performance, we combine datasets from task 1 and task 2 for data fusion. Our approach aims to tackle these diverse tasks in a comprehensive and integrated manner, showcasing LLMs' capacity to address diverse and complex financial tasks with improved accuracy and decision-making capabilities.

Computational Engineering, Finance, and Science,Artificial Intelligence,Machine Learning,Computational Finance

What problem does this paper attempt to address?

The paper aims to address several key challenges in the application of large language models (LLMs) in the financial domain, specifically including:

1. **Financial Classification**: The paper explores how to fine-tune large language models to distinguish between claims and premises in financial texts. This is fundamental to understanding the structure of financial narratives and is crucial for downstream applications such as sentiment analysis, risk assessment, and investment decision-making.

2. **Financial Text Summarization**: The study investigates how to compress lengthy financial documents into concise summaries, retaining key information and insights so that stakeholders can make quick and effective decisions without having to browse through extensive reports.

3. **Single Stock Trading**: It explores how to utilize large language models to analyze various financial texts and other related data to predict the future price trends of individual stocks and make trading decisions based on these predictions.

The paper employs parameter-efficient fine-tuning (PEFT) and low-rank adaptation (LoRA) techniques to fine-tune pre-trained large language models (such as Llama3-8B and Mistral-7B). It also combines datasets from tasks 1 and 2 for data fusion to enhance the model's performance in these diverse tasks. Experimental results show that the data fusion strategy significantly improves model performance in financial classification and text summarization tasks. However, it does not show noticeable improvement in the single stock trading task, possibly because the single stock trading task is more complex and may require more advanced data fusion steps or larger-scale datasets for support.