Abstract:Large Language Models (LLMs) have demonstrated impressive capabilities across diverse Natural Language Processing (NLP) tasks, including language understanding, reasoning, and generation. However, general-domain LLMs often struggle with financial tasks due to the technical and specialized nature of financial texts. This study investigates the efficacy of instruction fine-tuning smaller-scale LLMs, including Mistral-7B, Llama3-8B, and Phi3-mini, to enhance their performance in financial text classification tasks. We fine-tuned both instruction-tuned and base models across four financial classification tasks, achieving significant improvements in task-specific performance. Furthermore, we evaluated the zero-shot capabilities of these fine-tuned models on three unseen complex financial tasks, including argument classification, deal completeness classification, and causal classification. Our results indicate while base model fine-tuning led to greater degradation, instruction-tuned models maintained more robust performance. To address this degradation, we employed model merging techniques, integrating single-task domain-specific fine-tuned models with the base model. Using this merging method resulted in significant enhancements in zero-shot performance, even exceeding the original model's accuracy on certain datasets. Our findings underscore the effectiveness of instruction fine-tuning and model merging for adapting LLMs to specialized financial text classification tasks.

What problem does this paper attempt to address?

The paper attempts to address the issue that large language models (LLMs) in general domains perform poorly on financial text classification tasks due to the technicality and specialization of financial texts. Specifically, the paper investigates improving the performance of smaller-scale LLMs (such as Mistral-7B, Llama3-8B, and Phi3-mini) on financial text classification tasks through instruction fine-tuning. The main objectives of the paper include: 1. **Evaluating the effect of instruction fine-tuning**: Improving the performance of smaller-scale LLMs on financial text classification tasks through instruction fine-tuning. 2. **Comparing the performance of base models and instruction fine-tuned models**: Studying the performance differences between base models and instruction fine-tuned models on specific tasks. 3. **Assessing zero-shot capabilities**: Testing the zero-shot capabilities of these fine-tuned models on unseen complex financial tasks. 4. **Mitigating performance degradation**: Reducing performance degradation on unseen tasks after fine-tuning on specific tasks through model merging techniques. The paper validates the effectiveness of these methods through experiments and presents the following contributions: - **Instruction fine-tuning experiments**: Conducted instruction fine-tuning experiments on four financial domain datasets, finding that increasing the number of examples reduces the performance of small models. - **Fine-tuning smaller models**: Demonstrated the feasibility of using smaller, more efficient models for domain-specific tasks. - **Significant performance improvement**: Achieved significant performance improvements for all models through fine-tuning base models and instruction fine-tuned models, with instruction fine-tuned models showing stronger robustness in handling complex financial tasks. - **Model merging techniques**: Introduced model merging techniques that effectively learn domain-specific knowledge while mitigating zero-shot performance degradation, with significant improvements observed particularly in the Mistral-7B model. - **Resource efficiency**: Emphasized the advantages of using smaller LLMs for fine-tuning in the financial domain, which can reduce resource requirements while maintaining high performance. Overall, the paper aims to improve the performance of small LLMs on financial text classification tasks through instruction fine-tuning and model merging techniques, while maintaining their generalization ability on unseen tasks.

A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification

CatMemo at the FinLLM Challenge Task: Fine-Tuning Large Language Models using Data Fusion in Financial Applications

Fine-tuning and Utilization Methods of Domain-specific LLMs

FMDLlama: Financial Misinformation Detection based on Large Language Models

Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance

Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models

Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Large Language Model Adaptation for Financial Sentiment Analysis

The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Merging

SNFinLLM: Systematic and Nuanced Financial Domain Adaptation of Chinese Large Language Models

Empirical Study of LLM Fine-Tuning for Text Classification in Legal Document Review

InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning

PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance

Data-Centric Financial Large Language Models

Maybe Only 0.5 Training Data Instruction Tuning

Financial News Analytics Using Fine-Tuned Llama 2 GPT Model

DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning

Evaluating Large Language Models on Financial Report Summarization: An Empirical Study

FinTeamExperts: Role Specialized MOEs For Financial Analysis

Advancing Single- and Multi-task Text Classification through Large Language Model Fine-tuning

Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs