Beyond Classification: Financial Reasoning in State-of-the-Art Language Models

Guijin Son,Hanearl Jung,Moonjeong Hahm,Keonju Na,Sol Jin

2023-06-26

Abstract:Large Language Models (LLMs), consisting of 100 billion or more parameters, have demonstrated remarkable ability in complex multi-step reasoning tasks. However, the application of such generic advancements has been limited to a few fields, such as clinical or legal, with the field of financial reasoning remaining largely unexplored. To the best of our knowledge, the ability of LLMs to solve financial reasoning problems has never been dealt with, and whether it can be performed at any scale remains unknown. To address this knowledge gap, this research presents a comprehensive investigation into the potential application of LLMs in the financial domain. The investigation includes a detailed exploration of a range of subjects, including task formulation, synthetic data generation, prompting methods, and evaluation capability. Furthermore, the study benchmarks various GPT variants with parameter scales ranging from 2.8B to 13B, with and without instruction tuning, on diverse dataset sizes. By analyzing the results, we reveal that the ability to generate coherent financial reasoning first emerges at 6B parameters, and continues to improve with better instruction-tuning or larger datasets. Additionally, the study provides a publicly accessible dataset named sFIOG (Synthetic-Financial Investment Opinion Generation), consisting of 11,802 synthetic investment thesis samples, to support further research in the field of financial reasoning. Overall, this research seeks to contribute to the understanding of the efficacy of language models in the field of finance, with a particular emphasis on their ability to engage in sophisticated reasoning and analysis within the context of investment decision-making.

Computation and Language,Artificial Intelligence,Computers and Society

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the ability to apply large - language models (LLMs) for complex reasoning in the financial field. Although LLMs perform well in multi - step reasoning tasks, their applications in the field of financial reasoning have been rarely explored. Specifically, the paper discusses the following aspects:

1. **Task Definition**: How to use financial investment opinion generation (FIOG) as a task to train or prompt a language model so that it can generate logically coherent and persuasive investment opinions.
2. **Data Generation**: How to construct a synthetic dataset through analyst reports written by experts to support research on financial reasoning.
3. **Prompting Method**: A new prompting method - In - Context Question Answering - is proposed to generate more controlled outputs.
4. **Evaluation Ability**: The performance of different - sized GPT variants (from 2.8B to 13B parameters) on different datasets is studied, and the relationship between model size and financial reasoning ability is explored.
5. **Consistency between Human and Model Evaluation**: The differences between LLM - based evaluators and human evaluators in financial text evaluation are compared, revealing the limitations of LLMs in financial reasoning tasks.

The main goal of the paper is to fill the knowledge gap in the use of LLMs in the field of financial reasoning. Through systematic research, it provides an understanding of the reasoning ability of language models in the financial field, especially their application in investment decision - making.