Qianqian Xie,Weiguang Han,Zhengyu Chen,Ruoyu Xiang,Xiao Zhang,Yueru He,Mengxi Xiao,Dong Li,Yongfu Dai,Duanyu Feng,Yijing Xu,Haoqiang Kang,Ziyan Kuang,Chenhan Yuan,Kailai Yang,Zheheng Luo,Tianlin Zhang,Zhiwei Liu,Guojun Xiong,Zhiyang Deng,Yuechen Jiang,Zhiyuan Yao,Haohang Li,Yangyang Yu,Gang Hu,Jiajia Huang,Xiao-Yang Liu,Alejandro Lopez-Lira,Benyou Wang,Yanzhao Lai,Hao Wang,Min Peng,Sophia Ananiadou,Jimin Huang

Abstract:LLMs have transformed NLP and shown promise in various fields, yet their potential in finance is underexplored due to a lack of comprehensive evaluation benchmarks, the rapid development of LLMs, and the complexity of financial tasks. In this paper, we introduce FinBen, the first extensive open-source evaluation benchmark, including 36 datasets spanning 24 financial tasks, covering seven critical aspects: information extraction (IE), textual analysis, question answering (QA), text generation, risk management, forecasting, and decision-making. FinBen offers several key innovations: a broader range of tasks and datasets, the first evaluation of stock trading, novel agent and Retrieval-Augmented Generation (RAG) evaluation, and three novel open-source evaluation datasets for text summarization, question answering, and stock trading. Our evaluation of 15 representative LLMs, including GPT-4, ChatGPT, and the latest Gemini, reveals several key findings: While LLMs excel in IE and textual analysis, they struggle with advanced reasoning and complex tasks like text generation and forecasting. GPT-4 excels in IE and stock trading, while Gemini is better at text generation and forecasting. Instruction-tuned LLMs improve textual analysis but offer limited benefits for complex tasks such as QA. FinBen has been used to host the first financial LLMs shared task at the FinNLP-AgentScen workshop during IJCAI-2024, attracting 12 teams. Their novel solutions outperformed GPT-4, showcasing FinBen's potential to drive innovation in financial LLMs. All datasets, results, and codes are released for the research community: <a class="link-external link-https" href="https://github.com/The-FinAI/PIXIU" rel="external noopener nofollow">this https URL</a>.

FinMatcher at FinSim-2: Hypernym Detection in the Financial Services Domain using Knowledge Graphs

IITK at the FinSim Task: Hypernym Detection in Financial Domain via Context-Free and Contextualized Word Embeddings

JSI at the FinSim-2 task: Ontology-Augmented Financial Concept Classification

DICoE@FinSim-3: Financial Hypernym Detection using Augmented Terms and Distance-based Features

Learning Semantic Text Similarity to rank Hypernyms of Financial Terms

Yseop at FinSim-3 Shared Task 2021: Specializing Financial Domain Learning with Phrase Representations

The Financial Causality Extraction Shared Task (FinCausal 2022)

LongFin: A Multimodal Document Understanding Model for Long Financial Domain Documents

FinDKG: Dynamic Knowledge Graphs with Large Language Models for Detecting Global Trends in Financial Markets

FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

dzFinNlp at AraFinNLP: Improving Intent Detection in Financial Conversational Agents

FinBen: A Holistic Financial Benchmark for Large Language Models

FISHNET: Financial Intelligence from Sub-querying, Harmonizing, Neural-Conditioning, Expert Swarms, and Task Planning

Using contextual sentence analysis models to recognize ESG concepts

FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents

GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models

FinIR 2020: the First Workshop on Information Retrieval in Finance

FinBrain: when finance meets AI 2.0

German FinBERT: A German Pre-trained Language Model

FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction