Abstract:As large language models become increasingly prevalent in the financial sector, there is a pressing need for a standardized method to comprehensively assess their performance. However, existing finance benchmarks often suffer from limited language and task coverage, as well as challenges such as low-quality datasets and inadequate adaptability for LLM evaluation. To address these limitations, we propose "Golden Touchstone", the first comprehensive bilingual benchmark for financial LLMs, which incorporates representative datasets from both Chinese and English across eight core financial NLP tasks. Developed from extensive open source data collection and industry-specific demands, this benchmark includes a variety of financial tasks aimed at thoroughly assessing models' language understanding and generation capabilities. Through comparative analysis of major models on the benchmark, such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and limitations in processing complex financial information. Additionally, we open-sourced Touchstone-GPT, a financial LLM trained through continual pre-training and financial instruction tuning, which demonstrates strong performance on the bilingual benchmark but still has limitations in specific <a class="link-external link-http" href="http://tasks.This" rel="external noopener nofollow">this http URL</a> research not only provides the financial large language models with a practical evaluation tool but also guides the development and optimization of future research. The source code for Golden Touchstone and model weight of Touchstone-GPT have been made publicly available at \url{<a class="link-external link-https" href="https://github.com/IDEA-FinAI/Golden-Touchstone" rel="external noopener nofollow">this https URL</a>}, contributing to the ongoing evolution of FinLLMs and fostering further research in this critical area.

Customized FinGPT Search Agents Using Foundation Models

FinGPT: Open-Source Financial Large Language Models

FinGPT: Democratizing Internet-scale Data for Financial Large Language Models

FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets

FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models

A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist

FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models

FinTeamExperts: Role Specialized MOEs For Financial Analysis

FinRobot: AI Agent for Equity Research and Valuation with Large Language Models

BloombergGPT: A Large Language Model for Finance

FinBen: A Holistic Financial Benchmark for Large Language Models

TradingGPT: Multi-Agent System with Layered Memory and Distinct Characters for Enhanced Financial Trading Performance

PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methods

CFGPT: Chinese Financial Assistant with Large Language Model

FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis

Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing

FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

RA-CFGPT: Chinese financial assistant with retrieval-augmented large language model

Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models