Abstract:As large language models become increasingly prevalent in the financial sector, there is a pressing need for a standardized method to comprehensively assess their performance. However, existing finance benchmarks often suffer from limited language and task coverage, as well as challenges such as low-quality datasets and inadequate adaptability for LLM evaluation. To address these limitations, we propose "Golden Touchstone", the first comprehensive bilingual benchmark for financial LLMs, which incorporates representative datasets from both Chinese and English across eight core financial NLP tasks. Developed from extensive open source data collection and industry-specific demands, this benchmark includes a variety of financial tasks aimed at thoroughly assessing models' language understanding and generation capabilities. Through comparative analysis of major models on the benchmark, such as GPT-4o Llama3, FinGPT and FinMA, we reveal their strengths and limitations in processing complex financial information. Additionally, we open-sourced Touchstone-GPT, a financial LLM trained through continual pre-training and financial instruction tuning, which demonstrates strong performance on the bilingual benchmark but still has limitations in specific <a class="link-external link-http" href="http://tasks.This" rel="external noopener nofollow">this http URL</a> research not only provides the financial large language models with a practical evaluation tool but also guides the development and optimization of future research. The source code for Golden Touchstone and model weight of Touchstone-GPT have been made publicly available at \url{<a class="link-external link-https" href="https://github.com/IDEA-FinAI/Golden-Touchstone" rel="external noopener nofollow">this https URL</a>}, contributing to the ongoing evolution of FinLLMs and fostering further research in this critical area.

WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domain

Data-centric financial large language models

FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining

CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models

A Data-Centric Approach for Financial Large Language Models with Abductive Augmentation Reasoning

Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models

Evaluating Large Language Models on Financial Report Summarization: An Empirical Study

A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges

SNFinLLM: Systematic and Nuanced Financial Domain Adaptation of Chinese Large Language Models

FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models

UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models

No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks

Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset

Baichuan4-Finance Technical Report

Large Language Models in Finance: A Survey

CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model

BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark

Large Language Model Adaptation for Financial Sentiment Analysis

FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models

Pre-trained Large Language Models for Financial Sentiment Analysis