Abstract:Accurate and transparent financial information disclosure is crucial in the fields of accounting and finance, ensuring market efficiency and investor confidence. Among many information disclosure platforms, the Chinese stock exchanges' investor interactive platform provides a novel and interactive way for listed firms to disclose information of interest to investors through an online question-and-answer (Q&A) format. However, it is common for listed firms to respond to questions with limited or no substantive information, and automatically evaluating the quality of financial information disclosure on large amounts of Q&A pairs is challenging. This paper builds a benchmark FinTruthQA, that can evaluate advanced natural language processing (NLP) techniques for the automatic quality assessment of information disclosure in financial Q&A data. FinTruthQA comprises 6,000 real-world financial Q&A entries and each Q&A was manually annotated based on four conceptual dimensions of accounting. We benchmarked various NLP techniques on FinTruthQA, including statistical machine learning models, pre-trained language model and their fine-tuned versions, as well as the large language model GPT-4. Experiments showed that existing NLP models have strong predictive ability for real question identification and question relevance tasks, but are suboptimal for answer relevance and answer readability tasks. By establishing this benchmark, we provide a robust foundation for the automatic evaluation of information disclosure, significantly enhancing the transparency and quality of financial reporting. FinTruthQA can be used by auditors, regulators, and financial analysts for real-time monitoring and data-driven decision-making, as well as by researchers for advanced studies in accounting and finance, ultimately fostering greater trust and efficiency in the financial markets.

What problem does this paper attempt to address?

The paper aims to address the issue of evaluating the quality of financial information disclosure, particularly on the Investor Interactive Platforms of the Shanghai Stock Exchange (SSE) and Shenzhen Stock Exchange (SZSE) in China. These platforms provide a new way for listed companies to directly communicate with investors, disclosing information of interest to investors through an online question-and-answer format (Q&A). However, listed companies often provide limited or no substantive information in response to investor inquiries, making it challenging to assess the quality of financial information disclosure through a large number of Q&A. To address this issue, the paper constructs a benchmark dataset called FinTruthQA to evaluate the automated ability of advanced natural language processing (NLP) techniques in assessing the quality of information disclosure in financial Q&A data. FinTruthQA consists of 6000 real-world financial Q&A entries, each manually labeled based on four dimensions of accounting concepts: question identification, question relevance, answer readability, and answer relevance. By establishing this benchmark, the paper provides a solid foundation for evaluating information disclosure, significantly improving the transparency and quality of financial reporting. It can be used for real-time monitoring and data-driven decision-making by auditors, regulatory agencies, and financial analysts, as well as advanced research in the accounting and finance fields, ultimately promoting greater trust and efficiency in the financial market. The paper also explores the existing applications of NLP in automated financial and accounting text analysis, such as automated analysis of annual financial reports and automated sentiment analysis systems for financial news articles. However, it points out the lack of a dedicated standard dataset for evaluating the quality of financial information disclosure on these Q&A platforms. The development, implementation, and evaluation of FinTruthQA are detailed, highlighting potential impacts including technical challenges such as natural language interpretation, context sensitivity, and subjectivity in assessing relevance and readability. It emphasizes the potential impact on the financial sector and the broader significance of automated quality assessment in investor communication.

FinTruthQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure

FinTextQA: A Dataset for Long-form Financial Question Answering

FinanceBench: A New Benchmark for Financial Question Answering

FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models

FinQA: A Dataset of Numerical Reasoning over Financial Data

FinBen: A Holistic Financial Benchmark for Large Language Models

CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models

FinDABench: Benchmarking Financial Data Analysis Ability of Large Language Models

FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering

AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework

Fintech Key-Phrase: A New Chinese Financial High-Tech Dataset Accelerating Expression-Level Information Retrieval

Automatic Assessment of Information Disclosure Quality in Chinese Annual Reports

Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models

SEC-QA: A Systematic Evaluation Corpus for Financial QA

A Novel DeBERTa-based Model for Financial Question Answering Task

PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance

Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models

Measuring Consistency in Text-based Financial Forecasting Models

MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning

Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation

Natural Language Processing and Text Mining Algorithms for Financial Accounting Information Disclosure