Fine-tuning ClimateBert transformer with ClimaText for the disclosure analysis of climate-related financial risks

Eduardo C. Garrido-Merchán,Cristina González-Barthe,María Coronado Vaca
2023-03-21
Abstract:In recent years there has been a growing demand from financial agents, especially from particular and institutional investors, for companies to report on climate-related financial risks. A vast amount of information, in text format, can be expected to be disclosed in the short term by firms in order to identify these types of risks in their financial and non financial reports, particularly in response to the growing regulation that is being passed on the matter. To this end, this paper applies state-of-the-art NLP techniques to achieve the detection of climate change in text corpora. We use transfer learning to fine-tune two transformer models, BERT and ClimateBert -a recently published DistillRoBERTa-based model that has been specifically tailored for climate text classification-. These two algorithms are based on the transformer architecture which enables learning the contextual relationships between words in a text. We carry out the fine-tuning process of both models on the novel Clima-Text database, consisting of data collected from Wikipedia, 10K Files Reports and web-based claims. Our text classification model obtained from the ClimateBert fine-tuning process on ClimaText, outperforms the models created with BERT and the current state-of-the-art transformer in this particular problem. Our study is the first one to implement on the ClimaText database the recently published ClimateBert algorithm. Based on our results, it can be said that ClimateBert fine-tuned on ClimaText is an outstanding tool within the NLP pre-trained transformer models that may and should be used by investors, institutional agents and companies themselves to monitor the disclosure of climate risk in financial reports. In addition, our transfer learning methodology is cheap in computational terms, thus allowing any organization to perform it.
Computation and Language,Artificial Intelligence
What problem does this paper attempt to address?
The paper primarily aims to address the following issues: 1. **Increased demand for climate change-related risk reporting in the financial sector**: As investors and financial institutions become increasingly concerned with companies' disclosures of their financial risks related to climate change, the paper seeks to meet this demand by automatically analyzing this information using Natural Language Processing (NLP) technology. 2. **Automated identification and analysis of climate change-related information**: Given the vast amount of text-based climate change-related information, manual analysis becomes impractical. Therefore, one of the paper's goals is to develop a tool that can efficiently identify and analyze this information. 3. **Improving analysis accuracy using the latest NLP technology**: The paper employs the latest NLP technology—transfer learning methods to fine-tune pre-trained models (such as BERT and ClimateBert) to enhance performance on specific tasks, namely detecting climate change topics in text corpora. 4. **Evaluating the performance of the ClimateBert model**: The paper pays special attention to the performance evaluation of the ClimateBert model, a pre-trained model based on DistillRoBERTa designed specifically for climate text classification. By fine-tuning on a new database, ClimaText, which includes data collected from Wikipedia, 10-K filings, and web statements, the paper demonstrates ClimateBert's capability in identifying climate change-related risks. 5. **Providing practical tools for investors and regulators**: The ultimate goal of the paper is to provide a powerful tool for investors, regulators, and other stakeholders to monitor the disclosure of climate change risks in corporate financial reports. In summary, the main objective of the paper is to develop and validate an NLP-based method, specifically by fine-tuning the ClimateBert model, to automatically detect and analyze financial risk information related to climate change, thereby supporting financial decision-making.