Leveraging Natural Language and Item Response Theory Models for ESG Scoring

César Pedrosa Soares

2024-07-30

Abstract:This paper explores an innovative approach to Environmental, Social, and Governance (ESG) scoring by integrating Natural Language Processing (NLP) techniques with Item Response Theory (IRT), specifically the Rasch model. The study utilizes a comprehensive dataset of news articles in Portuguese related to Petrobras, a major oil company in Brazil, collected from 2022 and 2023. The data is filtered and classified for ESG-related sentiments using advanced NLP methods. The Rasch model is then applied to evaluate the psychometric properties of these ESG measures, providing a nuanced assessment of ESG sentiment trends over time. The results demonstrate the efficacy of this methodology in offering a more precise and reliable measurement of ESG factors, highlighting significant periods and trends. This approach may enhance the robustness of ESG metrics and contribute to the broader field of sustainability and finance by offering a deeper understanding of the temporal dynamics in ESG reporting.

Artificial Intelligence,General Finance,Methodology

What problem does this paper attempt to address?

### The Problem This Paper Attempts to Solve

This paper aims to explore an innovative approach to improve Environmental, Social, and Governance (ESG) scoring by combining Natural Language Processing (NLP) techniques and Item Response Theory (IRT), specifically the Rasch model. Specifically, the study utilizes a dataset of Portuguese news articles related to the Brazilian oil company Petrobras, collected in 2022 and 2023. These data are filtered and classified using advanced NLP methods to extract sentiment information related to ESG. Then, the Rasch model is applied to assess the psychometric properties of these ESG metrics, providing a detailed evaluation of ESG sentiment trends.

### Main Objectives

1. **Improve the accuracy and reliability of ESG scoring**: By combining NLP and IRT techniques, the paper aims to provide a more accurate and reliable method for ESG scoring.
2. **Enhance the robustness of ESG metrics**: Traditional ESG scoring methods may lack psychometric validation, leading to insufficient reliability and interpretability of the results. This paper addresses this shortcoming by introducing the IRT method, particularly the Rasch model.
3. **Reveal the temporal dynamics of ESG reporting**: By analyzing the changes in ESG sentiment over different periods, the paper aims to uncover key periods and trends, providing deeper insights for research in sustainability and finance.

### Method Overview

1. **Data Collection**: Download and scrape Portuguese news articles from the Global Database of Events, Language, and Tone (GDELT) for 2022 and 2023.
2. **Data Preprocessing**: Use the Portuguese BERT model for text embedding and similarity algorithms to filter out ESG-related news articles.
3. **Sentiment Classification**: Train a Portuguese BERT classification model to classify the sentiment of news articles as positive or negative.
4. **Psychometric Analysis**: Structure the classified data into a binary dataset and apply the Rasch model to assess the psychometric properties of ESG metrics.

### Results

1. **Optimal Model Parameters**: Through TOPSIS scoring, the optimal combination of model parameters was determined, including learning rate, number of layers, hidden layer size, batch size, number of training epochs, and maximum length.
2. **Sentiment Classification Performance**: The model performed excellently in distinguishing negative and positive ESG news, achieving an accuracy of 97.37%.
3. **Psychometric Analysis**: Item Information Curves (IIC) and Item Characteristic Curves (ICC) revealed the impact of different months on ESG sentiment, identifying key periods and trends.

### Discussion

1. **Advantages of the Rasch Model**: The Rasch model provides more precise ESG sentiment measurement by considering item difficulty and respondent ability, placing all data on an interval scale for more accurate comparison.
2. **Temporal Dynamics Analysis**: IIC and ICC can identify which time periods provided the most significant data, helping organizations track ESG sentiment trends more effectively.
3. **Potential for Comprehensive Application**: The output of the Rasch model can be combined with other ESG indicators and qualitative data to provide a more comprehensive view of ESG performance.

### Conclusion

This exploratory study introduces the Rasch model to ESG research, providing a new approach. The results indicate that this method has advantages in providing precise, reliable, and consistent measurements, helping to improve the accuracy of ESG accountability, support more informed decision-making processes, and enhance stakeholder trust through clearer and more credible reporting. Future research can further explore and optimize the application of the Rasch model in ESG sentiment analysis.