Abstract:Abstract Deep learning has recently been providing step-change capabilities, particularly using transformer models, for natural language processing applications such as question answering, query-based summarization, and language translation for general-purpose context. We have developed a geoscience-specific language processing solution using such models to enable geoscientists to perform rapid, fully-quantitative and automated analysis of large corpuses of data and gain insights. One of the key transformer-based model is BERT (Bidirectional Encoder Representations from Transformers). It is trained with a large amount of general-purpose text (e.g., Common Crawl). Use of such a model for geoscience applications can face a number of challenges. One is due to the insignificant presence of geoscience-specific vocabulary in general-purpose context (e.g. daily language) and the other one is due to the geoscience jargon (domain-specific meaning of words). For example, salt is more likely to be associated with table salt within a daily language but it is used as a subsurface entity within geosciences. To elevate such challenges, we retrained a pre-trained BERT model with our 20M internal geoscientific records. We will refer the retrained model as GeoBERT. We fine-tuned the GeoBERT model for a number of tasks including geoscience question answering and query-based summarization. BERT models are very large in size. For example, BERT-Large has 340M trained parameters. Geoscience language processing with these models, including GeoBERT, could result in a substantial latency when all database is processed at every call of the model. To address this challenge, we developed a retriever-reader engine consisting of an embedding-based similarity search as a context retrieval step, which helps the solution to narrow the context for a given query before processing the context with GeoBERT. We built a solution integrating context-retrieval and GeoBERT models. Benchmarks show that it is effective to help geologists to identify answers and context for given questions. The prototype will also produce a summary to different granularity for a given set of documents. We have also demonstrated that domain-specific GeoBERT outperforms general-purpose BERT for geoscience applications.

ESGPDE: an ESG Performance Data Extraction Model

ESGBERT: Language Model to Help with Classification Tasks Related to Companies Environmental, Social, and Governance Practices

ESG2PreEM: Automated ESG grade assessment framework using pre-trained ensemble models

Evaluating the performance of state-of-the-art esg domain-specific pre-trained large language models in text classification against existing models and traditional machine learning techniques

ESGReveal: An LLM-based approach for extracting structured data from ESG reports

Leveraging BERT Language Models for Multi-Lingual ESG Issue Identification

Bridging the gap in ESG measurement: Using NLP to quantify environmental, social, and governance communication

Using contextual sentence analysis models to recognize ESG concepts

ESG Classification by Implicit Rule Learning via GPT-4

EaSyGuide : ESG Issue Identification Framework leveraging Abilities of Generative Large Language Models

ESG Sentiment Analysis: comparing human and language model performance including GPT

Transforming Sentiment Analysis in the Financial Domain with ChatGPT

Exploring the Feasibility of ChatGPT for Event Extraction

Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness

Geoscience Language Processing for Exploration

Jetsons at FinNLP 2024: Towards Understanding the ESG Impact of a News Article using Transformer-based Models

Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors

Performance and sustainability of BERT derivatives in dyadic data

Towards Efficient NLP: A Standard Evaluation and A Strong Baseline

BERT vs GPT for financial engineering

ESG-FTSE: A corpus of news articles with ESG relevance labels and use cases