BERT vs GPT for financial engineering

Edward Sharkey,Philip Treleaven

2024-04-24

Abstract:The paper benchmarks several Transformer models [4], to show how these models can judge sentiment from a news event. This signal can then be used for downstream modelling and signal identification for commodity trading. We find that fine-tuned BERT models outperform fine-tuned or vanilla GPT models on this task. Transformer models have revolutionized the field of natural language processing (NLP) in recent years, achieving state-of-the-art results on various tasks such as machine translation, text summarization, question answering, and natural language generation. Among the most prominent transformer models are Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT), which differ in their architectures and objectives.

Statistical Finance,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

This paper compares the performance of two Transformer models, BERT and GPT, in commodity prediction. The researchers found that the fine-tuned BERT model performs better than the fine-tuned or original GPT model in sentiment analysis tasks. Despite GPT receiving more attention from the public, BERT may become a tool similar to XGBOOST in certain aspects, especially in terms of accuracy and interpretability.

The paper first introduces the basic architecture and objective differences between BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer). Then, the researchers conducted benchmark tests on four models (FinBERT, CopBERT, GPT, and CopGPT) to evaluate their performance in commodity prediction, particularly copper price prediction. The results show that CopBERT outperforms GPT4 and CopGPT in sentiment analysis tasks, with an F1 score approximately 10% and 16% higher, respectively.

The paper emphasizes the importance of understanding how models reach conclusions because the GPT model sometimes produces "illusions," i.e., inaccurate predictions, which poses interpretability challenges for financial engineering tasks. By creating a graph network based on cosine similarity, the researchers demonstrate the attention patterns of CopBERT to improve model interpretability.

In summary, although BERT may not match GPT in predictive ability, it offers higher interpretability, making it an interesting choice in financial engineering tasks, especially in scenarios that require a combination of accuracy and interpretability. The paper also proposes future research directions, including exploring BERT's mechanism interpretability and different transfer learning methods.