Muhammad Asad Arshed,Ștefan Cristian Gherghina,Dur-E-Zahra Dur-E-Zahra,Mahnoor Manzoor
Abstract:With the rise of Large Language Models (LLMs), distinguishing between genuine and AI-generated content, particularly in finance, has become challenging. Previous studies have focused on binary identification of ChatGPT-generated content, overlooking other AI tools used for text regeneration. This study addresses this gap by examining various AI-regenerated content types in the finance domain. Objective: The study aims to differentiate between human-generated financial content and AI-regenerated content, specifically focusing on ChatGPT, QuillBot, and SpinBot. It constructs a dataset comprising real text and AI-regenerated text for this purpose. Contribution: This research contributes to the field by providing a dataset that includes various types of AI-regenerated financial content. It also evaluates the performance of different models, particularly highlighting the effectiveness of the Bidirectional Encoder Representations from the Transformers Base Cased model in distinguishing between these content types. Methods: The dataset is meticulously preprocessed to ensure quality and reliability. Various models, including Bidirectional Encoder Representations Base Cased, are fine-tuned and compared with traditional machine learning models using TFIDF and Word2Vec approaches. Results: The Bidirectional Encoder Representations Base Cased model outperforms other models, achieving an accuracy, precision, recall, and F1 score of 0.73, 0.73, 0.73, and 0.72 respectively, in distinguishing between real and AI-regenerated financial content. Conclusion: This study demonstrates the effectiveness of the Bidirectional Encoder Representations base model in differentiating between human-generated financial content and AI-regenerated content. It highlights the importance of considering various AI tools in identifying synthetic content, particularly in the finance domain in Pakistan.
engineering, electrical & electronic,physics, applied,computer science, information systems
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to distinguish financial content generated by humans from that generated by AI tools (such as ChatGPT, QuillBot, and SpinBot). Specifically, the research aims to fill the gaps in existing research, that is, not only focus on the content generated by ChatGPT, but also consider the impact of other AI text - generation tools.
### Research Background and Problems
With the rise of large - language models (LLMs), especially in the financial field, it has become increasingly challenging to distinguish real content from AI - generated content. Previous research mainly focused on binary identification of ChatGPT - generated content, ignoring other AI tools for text regeneration. This has led to an important research gap: how to effectively distinguish financial content generated by different AI tools.
### Research Objectives
The objectives of this research are:
1. **Construct a Dataset**: Create a dataset containing real - text and financial content generated by multiple AI tools (such as ChatGPT, QuillBot, and SpinBot).
2. **Evaluate Model Performance**: Compare the performance of different models in distinguishing these content types, especially emphasizing the effectiveness of the Bidirectional Encoder Representations from Transformers (BERT Base Cased) model.
3. **Improve Recognition Accuracy**: Through experimental verification, find the most effective model to distinguish between real and AI - generated financial content.
### Main Contributions
- Provide a dataset containing multiple AI - generated financial content.
- Evaluate and verify the performance of different models, especially the superiority of the BERT Base Cased model in distinguishing between real and AI - generated content.
- Emphasize the importance of considering multiple AI tools when identifying synthetic content, especially in the financial field.
### Method Overview
1. **Dataset Preparation**: Collect finance - related tweets from Twitter and use ChatGPT, QuillBot, and SpinBot to generate corresponding AI texts to form a balanced dataset.
2. **Pre - processing**: Remove noisy information (such as emojis, URLs, and hashtags) from tweets and perform lemmatization on the text, etc.
3. **Model Training and Evaluation**: Use BERT and its variants (such as BERT Base Cased, BERT Large, etc.) for fine - tuning and compare with traditional machine - learning models (such as TF - IDF and Word2Vec).
### Experimental Results
The experimental results show that the BERT Base Cased model performs excellently in distinguishing between real and AI - generated financial content, with its accuracy, precision, recall, and F1 - score being 0.73, 0.73, 0.73, and 0.72 respectively.
### Conclusion
This research shows that the BERT Base Cased model is very effective in distinguishing financial content generated by humans from that generated by AI. In addition, the research also emphasizes the importance of considering multiple AI tools when identifying synthetic content, especially in the financial field.
### Formula Summary
The evaluation metric formulas involved in the paper are as follows:
- **Accuracy**:
\[
\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{FP}+\text{FN}+\text{TN}}
\]
- **Precision**:
\[
\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}
\]
- **Recall**:
\[
\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}
\]
- **F1 - Score**:
\[
F1 = 2\times\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}
\]