Abstract:In the current media landscape, understanding the framing of information is crucial for critical consumption and informed decision making. Framing analysis is a valuable tool for identifying the underlying perspectives used to present information, and has been applied to a variety of media formats, including television programs. However, manual analysis of framing can be time-consuming and labor-intensive. This is where large language models (LLMs) can play a key role. In this paper, we propose a novel approach to use prompt-engineering to identify the framing of spoken content in television programs. Our findings indicate that prompt-engineering LLMs can be used as a support tool to identify frames, with agreement rates between human and machine reaching up to 43\%. As LLMs are still under development, we believe that our approach has the potential to be refined and further improved. The potential of this technology for interactive media applications is vast, including the development of support tools for journalists, educational resources for students of journalism learning about framing and related concepts, and interactive media experiences for audiences.
What problem does this paper attempt to address?
### Problems the Paper Attempts to Solve
This paper attempts to address the issue of information framing analysis in television programs. Specifically, the authors propose a method for automatic framing analysis using large language models (LLMs) to identify the types of frames in spoken content of TV shows. Traditional framing analysis methods are time-consuming and labor-intensive, whereas LLMs can improve efficiency and accuracy by automating this process.
### Main Research Questions
1. **RQ1**: How accurate is the GPT model in identifying frames in TV shows using prompt-engineering methods?
2. **RQ2**: What are the limitations of the GPT-3.5 model in classifying the frames of TV show transcripts compared to human annotation?
3. **RQ3**: How can GPT models and prompt-engineering be used for framing analysis in potential interactive applications?
### Research Background
Framing analysis is a technique for understanding how media content shapes our perception of the world. By identifying frames in news stories, one can understand which aspects are emphasized. For example, two news reports on the Russia-Ukraine war might differ: one might emphasize the economic impact on Europe (economic frame), while the other might highlight displaced families due to the war (human interest frame). Understanding these frames can help media audiences consume information more critically and make informed decisions.
In recent years, with the development of large language models (LLMs), new possibilities have emerged in media content analysis. LLMs have shown excellent performance in various fields (such as news, natural language processing tasks, etc.), particularly in identifying frames in text. However, traditional framing analysis methods are often cumbersome and labor-intensive, limiting their application in large-scale analysis. Therefore, automating the framing detection process using LLMs can improve the efficiency and comprehensiveness of media content analysis.
### Research Methods
1. **Dataset**: A dataset was created containing transcripts of two current affairs programs from Dutch television.
2. **Annotation System**: A Google Forms-based annotation system was developed, where experts annotated the transcripts according to frame definitions.
3. **Machine Classification**: The GPT-3.5 model and prompt-engineering techniques were used to classify the transcripts and identify the main frames.
4. **Results Analysis**: The agreement and disagreement rates between human annotations and machine classifications were analyzed, and the impact of text length on classification results was explored.
### Results and Discussion
1. **EenVandaag**: Among 1000 samples, the agreement rate between human annotators and GPT-3.5 was 48.3%. The human interest frame had the highest agreement rate, followed by the conflict frame.
2. **Nieuwsuur**: Among 1000 samples, the agreement rate between human annotators and GPT-3.5 was 38.7%. Similarly, the human interest frame had the highest agreement rate, followed by the conflict frame.
3. **Impact of Text Length**: The study found that text length had little impact on the performance of GPT-3.5. Regardless of text length, the trends in agreement and disagreement rates for long and short texts were similar.
### Potential Applications
1. **Journalist Support**: Helps journalists quickly verify or identify frames in news articles, improving the accuracy and objectivity of reporting.
2. **Media Consumers**: Helps media consumers gain a deeper understanding of perspectives in TV news, promoting critical consumption and informed decision-making.
3. **Research Tool**: Provides researchers with more comprehensive and in-depth media analysis tools.
### Conclusion
This study proposes an automated method for TV program framing analysis using LLMs and prompt-engineering techniques. Although the current agreement rates need improvement, this method has great potential to significantly enhance the efficiency and comprehensiveness of media content analysis.