Mapping the Media Landscape: Predicting Factual Reporting and Political Bias Through Web Interactions

Dairazalia Sánchez-Cortés,Sergio Burdisso,Esaú Villatoro-Tello,Petr Motlicek

DOI: https://doi.org/10.1007/978-3-031-71736-9_7

2024-10-23

Abstract:Bias assessment of news sources is paramount for professionals, organizations, and researchers who rely on truthful evidence for information gathering and reporting. While certain bias indicators are discernible from content analysis, descriptors like political bias and fake news pose greater challenges. In this paper, we propose an extension to a recently presented news media reliability estimation method that focuses on modeling outlets and their longitudinal web interactions. Concretely, we assess the classification performance of four reinforcement learning strategies on a large news media hyperlink graph. Our experiments, targeting two challenging bias descriptors, factual reporting and political bias, showed a significant performance improvement at the source media level. Additionally, we validate our methods on the CLEF 2023 CheckThat! Lab challenge, outperforming the reported results in both, F1-score and the official MAE metric. Furthermore, we contribute by releasing the largest annotated dataset of news source media, categorized with factual reporting and political bias labels. Our findings suggest that profiling news media sources based on their hyperlink interactions over time is feasible, offering a bird's-eye view of evolving media landscapes.

Artificial Intelligence,Computers and Society,Machine Learning

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

This paper aims to address the issue of bias assessment in news media, specifically the evaluation of factual reporting and political bias. Specifically, the authors propose a network interaction modeling approach that uses reinforcement learning strategies to classify the bias of news media. The main contributions of the paper include:

1. **Prediction and Estimation of Bias Descriptors**: Predicting and estimating political bias and the accuracy of factual reporting through the interaction of news media with other media sources.
2. **Validation of Method Robustness**: Validating the effectiveness of the method in the CLEF 2023 CheckThat! challenge, surpassing reported results in F1 score and official MAE metrics.
3. **Release of a Large-Scale Dataset**: Publishing the largest annotated dataset containing standard political bias and factual reporting labels.

### Background and Motivation

With the proliferation of the internet, the speed of news information dissemination has greatly increased, but it has also brought about issues of misinformation and biased reporting. Traditional manual evaluation methods are difficult to cope with the rapidly increasing news content, thus automated methods are needed to assess the reliability and bias of news media. The paper provides a new solution by modeling the longitudinal network interactions between news media.

### Methods and Experiments

1. **Constructing a News Media Graph**: Constructing a weighted directed graph from the internet, where nodes represent news media and edges represent hyperlink relationships between media.
2. **Applying Reinforcement Learning Strategies**: Proposing four reinforcement learning strategies (F-property, P-property, FP-property, I-property) to infer the reliability and bias of news media.
3. **Datasets**: Using two datasets, one is the self-built MBFC dataset, and the other is the CLEF 2023 CheckThat! challenge dataset.
4. **Experimental Results**:
- In terms of factual reporting, the I-Factuality strategy performed the best, with an F1 score of 87.99.
- In terms of political bias, the I-Political strategy performed the best, with an F1 score of 77.77, and achieved the best MAE and F1 scores in the CLEF 2023 CheckThat! challenge.

### Conclusion

The paper demonstrates the effectiveness and robustness of network interaction modeling-based news media bias assessment by extending existing news media reliability assessment methods. Experimental results show that the proposed method significantly outperforms baseline methods in the evaluation of factual reporting and political bias, achieving new state-of-the-art results in the CLEF 2023 CheckThat! challenge. Additionally, the paper releases a large-scale annotated dataset, providing valuable resources for future research.