Jacopo D'Ignazi,Andreas Kaltenbrunner,Yelena Mejova,Michele Tizzani,Kyriaki Kalimeri,Mariano Beiró,Pablo Aragón
Abstract:Over the last few years, content verification through reliable sources has become a fundamental need to combat disinformation. Here, we present a language-agnostic model designed to assess the reliability of sources across multiple language editions of Wikipedia. Utilizing editorial activity data, the model evaluates source reliability within different articles of varying controversiality such as Climate Change, COVID-19, History, Media, and Biology topics. Crafting features that express domain usage across articles, the model effectively predicts source reliability, achieving an F1 Macro score of approximately 0.80 for English and other high-resource languages. For mid-resource languages, we achieve 0.65 while the performance of low-resource languages varies; in all cases, the time the domain remains present in the articles (which we dub as permanence) is one of the most predictive features. We highlight the challenge of maintaining consistent model performance across languages of varying resource levels and demonstrate that adapting models from higher-resource languages can improve performance. This work contributes not only to Wikipedia's efforts in ensuring content verifiability but in ensuring reliability across diverse user-generated content in various language communities.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: in multilingual versions of Wikipedia, how to evaluate the reliability of information sources through a language - agnostic method, so as to deal with false information and ensure the integrity and reliability of knowledge.
Specifically, the authors developed a model that uses edit - activity data to evaluate the reliability of sources in different articles. This model can be applied in multiple language versions of Wikipedia and can handle articles on different controversial topics, such as climate change, COVID - 19, history, media, and biology, etc. The key features of the model include:
1. **Cross - language applicability**: This model does not depend on the characteristics of a specific language, so it can be used in different language versions of Wikipedia.
2. **Multi - topic coverage**: The model can be applied to multiple controversial topics to help identify reliable sources in different fields.
3. **Edit - activity data**: By analyzing edit - activity data, the model can capture the permanence of sources in articles and other relevant features, thereby predicting their reliability.
4. **Performance evaluation**: The model performs well in high - resource languages (such as English), with an F1 Macro score of approximately 0.80; it is 0.65 in medium - resource languages; and its performance declines in low - resource languages but still has some practicality.
In addition, the authors also explored the following research questions:
- **RQ1**: When evaluating the reliability of Wikipedia sources, which language - agnostic features are the most predictive? A case study was carried out taking the climate change topic as an example.
- **RQ2**: What is the relationship between the model performance and the topics and scales of Wikipedia language versions?
- **RQ3**: Can these models be adapted between different topics or languages?
In conclusion, this paper aims to help Wikipedia editors more effectively identify unreliable information sources by developing a language - agnostic model, especially in language versions with fewer resources, thereby improving the knowledge reliability and integrity of the entire platform.