Can ChatGPT Predict Article Retraction Based on Twitter Mentions?

Er-Te Zheng,Hui-Zhen Fu,Zhichao Fang

DOI: https://doi.org/10.48550/arxiv.2403.16851

2024-01-01

Abstract:Detecting problematic research articles timely is a vital task. This studyexplores whether Twitter mentions of retracted articles can signal potentialproblems with the articles prior to retraction, thereby playing a role inpredicting future retraction of problematic articles. A dataset comprising3,505 retracted articles and their associated Twitter mentions is analyzed,alongside 3,505 non-retracted articles with similar characteristics obtainedusing the Coarsened Exact Matching method. The effectiveness of Twittermentions in predicting article retraction is evaluated by four predictionmethods, including manual labelling, keyword identification, machine learningmodels, and ChatGPT. Manual labelling results indicate that there are indeedretracted articles with their Twitter mentions containing recognizable evidencesignaling problems before retraction, although they represent only a limitedshare of all retracted articles with Twitter mention data (approximately 16Using the manual labelling results as the baseline, ChatGPT demonstratessuperior performance compared to other methods, implying its potential inassisting human judgment for predicting article retraction. This study uncoversboth the potential and limitation of social media events as an early warningsystem for article retraction, shedding light on a potential application ofgenerative artificial intelligence in promoting research integrity.