A Quantitative Study of Impact of Incentive to Quality of Software Reviews

Mingwei Tang,Zhiwei Xu,Yuhao Qin,Cui Su,Yi Zhu,Feifei Tao,Junhua Ding
DOI: https://doi.org/10.1109/dsa56465.2022.00016
2022-01-01
Abstract:Online reviews like product reviews are important references for a potential user to learn the basic information of a product. However, some reviewers were rewarded for writing the reviews, which may impact the objectiveness of the reviews. But on the other hand, the reviews written by reviewers who weren't rewarded could be in low quality. Research study already showed incentivized reviewers may give a higher overall score than non-incentivized reviewers (or called organic) do, but does that pattern also apply to review content? In this paper, a quantitative comparison study is conducted to investigate the differences between the incentivized reviews and organic reviews of software products. Four pairs of comparison including overall score, sentiment preference, correlation, and similarity are performed by using statistical and text mining methods. The results show there is no statistically significant difference between the incentive reviews and organic reviews except the sentiment of total, “Problems and Benefits” and “Summary” part of a review text. The results are unexpected since the reviews collected from the website for reviewing software product already filtered low quality reviews. It demonstrates that the incentivized action might not be necessary to produce biased reviews and it may be an effective way to attract more reviews since the website include more than 75% incentivized reviews. The paper also analyzed the possible reasons from the feature of reviewers' position in a company, a review's indicator, and reviewers' common actions. Based on the analysis, this study suggests that a potential user may pay attention to some quality dimensions of a review to mitigate the bias risk from the reviews.
What problem does this paper attempt to address?