Abstract:Problem definition: Platforms critically rely on rating systems to learn the quality of market participants. In practice, however, ratings are often highly inflated and therefore, not very informative. In this paper, we first investigate whether the platform can obtain less inflated, more informative ratings by altering the meaning and relative importance of the levels in the rating system. Second, we seek a principled approach for the platform to make these choices in the design of the rating system. Academic/practical relevance: Platforms critically rely on rating systems to learn the quality of market participants, and so, ensuring these ratings are informative is of first-order importance. Methodology: We analyze the results of a randomized, controlled trial on an online labor market in which an additional question was added to the feedback form. Between treatment conditions, we vary the question phrasing and answer choices; in particular, the treatment conditions include several positive-skewed verbal rating scales with descriptive phrases or adjectives providing specific interpretation for each rating level. We then develop a model-based framework to compare and select among rating system designs and apply this framework to the data obtained from the online labor market test. Results: Our test reveals that current inflationary norms can be countered by reanchoring the meaning of the levels of the rating system. In particular, positive-skewed verbal rating scales yield substantially deflated rating distributions that are much more informative about seller quality. Further, we demonstrate that our model-based framework for scale design and optimization can identify the most informative rating system and substantially improve the quality of information obtained over baseline designs. Managerial implications: Our study illustrates that practical, informative rating systems can be designed and demonstrates how to compare and design them in a principled manner.

Incentivizing Agents through Ratings

Incentive Design in Peer Review: Rating and Repeated Endogenous Matching

Agency in Hierarchies: Middle Managers and Performance Evaluations

Rating Protocol Design for Extortion and Cooperation in the Crowdsourcing Contest Dilemma

Optimal Rating Design under Moral Hazard

Extortion and Cooperation in Rating Protocol Design for Competitive Crowdsourcing.

Designing Informative Rating Systems: Evidence from an Online Labor Market

Designing Socially-Optimal Rating Protocols for Crowdsourcing Contest Dilemma

Fairness Incentives for Myopic Agents

Bribery in Rating Systems: A Game-Theoretic Perspective

Maximizing Welfare with Incentive-Aware Evaluation Mechanisms

Optimal incentive contract with endogenous monitoring technology

Delegating Performance Evaluation

Incentive Ratio: A Game Theoretical Analysis of Market Equilibria

Characterization of Incentive Compatibility of an Ex-ante Constrained Player

RewardRating: A Mechanism Design Approach to Improve Rating Systems

Satisfaction Degree Of Consumers And Incentive Mechanism

The Economic Value of Ratings in App Market

Non-Market Allocation Mechanisms: Optimal Design and Investment Incentives

Incentive Mechanism Design Aiming at Deflated Performance Manipulation in Retail Firms: Based on the Ratchet Effect and the Reputation Effect

Incentivizing Information Acquisition