Content Quality vs. Attention Allocation: An LLM-Based Case Study in Peer-to-peer Mental Health Networks

Teng Ye,Hanson Yan,Xuhuan Huang,Connor Grogan,Walter Yuan,Qiaozhu Mei,Matthew O. Jackson

2024-11-08

Abstract:With the rise of social media and peer-to-peer networks, users increasingly rely on crowdsourced responses for information and assistance. However, the mechanisms used to rank and promote responses often prioritize and end up biasing in favor of timeliness over quality, which may result in suboptimal support for help-seekers. We analyze millions of responses to mental health-related posts, utilizing large language models (LLMs) to assess the multi-dimensional quality of content, including relevance, empathy, and cultural alignment, among other aspects. Our findings reveal a mismatch between content quality and attention allocation: earlier responses - despite being relatively lower in quality - receive disproportionately high fractions of upvotes and visibility due to platform ranking algorithms. We demonstrate that the quality of the top-ranked responses could be improved by up to 39 percent, and even the simplest re-ranking strategy could significantly improve the quality of top responses, highlighting the need for more nuanced ranking mechanisms that prioritize both timeliness and content quality, especially emotional engagement in online mental health communities.

Social and Information Networks

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in social media and peer - to - peer networks, users are increasingly relying on crowdsourced responses to obtain information and help, but the mechanisms used to rank and promote responses often prioritize timeliness over quality, which may lead to sub - optimal support for those seeking help. Specifically, the paper analyzed millions of mental - health - related post responses and used large language models (LLMs) to evaluate the multi - dimensional quality of the content, including aspects such as relevance, empathy, and cultural adaptability. The study found that earlier responses, despite their lower quality, still received a disproportionately high number of votes and visibility due to the platform's ranking algorithm. The paper shows that through a simple re - ranking strategy, the quality of top - ranked responses can be significantly improved, emphasizing the need for a more refined ranking mechanism that takes into account both timeliness and content quality, especially the emotional engagement in online mental - health communities.