Abstract:A typical benchmark dataset for recommender system (RecSys) evaluation consists of user-item interactions generated on a platform within a time period. The interaction generation mechanism partially explains why a user interacts with ( e.g. , like, purchase, rate) an item, and the context of when a particular interaction happened. In this study, we conduct a meticulous analysis of the MovieLens dataset and explain the potential impact of using the dataset for evaluating recommendation algorithms. We make a few main findings from our analysis. First, there are significant differences in user interactions at the different stages when a user interacts with the MovieLens platform. The early interactions largely define the user portrait which affect the subsequent interactions. Second, user interactions are highly affected by the candidate movies that are recommended by the platform’s internal recommendation algorithm(s). Third, changing the order of user interactions makes it more difficult for sequential algorithms to capture the progressive interaction process. We further discuss the discrepancy between the interaction generation mechanism that is employed by the MovieLens system and that of typical real-world recommendation scenarios. That is, the MovieLens dataset records \(\langle user-MovieLens\rangle\) interactions, but not \(\langle user-movie\rangle\) interactions. All research papers using the MovieLens dataset model the \(\langle user-MovieLens\rangle\) rather than the \(\langle user-movie\rangle\) interactions, making their results less generalizable to many practical recommendation scenarios in real-world settings. In summary, the MovieLens platform demonstrates an efficient and effective way of collecting user preferences to address cold-starts. However, models that achieve excellent recommendation accuracy on the MovieLens dataset may not demonstrate superior performance in practice , for at least two kinds of differences: (i) the differences in the contexts of user-item interaction generation, and (ii) the differences in user knowledge about the item collections. While results on MovieLens can be useful as a reference, they should not be solely relied upon as the primary justification for the effectiveness of a recommendation system model.

Effect of User’s Judging Power on the Recommendation Performance

User Preference Quantity Versus Recommendation Performance: A Preliminary Study

A Novel Framework To Process The Quantity And Quality Of User Behavior Data In Recommender Systems

A Novel Social Recommendation Method Fusing User’s Social Status and Homophily Based on Matrix Factorization Techniques

Impact of Purchasing Power on User Rating Behavior and Purchasing Decision.

User Interest Dynamics on Personalized Recommendation

Hybrid Recommendation Model Based on Social Media and Social Networks

Improving accuracy and diversity of personalized recommendation through power law adjustments of user similarities.

SOCIAL INTEREST FOR USER SELECTING ITEMS IN RECOMMENDER SYSTEMS

Improving Accuracy and Diversity of Personalized Recommendation Through Power Law Adjustments of User Similarities

A two-step Recommendation Algorithm via Iterative Local Least Squares

User-oriented Fairness in Recommendation

Our Model Achieves Excellent Performance on MovieLens: What Does It Mean?

Rating Prediction by Exploring User’s Preference and Sentiment

Bigger data set, better personalized recommendation performance?

Rating Prediction in Recommender Systems Based on User Behavior Probability and Complex Network Modeling

Can Dissimilar Users Contribute to Accuracy and Diversity of Personalized Recommendation?

The effect of rating variance on personalized recommendation

Investigating the Temporal Effect of User Preferences with Application in Movie Recommendation.

Exploring Users' Internal Influence from Reviews for Social Recommendation

Analytical and Empirical Study of Herding Effects in Recommendation Systems