Evaluation Of Recommendation System On A Movie Dataset
Ronit Malik
DOI: https://doi.org/10.2139/ssrn.3791837
2020-01-01
SSRN Electronic Journal
Abstract:This thesis provides an evaluation of recommendation system on a movie dataset and aims to give recommendation to its users through user ratings and also create a typical user profile using implicit data by using recommendation system techniques like content – based and collaborative – filtering.The literature review explains the fundamentals of recommendation system and also provides an in-depth review for the algorithms used in recommendation systems. It outlines content- based and collaborative filtering techniques used for building the recommendation system, the content-based filtering techniques focuses on providing recommendation using TF-IDF (Term frequency – inverse document frequency) which calculates the frequency of a given term in the dataset and plots the term frequency in a N-dimensional space from which cosine distance is calculated between the given vectors to determine the similarity between items and users. The closer the distance or angle between the two given vector the higher the similarity percentage between them.On the other hand, collaborative filtering works on the assumption of ‘homophily’ which means that similar users like similar things, it is used for predicting the user ratings for the given item a user has not viewed, there are two methods to approach collaborative filtering; memory based, and model based. Memory- based collaborative filtering presents a user – item matrix (also known as a utility matrix) where typically a K- nearest neighbourhood framework is considered where top N recommendations are given to the user based on various similarity measures like; Cosine, Euclidean and Mahalanobis distance etc. Moreover, model – based collaborative filtering technique also creates a utility matrix from which various modelling techniques like; matrix factorisation and deep learning can be used to predict user ratings. Deep learning model leverages LSTM (Long short-term memory) model to predict user rating, whereas the matrix factorisation technique decomposes a given matrix. This thesis focuses on matrix factorisation techniques like singular value decomposition (SVD) and alternating least square (ALS), the singular value decomposition matrix works with explicit data which decomposes a given matrix M into three small metrices whereas the alternating least square works with implicit data that converts a given matrix R into two small metrics which iterate random number between them till it finds a number when multiplied together gives an approximate of matrix R. The literature outlines singular value decomposition method as a popular technique used in movie recommendation systems.This thesis follows an onion research methodology which ensures a structure within the thesis, moreover the thesis also follows a knowledge discovery in database (KDD) process which is an industry standard process used in end – to -end machine learning problems. In addition, python programming was used to build the recommendation system as python is a versatile software for machine learning modelling and is popular compared to R in the data science community.The models are evaluated using root mean square error (RSME) for singular value decomposition and alternating least square whereas the K-Nearest neighbour model is evaluated using precision@K and recall@K, these evaluation metrics for K-Nearest neighbour gives more weightage to precision@K, which shows at what precision or recall is the model able to recommend the top K-items, the K for this thesis is 10 which means that the K-NN model will calculate the precision and recall for recommending 10 items to users. These models are compared against Netflix “Cinematch software”, “BellKor’s pragmatic chaos” and “surprise Movielens 100K benchmark”, the model for K-nearest neighbour while using the cosine similarity measure performed better than K-nearest neighbour with Euclidean distance, the KNN model resulted in a precision of; 82% and 79% respectively. The 82% precision@10 shows that the KNN model is able to predict and give out 10 recommendations at a precision of 82%. The literature review also highlights that precision is often given more weightage compared to recall while evaluating recommendation systems. On the other hand, singular value decomposition model resulted in an 86% RSME which was in par with BellKor’s pragmatic chaos (~85% RSME) who won the Netflix competition, the model also outperformed the Netflix Cinematch system by ~9% and surprise Movielens benchmark by ~4% improvement.However, there are some limitations to recommendations system, these include the cold-start problem which usually arises when a new user or a new movie enters the catalogue, they both cause problems when predicting the user preference as lack of historical data leads to inaccurate recommendation for the user. The second limitation is the sparsity of data, this is often caused when modelling for singular value decomposition and alternating least square where user ratings are to be predicted. Insufficient user rating in the user-item matrix may lead to a sparse matrix, therefore to overcome this problem a user rating threshold was set for the MovieLens dataset at 0.5% which meant the a user should rate at least 0.5% of the movies to be considered for content – based and collaborative filtering modelling. Lastly, scalability is another limitation for recommendation systems as it becomes really difficult to scale the model, this is linked to the cold-start problem. As new user or movie enter the catalogue the models would need to be re-trained on the new data. Therefore, a solution to this is to have multiple systems running to increase computational power and system memory to re-train models faster.The thesis also came to a conclusion that singular value decomposition was a better performing model for recommendation system. However, limitation was also highlighted which showed that singular value decomposition model does not take into account timestamp, and also if more variables were presented in the MovieLens dataset it would give better recommendations. Lastly, it was also proposed that deep learning models will be leveraged from which recurring neural networks will be used that could provide better recommendation with more variables and timestamps.