Semi-sparse Algorithm Based on Multi-Layer Optimization for Recommender System

Hu Guan,Huakang Li,Cheng-Zhong Xu,Minyi Guo
DOI: https://doi.org/10.1007/s11227-012-0830-6
2012-01-01
Abstract:Similarity among vectors is basic knowledge required to carry out recommendation and classification in recommender systems, which support personalized recommendation during online interactions. In this paper, we propose a Semi-sparse Algorithm based on Multi-layer Optimization to speed up the Pearson Correlation Coefficient, which is conventionally used in obtaining similarity among sparse vectors. In accelerating the batch of similarity-comparisons within one thread, the semi-sparse algorithm spares out over-reduplicated accesses and judgements on the selected sparse vector by making this vector dense locally. Moreover, a reduce-vector is proposed to restrict using locks on critical resources in the thread-pool, which is wrapped with Pthreads on a multi-core node to improve parallelism. Furthermore, among processes in our framework, a shared zip file is read to cut down messages within the Message Passing Interface package. Evaluation shows that the optimized multi-layer framework achieves a brilliant speedup on three benchmarks, Netflix, MovieLens and MovieLen1600.
What problem does this paper attempt to address?