Low Rank Field-Weighted Factorization Machines for Low Latency Item Recommendation

Alex Shtoff,Michael Viderman,Naama Haramaty-Krasne,Oren Somekh,Ariel Raviv,Tularam Ban

2024-07-22

Abstract:Factorization machine (FM) variants are widely used in recommendation systems that operate under strict throughput and latency requirements, such as online advertising systems. FMs are known both due to their ability to model pairwise feature interactions while being resilient to data sparsity, and their computational graphs that facilitate fast inference and training. Moreover, when items are ranked as a part of a query for each incoming user, these graphs facilitate computing the portion stemming from the user and context fields only once per query. Consequently, in terms of inference cost, the number of user or context fields is practically unlimited. More advanced FM variants, such as FwFM, provide better accuracy by learning a representation of field-wise interactions, but require computing all pairwise interaction terms explicitly. The computational cost during inference is proportional to the square of the number of fields, including user, context, and item. When the number of fields is large, this is prohibitive in systems with strict latency constraints. To mitigate this caveat, heuristic pruning of low intensity field interactions is commonly used to accelerate inference. In this work we propose an alternative to the pruning heuristic in FwFMs using a diagonal plus symmetric low-rank decomposition. Our technique reduces the computational cost of inference, by allowing it to be proportional to the number of item fields only. Using a set of experiments on real-world datasets, we show that aggressive rank reduction outperforms similarly aggressive pruning, both in terms of accuracy and item recommendation speed. We corroborate our claim of faster inference experimentally, both via a synthetic test, and by having deployed our solution to a major online advertising system. The code to reproduce our experimental results is at <a class="link-external link-https" href="https://github.com/michaelviderman/pytorch-fm/tree/dev" rel="external noopener nofollow">this https URL</a>.

Information Retrieval,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the high latency issue encountered when using Field-weighted Factorization Machines (FwFM) in recommendation systems. Specifically, FwFM improves recommendation accuracy by introducing a trainable symmetric field interaction matrix, but this improvement comes at the cost of significantly increased computational complexity, especially in scenarios requiring real-time processing of a large number of items. The paper proposes a method based on diagonal plus low-rank decomposition to reduce the computational cost of FwFM, allowing it to meet strict latency requirements while maintaining high recommendation accuracy.

### Main Issues
1. **High Computational Complexity**: The traditional FwFM model has a computational complexity proportional to the square of the number of fields when making item recommendations, leading to high latency in large-scale, real-time recommendation systems.
2. **Limitations of Model Pruning**: Although pruning techniques can reduce computational load, they often sacrifice model accuracy.

### Solution
The paper proposes a new model called DPLR-FwFM, which addresses the above issues through the following methods:
1. **Diagonal Plus Low-Rank Decomposition**: Decomposes the field interaction matrix \( \mathbf{R} \) in FwFM into the sum of a diagonal matrix and a low-rank matrix, i.e., \( \mathbf{R} = \mathbf{U}^T \text{diag}(\mathbf{e}) \mathbf{U} + \text{diag}(\mathbf{d}) \). This reduces the computational complexity from \( O(m^2 k) \) to \( O(\rho m k) \), where \( \rho \) is the rank of the low-rank part, typically much smaller than \( m \).
2. **Efficient Inference Algorithm**: By precomputing and caching context-related parts, the computational complexity for each item is further reduced to \( O(\rho |I| k) \), where \( |I| \) is the number of item fields.

### Experimental Results
The paper validates the effectiveness of DPLR-FwFM through experiments on multiple public datasets:
1. **Criteo Display Advertising Challenge Dataset**: Contains 39 fields and approximately 45 million samples.
2. **Avazu CTR Prediction Dataset**: Contains 33 fields and approximately 40 million samples.
3. **Movielens 1M Dataset**: Contains 8 fields and approximately 1 million samples.

The experimental results show that DPLR-FwFM not only significantly outperforms pruned FwFM in recommendation speed but also maintains a high level of accuracy.

### Main Contributions
1. **Improved Computational Efficiency**: The computational complexity of DPLR-FwFM is comparable to or only slightly higher than that of ordinary FM, but significantly lower than traditional FwFM, while retaining the higher accuracy brought by field information.
2. **Validation in Practical Applications**: Deploying DPLR-FwFM in a large-scale online advertising system shows that its prediction module's latency is significantly lower than that of the pruned FwFM model currently in production.
3. **Open Source Code**: Provides code to reproduce experimental results on public datasets and synthetic tests.

In summary, by proposing the DPLR-FwFM model, the paper successfully addresses the high computational complexity issue of FwFM in large-scale, low-latency recommendation systems, offering a new solution for optimizing the performance of real-time recommendation systems.