Abstract:Interactions among large number of entities is naturally high-dimensional and incomplete (HDI) in many big data related tasks. Behavioral characteristics of users are hidden in these interactions, hence, effective representation of the HDI data is a fundamental task for understanding user behaviors. Latent factor analysis (LFA) model has proven to be effective in representing HDI data. The performance of an LFA model relies heavily on its training process, which is a non-convex optimization. It has been proven that incorporating local curvature and preprocessing gradients during its training process can lead to superior performance compared to LFA models built with first-order family methods. However, with the escalation of data volume, the feasibility of second-order algorithms encounters challenges. To address this pivotal issue, this paper proposes a mini-block diagonal hessian-free (Mini-Hes) optimization for building an LFA model. It leverages the dominant diagonal blocks in the generalized Gauss-Newton matrix based on the analysis of the Hessian matrix of LFA model and serves as an intermediary strategy bridging the gap between first-order and second-order optimization methods. Experiment results indicate that, with Mini-Hes, the LFA model outperforms several state-of-the-art models in addressing missing data estimation task on multiple real HDI datasets from recommender system. (The source code of Mini-Hes is available at https://github.com/Goallow/Mini-Hes)

What problem does this paper attempt to address?

This paper attempts to solve the problem of how to efficiently optimize the Latent Factor Analysis (LFA) model in high - dimensional and incomplete (HDI) data. Specifically, the paper focuses on how to improve the performance of the LFA model by introducing local curvature information and pre - processing gradients on large - scale data sets, while overcoming the computational challenges faced by second - order optimization algorithms as the amount of data increases. ### Main problems: 1. **HDI data representation**: User - item interaction data are usually high - dimensional and incomplete. How to effectively represent these data to understand user behavior is a key task. 2. **Optimization method selection**: Existing LFA model optimization methods are divided into first - order and second - order methods. First - order methods such as gradient descent are simple but have limited effectiveness; second - order methods such as Hessian - free optimization have better performance but are too computationally expensive on large - scale data sets and difficult to parallelize. 3. **Balance between computational efficiency and accuracy**: How to improve computational efficiency while ensuring the prediction accuracy of the model, especially in the missing data estimation task. ### Proposed solutions: To solve the above problems, the paper proposes a new optimization method - Mini - Hes (mini - block diagonal Hessian - free optimization). The main features of this method include: - **Block - diagonal approximation**: Taking advantage of the fact that the Hessian matrix of the LFA model has block - diagonal dominant characteristics, the calculation is simplified by only retaining the main diagonal blocks, thereby reducing the storage and computational complexity. - **Parallel computing**: Each block can be calculated independently, so the calculation of the Hessian - vector product can be accelerated by multi - threading technology, significantly reducing the calculation time. - **Combining curvature information**: By introducing curvature information (such as the generalized Gauss - Newton matrix), Mini - Hes can effectively cope with the challenges of large - scale data sets while maintaining high prediction accuracy. ### Experimental results: The experimental results show that Mini - Hes performs better than existing first - order and second - order optimization methods on multiple real - world HDI data sets. In particular, in the missing data estimation task in the recommendation system, Mini - Hes not only performs excellently in prediction accuracy but also has a significant improvement in computational efficiency. In conclusion, this paper aims to solve the balance problem between computational efficiency and accuracy in the optimization of the LFA model under large - scale HDI data by proposing the Mini - Hes optimization method, thereby providing more effective solutions for applications such as recommendation systems.

Mini-Hes: A Parallelizable Second-order Latent Factor Analysis Model

A Multilayered-and-Randomized Latent Factor Model for High-Dimensional and Sparse Matrices

Hyper-parameter-evolutionary latent factor analysis for high-dimensional and sparse data from recommender systems

A Nonlinear PID-Incorporated Adaptive Stochastic Gradient Descent Algorithm for Latent Factor Analysis

Fast Latent Factor Analysis via a Fuzzy PID-Incorporated Stochastic Gradient Descent Algorithm

Hierarchical Particle Swarm Optimization-incorporated Latent Factor Analysis for Large-Scale Incomplete Matrices

An ADRC-Incorporated Stochastic Gradient Descent Algorithm for Latent Factor Analysis

A Dynamic Linear Bias Incorporation Scheme for Nonnegative Latent Factor Analysis

A proportional-integral-derivative-incorporated stochastic gradient descent-based latent factor analysis model

Adaptive Divergence-based Non-negative Latent Factor Analysis

Discrete Social Recommendation.

A Differential Evolution-Enhanced Latent Factor Analysis Model for High-dimensional and Sparse Data

Useful Compact Representations for Data-Fitting

Large-scale and Scalable Latent Factor Analysis via Distributed Alternative Stochastic Gradient Descent for Recommender Systems

A fast self-adaptive intuitionistic fuzzy latent factor model

NeuLFT: A Novel Approach to Nonlinear Canonical Polyadic Decomposition on High-Dimensional Incomplete Tensors

An L1-and-L2-Norm-Oriented Latent Factor Model for Recommender Systems

PSLF: A PID Controller-incorporated Second-order Latent Factor Analysis Model for Recommender System

An Alternating-Direction-Method of Multipliers-Incorporated Approach to Symmetric Non-Negative Latent Factor Analysis

An Unconstrained Symmetric Nonnegative Latent Factor Analysis for Large-scale Undirected Weighted Networks

Statistical Inference for Large-dimensional Matrix Factor Model from Least Squares and Huber Loss Points of View