$L_0$ Regularization of Field-Aware Factorization Machine through Ising Model

Yasuharu Okamoto

2024-03-04

Abstract:We examined the use of the Ising model as an $L_0$ regularization method for field-aware factorization machines (FFM). This approach improves generalization performance and has the advantage of simultaneously determining the best feature combinations for each of several groups. We can deepen the interpretation and understanding of the model from the similarities and differences in the features selected in each group.

Machine Learning,Disordered Systems and Neural Networks

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to introduce the Ising model for L0 regularization in order to improve the generalization performance of the Field - aware Factorization Machine (FFM) and determine the best feature combinations in multiple groups simultaneously. Specifically, the paper aims to:

1. **Solve the over - fitting problem**: Due to the large number of feature cross - terms in FFM, the number of features increases significantly, and the over - fitting problem is likely to occur. The common L1 and L2 regularization methods have limited effectiveness in some cases, while L0 regularization can directly control the number of features included in the model, thereby effectively preventing over - fitting.

2. **Optimize feature selection**: By performing L0 regularization with the Ising model, the optimal feature combinations can be selected simultaneously in different groups (such as age groups). This not only improves the interpretability of the model but also avoids the multicollinearity problem common in L1 regularization, that is, only one arbitrary feature is selected among a set of strongly correlated features.

3. **Improve model interpretability**: By comparing the features selected in different groups, the model can be understood more in - depth. For example, in the diabetes patient dataset, the feature selection results of different age groups can help researchers better understand the disease progression characteristics of patients in each age group.

4. **Verify the effectiveness of the method**: By converting quantitative variables into categorical variables and applying the Ising model for optimization, the paper verifies the effectiveness of this method when dealing with actual datasets (such as the diabetes patient dataset). The results show that the generalization performance of this method is comparable to or better than that of traditional methods such as Random Forest (RF) and Elastic Net (EN).

### Specific problem description

The paper uses a diabetes patient dataset to study the relationships between features such as age, gender, BMI, and blood pressure levels and the disease progression of patients one year later. By converting quantitative variables (such as age, BMI, etc.) into categorical variables and applying the Ising model for L0 regularization, the paper explores how to improve the generalization performance while maintaining the descriptive ability of the model.

### Main contributions

- Proposed an L0 regularization method based on the Ising model for feature selection in FFM.
- Proved that converting quantitative variables into categorical variables does not significantly reduce the descriptive ability of the model.
- Verified the effectiveness of this method on the diabetes patient dataset through experiments, showing a generalization performance comparable to or better than that of traditional methods.
- Provided a detailed analysis of the feature selection results in different groups, which is helpful for better understanding and interpreting the model.

### Summary of mathematical formulas

- FFM prediction value formula:
\[
\hat{y}_i = w_0+\sum_{l = 1}^{D}w_lx_{il}+\sum_{l_2>l_1}w_{l_1l_2}x_{il_1}x_{il_2}
\]

- Approximate expression of the second - order parameter:
\[
w_{l_1l_2}\approx\sum_{m = 1}^{K}v_{l_1f(l_2)m}v_{l_2f(l_1)m}
\]

- Objective function:
\[
\mathcal{F}=\sum_i\left(y_i - w_0-\sum_{s,g}\alpha_sX_{is}p_{ig}q_{sg}\right)^2+A\sum_g\left(\sum_sq_{sg}-M_f\right)^2
\]

where \(q_{sg}\) is a binary variable optimized by the Ising model, indicating whether the \(g\) - th group has selected the \(s\) - th extended feature; \(M_f\) is the number of features selected in each group; \(A\) is the strength hyperparameter of the constraint term.