Leaf-FM: A Learnable Feature Generation Factorization Machine for Click-Through Rate Prediction

Qingyun She,Zhiqiang Wang,Junlin Zhang

DOI: https://doi.org/10.48550/arXiv.2107.12024

2021-07-26

Abstract:Click-through rate (CTR) prediction plays important role in personalized advertising and recommender systems. Though many models have been proposed such as FM, FFM and DeepFM in recent years, feature engineering is still a very important way to improve the model performance in many applications because using raw features can rarely lead to optimal results. For example, the continuous features are usually transformed to the power forms by adding a new feature to allow it to easily form non-linear functions of the feature. However, this kind of feature engineering heavily relies on peoples experience and it is both time consuming and labor consuming. On the other side, concise CTR model with both fast online serving speed and good model performance is critical for many real life applications. In this paper, we propose LeafFM model based on FM to generate new features from the original feature embedding by learning the transformation functions automatically. We also design three concrete Leaf-FM models according to the different strategies of combing the original and the generated features. Extensive experiments are conducted on three real-world datasets and the results show Leaf-FM model outperforms standard FMs by a large margin. Compared with FFMs, Leaf-FM can achieve significantly better performance with much less parameters. In Avazu and Malware dataset, add version Leaf-FM achieves comparable performance with some deep learning based models such as DNN and AutoInt. As an improved FM model, Leaf-FM has the same computation complexity with FM in online serving phase and it means Leaf-FM is applicable in many industry applications because of its better performance and high computation efficiency.

Information Retrieval,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve model performance by automatically generating new features in click - through rate (CTR) prediction while reducing the dependence on manual feature engineering. Specifically, the authors propose the Leaf - FM model. Based on the factorization machine (FM), this model automatically generates new features from the original feature embeddings, thereby enhancing the expressive ability of the model and improving the model performance while maintaining the online service speed.

### Background and Problem Description of the Paper

Click - through rate (CTR) prediction plays an important role in personalized advertising and recommendation systems. Although many models have been proposed in recent years, such as FM (Factorization Machine), FFM (Field - aware Factorization Machine) and DeepFM, feature engineering is still an important means to improve model performance. Using the original features usually makes it difficult to achieve the optimal results. For example, continuous features usually need to be transformed into power forms to form nonlinear functions. However, this kind of feature engineering is highly dependent on human experience and is both time - consuming and labor - intensive.

In addition, in practical applications, the production ranking system needs to process a large number of requests under strict latency constraints. Even for the simple DNN CTR model used by Facebook, a large amount of machine resources and additional work are required to meet these requirements. Therefore, a concise and high - performance CTR model is crucial for practical applications.

### Solutions

To address the above challenges, the authors propose the Leaf - FM model. The main contributions of the Leaf - FM model include:

1. **Feature Generation Network (FGNet)**: A new feature generation network is proposed, which can automatically generate new features from the original features, including continuous features and categorical features. FGNet can significantly reduce the feature engineering requirements in actual CTR tasks.
2. **Leaf - FM Model**: Three specific Leaf - FM models are designed, namely the additive version (LA - FM), the summation version (LS - FM) and the multiplication version (LP - FM). These models combine the original features and the generated features through different strategies.
3. **Experimental Verification**: Extensive experiments were carried out on three real - world datasets. The results show that the Leaf - FM model significantly outperforms the standard FM model on multiple datasets and has a performance comparable to that of the DNN model on some datasets.

### Model Details

#### 4.1 Feature Generation Network (FGNet)

FGNet generates new features from feature embeddings through two fully - connected layers (FC layers). The first FC layer is a dimension - expansion layer, and the second FC layer is used for dimension reduction. The generated new features have the same dimension as the original features. The specific formula of FGNet is as follows:

\[ g_i = F_{FGNet}(v_i, W_k) = \delta(W_{k2} \cdot \delta(W_{k1} \cdot v_i)) \]

where \(\delta\) is a nonlinear function, \(W_{k1} \in \mathbb{R}^{r \times d}\) and \(W_{k2} \in \mathbb{R}^{d \times r}\) are the parameters of the two FC layers, and \(r\) is the expansion ratio.

#### 4.2 Leaf - FM Model

The Leaf - FM model combines the original features and the generated features through different strategies. Specifically:

- **Additive Version (LA - FM)**:
\[ \hat{y}(x) = w_0 + \sum_{i = 1}^{m} w_i x_i + \sum_{i = 1}^{(u + 1)m} \sum_{j = i + 1}^{(u + 1)m} \langle v_i, v_j \rangle x_i x_j \]
- **Summation Version (LS - FM)**:
\[ F_s(v_i) = v_i + \sum_{j = 1}^{u} g_{ij} \]
\[ \hat{y}(x) = w_0 + \sum_{i = 1}^{m} w_i x_i + \sum_{i = 1}^{m} \sum_{j = i + 1}^{m} \langle