Regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features

Florian Pargent,Florian Pfisterer,Janek Thomas,Bernd Bischl

DOI: https://doi.org/10.1007/s00180-022-01207-6

2022-03-04

Abstract:Since most machine learning (ML) algorithms are designed for numerical inputs, efficiently encoding categorical variables is a crucial aspect in data analysis. A common problem are high cardinality features, i.e. unordered categorical predictor variables with a high number of levels. We study techniques that yield numeric representations of categorical variables which can then be used in subsequent ML applications. We focus on the impact of these techniques on a subsequent algorithm's predictive performance, and -- if possible -- derive best practices on when to use which technique. We conducted a large-scale benchmark experiment, where we compared different encoding strategies together with five ML algorithms (lasso, random forest, gradient boosting, k-nearest neighbors, support vector machine) using datasets from regression, binary- and multiclass- classification settings. In our study, regularized versions of target encoding (i.e. using target predictions based on the feature levels in the training set as a new numerical feature) consistently provided the best results. Traditionally widely used encodings that make unreasonable assumptions to map levels to integers (e.g. integer encoding) or to reduce the number of levels (possibly based on target information, e.g. leaf encoding) before creating binary indicator variables (one-hot or dummy encoding) were not as effective in comparison.

Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to efficiently encode features with high cardinality (i.e., categorical variables with a large number of distinct levels but no natural order) in supervised machine learning. Specifically, the authors study the impact of different encoding techniques on the prediction performance of subsequent machine - learning algorithms and attempt to determine best practices to guide when to use which technique. The paper focuses particularly on Regularized Target Encoding and compares it with traditional encoding methods, such as integer encoding, frequency encoding, hash encoding, leaf encoding, impact encoding, and generalized linear mixed - model encoding. Through large - scale benchmark experiments, the authors evaluate the performance of these encoding techniques in regression, binary - classification, and multi - classification settings, aiming to provide effective solutions for handling high - cardinality categorical features.