Abstract:Decision Trees (DTs) are a class of supervised learning models that are widely used for both classification and regression applications. They are well-known for their interpretability and robustness, which have led them to remain popular even 60 years after they were first proposed. However, because traditional tree algorithms use greedy methods that are prone to suboptimality, several works have explored the usage of evolutionary algorithms instead. Although these algorithms are often reported to outperform the traditional greedy approach, their computational cost is much higher, since the evolutionary component requires a large number (millions or billions) of function evaluations in order to produce a single tree. Aiming to reduce this computational cost, in this work we propose an encoding that allows the training and evaluation of DTs using only matrix operations. The proposed procedure is shown to be much faster than the traditional tree implementation for complete trees with depths ranging from 2 to 6, and for datasets ranging in size from 100 to 100,000 observations. In particular, the results show speedups of nearly up to 20 times, especially when the dataset is large and the desired tree is small enough to be interpretable. The proposed procedure also benefits from GPU parallelization, although it is still highly performing without it. Furthermore, we propose an evolutionary algorithm, called Coral Reef Optimization for Decision Trees (CRO-DT), that integrates this encoding with a pre-existing ensemble algorithm to evolve better univariate trees. The results obtained show that the proposed CRO-DT is competitive with traditional and modern tree algorithms, consistently producing models of good quality across 14 tested UCI Datasets. We conclude that for most relevant situations, the proposed matrix encoding provides significant speedups over the traditional implementation, and also may serve as a basis for high quality evolutionary DT algorithms.

Optimization of Decision Tree Evaluation Using SIMD Instructions

Optimization of Oblivious Decision Tree Ensembles Evaluation for CPU

Register Your Forests: Decision Tree Ensemble Optimization by Explicit CPU Register Allocation

SIMD-ified R-tree Query Processing and Optimization

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

Efficient evolution of decision trees via fully matrix-based fitness evaluation

Efficient Realization of Decision Trees for Real-Time Inference

Vectorization of Gradient Boosting of Decision Trees Prediction in the CatBoost Library for RISC-V Processors

Single MCMC Chain Parallelisation on Decision Trees

A Comparison of Decision Forest Inference Platforms from A Database Perspective

Towards Efficient and Scalable Acceleration of Online Decision Tree Learning on FPGA

Enterprise-Scale Search: Accelerating Inference for Sparse Extreme Multi-Label Ranking Trees

Case Study: Optimization Methods With TVM Hybrid-OP on RISC-V Packed SIMD

Learn Smart with Less: Building Better Online Decision Trees with Fewer Training Examples

Dynamic Decision Tree Ensembles for Energy-Efficient Inference on IoT Edge Nodes

Taiga: Performance Optimization of the C4.5 Decision Tree Construction Algorithm

Optimizing Tensor Computation Graphs with Equality Saturation and Monte Carlo Tree Search

A Hardware-Efficient ADMM-Based SVM Training Algorithm for Edge Computing

SIMD$^2$: A Generalized Matrix Instruction Set for Accelerating Tensor Computation beyond GEMM

SIMD Compression and the Intersection of Sorted Integers

An Optimization and Auto-Tuning Method for Scale-Free Graph Algorithms on SIMD Architectures