Abstract:In machine learning (ML), ensemble methods such as bagging, boosting, and stacking are widely-established approaches that regularly achieve top-notch predictive performance. Stacking (also called "stacked generalization") is an ensemble method that combines heterogeneous base models, arranged in at least one layer, and then employs another metamodel to summarize the predictions of those models. Although it may be a highly-effective approach for increasing the predictive performance of ML, generating a stack of models from scratch can be a cumbersome trial-and-error process. This challenge stems from the enormous space of available solutions, with different sets of data instances and features that could be used for training, several algorithms to choose from, and instantiations of these algorithms using diverse parameters (i.e., models) that perform differently according to various metrics. In this work, we present a knowledge generation model, which supports ensemble learning with the use of visualization, and a visual analytics system for stacked generalization. Our system, StackGenVis, assists users in dynamically adapting performance metrics, managing data instances, selecting the most important features for a given data set, choosing a set of top-performant and diverse algorithms, and measuring the predictive performance. In consequence, our proposed tool helps users to decide between distinct models and to reduce the complexity of the resulting stack by removing overpromising and underperforming models. The applicability and effectiveness of StackGenVis are demonstrated with two use cases: a real-world healthcare data set and a collection of data related to sentiment/stance detection in texts. Finally, the tool has been evaluated through interviews with three ML experts.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on how to simplify and optimize the construction process of stacking ensemble learning, specifically including the following points: 1. **Avoid trial - and - error methods and improve efficiency**: The paper attempts to solve the cumbersome trial - and - error process when building a stacking ensemble model from scratch. Due to the existence of a large number of combinations such as data instances, feature selection, algorithm selection, and parameter configuration, building an efficient stacking ensemble model often requires a large number of experiments and adjustments. 2. **Select the best algorithm and model combination**: How to select the best algorithm and model combination when designing a stacking ensemble model is a major challenge. The performance of different algorithms and models will vary according to different performance metrics, so an effective method is needed to evaluate and select the most appropriate combination. 3. **Monitor and control the training process and maintain trust**: During the process of training a stacking ensemble model, how to monitor the entire process and ensure the credibility of the results is also a key issue. This involves continuous evaluation and adjustment of the model performance to ensure the quality and reliability of the final model. 4. **Select suitable performance verification metrics**: Different datasets may require different performance metrics for evaluation. The paper explores how to select the most appropriate performance metrics for a specific dataset and combine these metrics to obtain more comprehensive evaluation results. To address these problems, the paper proposes a visual analysis system named StackGenVis. This system helps users solve the above problems through the following aspects: - **Dynamically adjust performance metrics**: Users can select and adjust performance metrics according to task requirements, thereby better guiding model selection and optimization. - **Manage data instances and feature selection**: Provide tools to help users select the most important features and process data instances to improve model performance. - **Explore and select top - performing and diverse algorithms**: Through the visual interface, users can compare the performance of different algorithms and select the most suitable algorithm. - **Measure prediction performance**: Provide multiple ways to evaluate the prediction performance of the model, helping users decide which models should be retained in the final stacking ensemble. - **Reduce complexity**: Simplify the final stacking ensemble structure by removing poorly performing or redundant models. Through these functions, StackGenVis aims to help users build and optimize stacking ensemble models more efficiently, while improving the interpretability and credibility of the models.

StackGenVis: Alignment of Data, Algorithms, and Models for Stacking Ensemble Learning Using Performance Metrics

MetaStackVis: Visually-Assisted Performance Evaluation of Metamodels

A Generalized Stacking for Implementing Ensembles of Gradient Boosting Machines

Enhancing binary classification: A new stacking method via leveraging computational geometry

Stacking and stability

Issues in Stacked Generalization

Stacking Ensemble Technique for Classifying Breast Cancer

Cost-Sensitive Stacking: an Empirical Evaluation

Feature-Weighted Linear Stacking

HPOSS: A hierarchical portfolio optimization stacking strategy to reduce the generalization error of ensembles of models

View selection in multi-view stacking: choosing the meta-learner

Systematic Ensemble Learning for Regression

Effective training-time stacking for ensembling of deep neural networks

Stacked Generalization: when Does It Work?

Exploration of the Stacking Ensemble Machine Learning Algorithm for Cheating Detection in Large-Scale Assessment

ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking

A Stack-Centric Processing Model for Iterative Processing

Evaluating StackingC and ensemble models for enhanced lithological classification in geological mapping

pystacked: Stacking generalization and machine learning in Stata

SIMstack: A Generative Shape and Instance Model for Unordered Object Stacks

Dynamic Stacked Generalization for Node Classification on Networks