Abstract:In matters of great importance that have financial, medical, social, or other implications, we often seek a second opinion before making a decision, sometimes a third, and sometimes many more. In doing so, we weigh the individual opinions, and combine them through some thought process to reach a final decision that is presumably the most informed one. The process of consulting "several experts" before making a final decision is perhaps second nature to us; yet, the extensive benefits of such a process in automated decision making applications have only recently been discovered by computational intelligence community. Also known under various other names, such as multiple classifier systems, committee of classifiers, or mixture of experts, ensemble based systems have shown to produce favorable results compared to those of single-expert systems for a broad range of applications and under a variety of scenarios. Design, implementation and application of such systems are the main topics of this article. Specifically, this paper reviews conditions under which ensemble based systems may be more beneficial than their single classifier counterparts, algorithms for generating individual components of the ensemble systems, and various procedures through which the individual classifiers can be combined. We discuss popular ensemble based algorithms, such as bagging, boosting, AdaBoost, stacked generalization, and hierarchical mixture of experts; as well as commonly used combination rules, including algebraic combination of outputs, voting based techniques, behavior knowledge space, and decision templates. Finally, we look at current and future research directions for novel applications of ensemble systems. Such applications include incremental learning, data fusion, feature selection, learning with missing features, confidence estimation, and error correcting output codes; all areas in which ensemble systems have shown great promise

Human-AI Ensembles: When Can They Work?

Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork

Human–AI collaborative decision-making as an organization design problem

Three Challenges for AI-Assisted Decision-Making

Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making

Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making

Human-AI Learning Performance in Multi-Armed Bandits

On the Effect of Information Asymmetry in Human-AI Teams

A2C: A Modular Multi-stage Collaborative Decision Framework for Human-AI Teams

AI-teaming: Redefining collaboration in the digital era

Human-machine collaboration in managerial decision making

Ensemble based systems in decision making

When combinations of humans and AI are useful: A systematic review and meta-analysis

Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence

Towards Effective Human-AI Decision-Making: The Role of Human Learning in Appropriate Reliance on AI Advice

The Utility of Explainable AI in Ad Hoc Human-Machine Teaming

How Being Outvoted by AI Teammates Impacts Human-AI Collaboration

Deep Ensembles Work, But Are They Necessary?

Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making

Artificial prediction markets present a novel opportunity for human-AI collaboration

Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction