Abstract:The traditional viewpoint on Sparse Mixture of Experts (MoE) models is that instead of training a single large expert, which is computationally expensive, we can train many small experts. The hope is that if the total parameter count of the small experts equals that of the singular large expert, then we retain the representation power of the large expert while gaining computational tractability and promoting expert specialization. The recently introduced Soft MoE replaces the Sparse MoE's discrete routing mechanism with a differentiable gating function that smoothly mixes tokens. While this smooth gating function successfully mitigates the various training instabilities associated with Sparse MoE, it is unclear whether it induces implicit biases that affect Soft MoE's representation power or potential for expert specialization. We prove that Soft MoE with a single arbitrarily powerful expert cannot represent simple convex functions. This justifies that Soft MoE's success cannot be explained by the traditional viewpoint of many small experts collectively mimicking the representation power of a single large expert, and that multiple experts are actually necessary to achieve good representation power (even for a fixed total parameter count). Continuing along this line of investigation, we introduce a notion of expert specialization for Soft MoE, and while varying the number of experts yet fixing the total parameter count, we consider the following (computationally intractable) task. Given any input, how can we discover the expert subset that is specialized to predict this input's label? We empirically show that when there are many small experts, the architecture is implicitly biased in a fashion that allows us to efficiently approximate the specialized expert subset. Our method can be easily implemented to potentially reduce computation during inference.

FairMOE: counterfactually-fair mixture of experts with levels of interpretability

FEAMOE: Fair, Explainable and Adaptive Mixture of Experts

Counterfactual Fair Opportunity: Measuring Decision Model Fairness with Counterfactual Reasoning

FairerML: an Extensible Platform for Analysing, Visualising, and Mitigating Biases in Machine Learning [application Notes]

Explainable Fairness in Recommendation

Counterfactual Fairness by Combining Factual and Counterfactual Predictions

Fairness and Explainability: Bridging the Gap Towards Fair Model Explanations

FairerML: An Extensible Platform for Analysing, Visualising, and Mitigating Biases in Machine Learning

FairLay-ML: Intuitive Remedies for Unfairness in Data-Driven Social-Critical Algorithms

Constructing Fair Latent Space for Intersection of Fairness and Explainability

Implicit Mixture of Interpretable Experts for Global and Local Interpretability

Explainability for fair machine learning

Revealing Unfair Models by Mining Interpretable Evidence

Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts

Beyond Incompatibility: Trade-offs between Mutually Exclusive Fairness Criteria in Machine Learning and Law

Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models

Making Fair ML Software using Trustworthy Explanation

Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization

Wasserstein-based fairness interpretability framework for machine learning models

On the Interplay between Fairness and Explainability

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts