Abstract:Traditional models grounded in first principles often struggle with accuracy as the system's complexity increases. Conversely, machine learning approaches, while powerful, face challenges in interpretability and in handling physical constraints. Efforts to combine these models often often stumble upon difficulties in finding a balance between accuracy and complexity. To address these issues, we propose a comprehensive framework based on a "mixture of experts" rationale. This approach enables the data-based fusion of diverse local models, leveraging the full potential of first-principle-based priors. Our solution allows independent training of experts, drawing on techniques from both machine learning and system identification, and it supports both collaborative and competitive learning paradigms. To enhance interpretability, we penalize abrupt variations in the expert's combination. Experimental results validate the effectiveness of our approach in producing an interpretable combination of models closely resembling the target phenomena.

What problem does this paper attempt to address?

The paper attempts to address the issue where traditional first-principles-based models lose accuracy as system complexity increases, while machine learning methods, though powerful, face challenges in interpretability and handling physical constraints. To tackle these problems, the authors propose a "mixture of experts" approach, aiming to integrate different local models in a data-driven manner, fully leveraging first-principles-based prior knowledge. Specifically, this method allows for the independent training of expert models and supports collaborative and competitive learning paradigms, while enhancing interpretability by penalizing abrupt changes in the expert combinations. ### Main Objectives: 1. **Improve model accuracy and interpretability**: By combining first-principles-based models and machine learning models, the approach maintains predictive capability while enhancing interpretability. 2. **Balance accuracy and complexity**: Find a balance between model accuracy and complexity to avoid computational burden and lack of interpretability caused by overly complex models. 3. **Integrate gray-box and black-box models**: Through the "mixture of experts" framework, effectively combine gray-box models (partially based on first principles) and black-box models (purely data-driven) to form an interpretable hybrid model. ### Key Features of the Solution: - **Independent training of expert models**: Each expert model can be trained independently, utilizing techniques from machine learning and system identification. - **Collaborative and competitive learning**: Support collaboration and competition among expert models to better describe different behaviors of the system. - **Penalty for abrupt changes**: Introduce penalty terms to reduce abrupt changes in expert combinations over consecutive time steps, thereby enhancing model smoothness and interpretability. - **Statistical interpretation**: Provide a statistical interpretation of the cost function, clarifying the role of hyperparameters, making the model more transparent and understandable. ### Experimental Validation: Experimental results validate the effectiveness of the proposed method, demonstrating that it can generate interpretable model combinations that closely resemble the target phenomena. In summary, the paper proposes a new "mixture of experts" framework that resolves the trade-off between accuracy and interpretability in traditional models for complex systems, providing an effective solution for data-driven modeling.

Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models

A Grey-Box Ensemble Model Exploiting Black-Box Accuracy and White-Box Intrinsic Interpretability

Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization

Incorporating Experts' Judgment into Machine Learning Models

FEAMOE: Fair, Explainable and Adaptive Mixture of Experts

FairMOE: counterfactually-fair mixture of experts with levels of interpretability

Dynamic Mixture of Experts Models for Online Prediction

Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts

Mixture of experts models for multilevel data: modelling framework and approximation theory

Implicit Mixture of Interpretable Experts for Global and Local Interpretability

Multi-Task Model Fusion with Mixture of Experts Structure

Biased Mixtures Of Experts: Enabling Computer Vision Inference Under Data Transfer Limitations

An Interpretable Probabilistic Approach for Demystifying Black-box Predictive Models

An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling

Benchmarking and survey of explanation methods for black box models

Explainability meets uncertainty quantification: Insights from feature-based model fusion on multimodal time series

Prediction Sets for High-Dimensional Mixture of Experts Models

Learning factored representations in a deep mixture of experts

Functional Mixtures-of-Experts

Combining Classifiers and Learning Mixture-of-Experts