Self-Adapting Machine Learning-based Systems via a Probabilistic Model Checking Framework

Maria Casimiro,Diogo Soares,David Garlan,Luís Rodrigues,Paolo Romano
DOI: https://doi.org/10.1145/3648682
2024-03-07
ACM Transactions on Autonomous and Adaptive Systems
Abstract:This paper focuses on the problem of optimizing system utility of Machine-Learning (ML) based systems in the presence of ML mispredictions. This is achieved via the use of self-adaptive systems and through the execution of adaptation tactics, such as model retraining , which operate at the level of individual ML components. To address this problem, we propose a probabilistic modeling framework that reasons about the cost/benefit trade-offs associated with adapting ML components. The key idea of the proposed approach is to decouple the problems of estimating (i) the expected performance improvement after adaptation and (ii) the impact of ML adaptation on overall system utility. We apply the proposed framework to engineer a self-adaptive ML-based fraud-detection system, which we evaluate using a publicly-available, real fraud detection data-set. We initially consider a scenario in which information on model’s quality is immediately available. Next we relax this assumption by integrating (and extending) state-of-the-art techniques for estimating model’s quality in the proposed framework. We show that by predicting the system utility stemming from retraining a ML component, the probabilistic model checker can generate adaptation strategies that are significantly closer to the optimal, as compared against baselines such as periodic or reactive retraining.
computer science, information systems, artificial intelligence, theory & methods
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to optimize the system utility of machine - learning - based systems in the presence of machine learning (ML) prediction errors. Specifically, the paper focuses on how to optimize the performance of these systems through adaptive systems and implementing adaptation strategies (such as model retraining) to reduce the loss of system utility due to ML prediction errors. To achieve this goal, the authors propose an approach based on a probabilistic modeling framework, which can evaluate the cost - benefit trade - off of adapting ML components. The key idea is to decouple the two problems of estimating (i) the expected performance improvement after adaptation and (ii) the impact of ML adaptation on the overall system utility. In this way, the proposed framework can generate near - optimal adaptation strategies more effectively, especially when compared with baseline strategies such as periodic or reactive retraining. The paper also specifically mentions applying this framework to build an adaptive machine - learning - based fraud detection system and evaluates it using publicly available real - world fraud detection datasets. In addition, the paper also explores how to use advanced techniques (such as ATC and CB - ATC) to estimate the quality of the model in the case of label delay (that is, the true label is available only after a period of time), thereby further enhancing the robustness and adaptability of the system.