Abstract:AIOps (Artificial Intelligence for IT Operations) solutions leverage the massive data produced during the operation of large-scale systems and machine learning models to assist software engineers in their system operations. As operation data produced in the field are constantly evolving due to factors such as the changing operational environment and user base, the models in AIOps solutions need to be constantly maintained after deployment. While prior works focus on innovative modeling techniques to improve the performance of AIOps models before releasing them into the field, when and how to update AIOps models remain an under-investigated topic. In this work, we performed a case study on three large-scale public operation data and empirically assessed five different types of model update strategies for supervised learning regarding their performance, updating cost, and stability. We observed that active model update strategies (e.g., periodical retraining, concept drift guided retraining, time-based model ensembles, and online learning) achieve better and more stable performance than a stationary model. Particularly, applying sophisticated model update strategies could provide better performance, efficiency, and stability than simply retraining AIOps models periodically. In addition, we observed that, although some update strategies can save model training time, they significantly sacrifice model testing time, which could hinder their applications in AIOps solutions where the operation data arrive at high pace and volume and where immediate inferences are required. Our findings highlight that practitioners should consider the evolution of operation data and actively maintain AIOps models over time. Our observations can also guide researchers and practitioners in investigating more efficient and effective model update strategies that fit in the context of AIOps.

An Automatic Model Management System and Its Implementation for AIOps on Microservice Platforms

On the Model Update Strategies for Supervised Learning in AIOps Solutions

Automating the Training and Deployment of Models in MLOps by Integrating Systems with Machine Learning

Model Management for Refinery Multi-level Simulation and Optimization

AASPMP: Design and Implementation of Production Management Platform Based on AAS

A Scenario-Oriented Benchmark for Assessing AIOps Algorithms in Microservice Management

Accelerating the Machine Learning Model Deployment using MLOps

A model-based framework for platform management in Cloud.

Quality Monitoring and Assessment of Deployed Deep Learning Models for Network AIOps

Automated end-to-end management of the modeling lifecycle in deep learning

OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System

AIOps in Action: Automating AI Deployment and Management of Large Language Models for Scalable and Ethical Operations

Model Based Monitoring and Controlling for Platform-as-a-Service PaaS

Time Series Prediction Method of Metrics of Dispatching Automation System Based on AI Platform

On Continuous Integration / Continuous Delivery for Automated Deployment of Machine Learning Models using MLOps

Prometheus and AIOps for the orchestration of Cloud-native applications in Ananke

Towards a consistent interpretation of AIOps models

Towards Automating the AI Operations Lifecycle

CMS: A Continuous Machine-Learning and Serving Platform for Industrial Big Data

Learning to Diagnose: Meta-Learning for Efficient Adaptation in Few-Shot AIOps Scenarios

CO-AutoML: an Optimizable Automated Machine Learning System