Abstract:With the rapid growth of the data scale in data centers, the high reliability of storage is facing various challenges. Specifically, hardware failures such as disk faults occur frequently, causing serious system availability issues. In this context, hardware fault prediction based on AI and big data technologies has become a research hotspot, aiming to guide operation and maintenance personnel to implement preventive replacement through accurate prediction to reduce hardware failure rates. However, existing methods still have weaknesses in terms of accuracy due to the impacts of data quality issues such as the sample imbalance. This article proposes a disk fault prediction method based on classification intensity resampling, which fills the gap between the degree of data imbalance and the actual classification intensity of the task by introducing a base classifier to calculate the classification intensity, thus better preserving the data features of the original dataset. In addition, using ensemble learning methods such as random forests, combined with resampling, an integrated classifier for imbalanced data is developed to further improve the prediction accuracy. Experimental verification shows that compared with traditional methods, the F1-score of disk fault prediction is improved by 6%, and the model training time is also greatly reduced. The fault prediction method proposed in this paper has been applied to approximately 80 disk drives and nearly 40,000 disks in the production environment of a large bank’s data center to guide preventive replacements. Compared to traditional methods, the number of preventive replacements based on our method has decreased by approximately 21%, while the overall disk failure rate remains unchanged, thus demonstrating the effectiveness of our method.

Tree-Based Model with Advanced Data Preprocessing for Large Scale Hard Disk Failure Prediction

Hard Drive Failure Prediction Using Classification and Regression Trees

A Failure Prediction Approach Based on BiLSTM and Deep Feature Extractor for Hard Disk Drives.

Hard drive failure prediction using Decision Trees

Optimizing Small-Sample Disk Fault Detection Based on LSTM-GAN Model

Disk failure prediction based on multi-layer domain adaptive learning

Data-level hybrid strategy selection for disk fault prediction model based on multivariate GAN

Disk Failure Prediction in Data Centers Via Online Learning

Hard Drive Failure Prediction Using Big Data

Prediction of Disk Failure Based on Classification Intensity Resampling

An Attention-augmented Deep Architecture for Hard Drive Status Monitoring in Large-scale Storage Systems

Proactive drive failure prediction for large scale storage systems

DiskTransformer: A Transformer Network for Hard Disk Failure Prediction

Layerwise Perturbation-Based Adversarial Training for Hard Drive Health Degree Prediction

Adversarial Domain Adaptation with Correlation-Based Association Networks for Longitudinal Disk Fault Prediction

Hard Disk Failure Prediction Based on Lightgbm with CID.

Predicting Hard Drive Failures for Cloud Storage Systems

Hard Disk Failure Prediction Based on Blending Ensemble Learning

Optimizing Efficiency of Machine Learning Based Hard Disk Failure Prediction by Two-Layer Classification-Based Feature Selection

A Bearing Fault Diagnosis Method for High-Noise and Unbalanced Dataset

SPAE: Lifelong disk failure prediction via end-to-end GAN-based anomaly detection with ensemble update