Abstract:Concept drift poses a critical challenge in deploying machine learning models to mitigate practical malware threats. It refers to the phenomenon that the distribution of test data changes over time, gradually deviating from the original training data and degrading model performance. A promising direction for addressing concept drift is to detect drift samples and then retrain the model. However, this field currently lacks a unified, well-curated, and comprehensive benchmark, which often leads to unfair comparisons and inconclusive outcomes. To improve the evaluation and advance further, this paper presents a new Bench mark dataset for trustworthy M alware F amily C lassification (BenchMFC), which includes 223 K samples of 526 families that evolve over years. BenchMFC provides clear family, packer, and timestamp tags for each sample, it thus can support research on three types of malware concept drift: 1) unseen families, 2) packed families, and 3) evolved families. To collect unpacked family samples from large-scale candidates, we introduce a novel crowdsourcing malware annotation pipeline, which unifies packing detection and family annotation as a consensus inference problem to prevent costly packing detection. Moreover, we provide two case studies to illustrate the application of BenchMFC in 1) concept drift detection and 2) model retraining. The first case demonstrates the impact of three types of malware concept drift and compares nine notable concept drift detectors. The results show that existing detectors have their own advantages in dealing with different types of malware concept drift, and there is still room for improvement in malware concept drift detection. The second case explores how static feature-based machine learning operates on packed samples when retraining a model. The experiments illustrate that packers do preserve some kind of signals that appear to be "effective" for machine learning models, but the robustness of these signals requires further research. BenchMFC has been released to the community at https://github.com/crowdma/benchmfc .

Revisiting Concept Drift in Windows Malware Detection: Adaptation to Real Drifted Malware with Minimal Samples

Optimized Deep Learning Models for Malware Detection under Concept Drift

Going Proactive and Explanatory Against Malware Concept Drift

MORPH: Towards Automated Concept Drift Adaptation for Malware Detection

Efficient Concept Drift Handling for Batch Android Malware Detection Models

DREAM: Combating Concept Drift with Explanatory Detection and Adaptation in Malware Classification

Counteracting Concept Drift by Learning with Future Malware Predictions

Learn to Adapt: Robust Drift Detection in Security Domain

Fast & Furious: Modelling Malware Detection as Evolving Data Streams

Is It Overkill? Analyzing Feature-Space Concept Drift in Malware Detectors

Adaptive Malicious Url Detection: Learning In The Presence Of Concept Drifts

Continuous Learning for Android Malware Detection

Android Malware Concept Drift using System Calls: Detection, Characterization and Challenges

ReCDA: Concept Drift Adaptation with Representation Enhancement for Network Intrusion Detection

Recent Advances in Concept Drift Adaptation Methods for Deep Learning.

Adaptive and Scalable Android Malware Detection through Online Learning

Malware Analysis Using Machine Learning and Deep Learning Techniques

BenchMFC: A Benchmark Dataset for Trustworthy Malware Family Classification under Concept Drift

Transcending Transcend: Revisiting Malware Classification in the Presence of Concept Drift

Evolving malware detection through instant dynamic graph inverse reinforcement learning

Online Learning Based Self-updating Incremental Malware Detection Model.