Abstract:Malware is a major threat to computer systems and imposes many challenges to cyber security. Targeted threats, such as ransomware, cause millions of dollars in losses every year. The constant increase of malware infections has been motivating popular antiviruses (AVs) to develop dedicated detection strategies, which include meticulously crafted machine learning (ML) pipelines. However, malware developers unceasingly change their samples' features to bypass detection. This constant evolution of malware samples causes changes to the data distribution (i.e., concept drifts) that directly affect ML model detection rates, something not considered in the majority of the literature work. In this work, we evaluate the impact of concept drift on malware classifiers for two Android datasets: DREBIN (about 130K apps) and a subset of AndroZoo (about 285K apps). We used these datasets to train an Adaptive Random Forest (ARF) classifier, as well as a Stochastic Gradient Descent (SGD) classifier. We also ordered all datasets samples using their VirusTotal submission timestamp and then extracted features from their textual attributes using two algorithms (Word2Vec and TF-IDF). Then, we conducted experiments comparing both feature extractors, classifiers, as well as four drift detectors (DDM, EDDM, ADWIN, and KSWIN) to determine the best approach for real environments. Finally, we compare some possible approaches to mitigate concept drift and propose a novel data stream pipeline that updates both the classifier and the feature extractor. To do so, we conducted a longitudinal evaluation by (i) classifying malware samples collected over nine years (2009-2018), (ii) reviewing concept drift detection algorithms to attest its pervasiveness, (iii) comparing distinct ML approaches to mitigate the issue, and (iv) proposing an ML data stream pipeline that outperformed literature approaches.

Online Learning Based Self-updating Incremental Malware Detection Model.

Adaptive and Scalable Android Malware Detection through Online Learning

Mitigating the Impact of Malware Evolution on API Sequence-based Windows Malware Detector

Continuous Learning for Android Malware Detection

A Real-Time and Adaptive-Learning Malware Detection Method Based on API-Pair Graph

Adaptive Malicious Url Detection: Learning In The Presence Of Concept Drifts

Fast & Furious: Modelling Malware Detection as Evolving Data Streams

Going Proactive and Explanatory Against Malware Concept Drift

Malware Analysis Using Machine Learning and Deep Learning Techniques

Optimized Deep Learning Models for Malware Detection under Concept Drift

Incremental Update Based Closed-Loop Malicious URL Detection System

Revisiting Concept Drift in Windows Malware Detection: Adaptation to Real Drifted Malware with Minimal Samples

DREAM: Combating Concept Drift with Explanatory Detection and Adaptation in Malware Classification

A Study on the Application of Distributed System Technology-Guided Machine Learning in Malware Detection

MORPH: Towards Automated Concept Drift Adaptation for Malware Detection

IMCDCF: An Incremental Malware Detection Approach Using Hidden Markov Models

Malware Detected and Tell MeWhy: An Verifiable Malware Detection Model with Graph Metric Learning

Efficient Concept Drift Handling for Batch Android Malware Detection Models

Evolving malware detection through instant dynamic graph inverse reinforcement learning

An Efficient DenseNet-Based Deep Learning Model for Malware Detection

Malware Detection Approach Based on Improved SOINN