EarlyMalDetect: A Novel Approach for Early Windows Malware Detection Based on Sequences of API Calls

Pascal Maniriho,Abdun Naser Mahmood,Mohammad Jabed Morshed Chowdhury

2024-07-18

Abstract:In this work, we propose EarlyMalDetect, a novel approach for early Windows malware detection based on sequences of API calls. Our approach leverages generative transformer models and attention-guided deep recurrent neural networks to accurately identify and detect patterns of malicious behaviors in the early stage of malware execution. By analyzing the sequences of API calls invoked during execution, the proposed approach can classify executable files (programs) as malware or benign by predicting their behaviors based on a few shots (initial API calls) invoked during execution. EarlyMalDetect can predict and reveal what a malware program is going to perform on the target system before it occurs, which can help to stop it before executing its malicious payload and infecting the system. Specifically, EarlyMalDetect relies on a fine-tuned transformer model based on API calls which has the potential to predict the next API call functions to be used by a malware or benign executable program. Our extensive experimental evaluations show that the proposed approach is highly effective in predicting malware behaviors and can be used as a preventive measure against zero-day threats in Windows systems.

Cryptography and Security

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are two major challenges in early Windows malware detection:

1. **Predicting API call sequences (RQ1)**: Based on the API sequences called at the initial execution stage of Windows executable programs (whether they are malware or benign programs), can the API sequences that the program will call next be predicted?
2. **Detecting at the early stage of malware execution (RQ2)**: Can malware be detected before it infects the system by predicting its behavior?

### Specific problem description

#### 1. Predicting API call sequences
Existing malware detection techniques based on API call sequences lack effective early - prediction mechanisms. Many methods rely on log data after static or dynamic analysis to identify malicious behavior, which usually takes a long time to collect sufficient behavior data, thus increasing the risk of malware successfully executing malicious payloads. Therefore, the paper proposes a new method, using generative Transformer models and attention - guided deep recurrent neural networks to predict API call sequences, thereby achieving early detection.

#### 2. Detecting at the early stage of malware execution
Traditional behavior - based malware detection models can usually detect the activities of malware only after it has infected the system, which is too late for critical systems. To overcome this limitation, the paper proposes a new detection method that can predict the behavior of malware when it starts to execute and prevent it before the malicious payload is executed.

### Solution overview

To solve the above problems, the paper proposes a new method named **EarlyMalDetect**, which has the following characteristics:

- **Generative Transformer model**: By fine - tuning the pre - trained GPT - 2 model, it can predict subsequent API calls according to the initial API call sequences.
- **Attention - guided deep recurrent neural network**: Combine bidirectional recurrent neural networks (BiGRU) and attention mechanisms to more accurately identify and classify malicious behavior.
- **Early prediction and detection**: By predicting API call sequences, potential threats can be identified at the early stage of malware execution, so that appropriate preventive measures can be taken.

### Experimental verification

The paper conducts extensive experimental evaluations on multiple API call sequence data sets. The results show that the proposed method performs excellently in predicting malware behavior and can effectively detect it before the malware infects the system. Specifically, the experimental results show that this method outperforms other existing methods under different conditions.

### Summary

By solving the above two research problems, the paper aims to demonstrate the potential of prediction models in malware detection based on API call sequences, especially in early detection and prevention. This method not only improves the detection accuracy but also significantly reduces the potential damage of malware to the system.