Abstract:In malicious URLs detection, traditional classifiers are challenged because the data volume is huge, patterns are changing over time, and the correlations among features are complicated. Feature engineering plays an important role in addressing these problems. To better represent the underlying problem and improve the performances of classifiers in identifying malicious URLs, this paper proposed a combination of linear and non-linear space transformation methods. For linear transformation, a two-stage distance metric learning approach was developed: first, singular value decomposition was performed to get an orthogonal space, and then a linear programming was used to solve an optimal distance metric. For nonlinear transformation, we introduced Nyström method for kernel approximation and used the revised distance metric for its radial basis function such that the merits of both linear and non-linear transformations can be utilized. 33,1622 URLs with 62 features were collected to validate the proposed feature engineering methods. The results showed that the proposed methods significantly improved the efficiency and performance of certain classifiers, such as k-Nearest Neighbor, Support Vector Machine, and neural networks. The malicious URLs’ identification rate of k-Nearest Neighbor was increased from 68% to 86%, the rate of linear Support Vector Machine was increased from 58% to 81%, and the rate of Multi-Layer Perceptron was increased from 63% to 82%. We also developed a website to demonstrate a malicious URLs detection system which uses the methods proposed in this paper. The system can be accessed at: http://url.jspfans.com.

MalFilter: A Lightweight Real-Time Malicious URL Filtering System in Large-Scale Networks

Malicious URL Detection Using Multi-Layer Filtering Model

A practical approach for clustering large data flows of malicious URLs

A Convolution-Based System for Malicious URLs Detection

Robust Detection of Malicious URLs with Self-Paced Wide & Deep Learning

Research and Application of a High Speed URL Filtering Algorithm

A Deep Learning Approach to Fast, Format-Agnostic Detection of Malicious Web Content

Malicious URL Detection via Pretrained Language Model Guided Multi-Level Feature Attention Network

A Malicious URL Detection Method Based on CNN

Finding Effective Classifier for Malicious URL Detection

Adaptive Malicious Url Detection: Learning In The Presence Of Concept Drifts

Malicious URL Detection Based on Improved Multilayer Recurrent Convolutional Neural Network Model

Malicious Web Page Detection Based on On-Line Learning Algorithm

TSMWD: A High-Speed Malicious Web Page Detection System Based on Two-Step Classifiers

Research on Malicious Traffic Detection Based on Word Embedding Algorithms and Neural Networks

TFD: A Multi-Pattern Matching Algorithm for Large-Scale URL Filtering

Optimal Filtering of Malicious IP Sources

Fast3DS: A real-time full-convolutional malicious domain name detection system

Improving Malicious URLs Detection Via Feature Engineering: Linear and Nonlinear Space Transformation Methods

DeepMAL -- Deep Learning Models for Malware Traffic Detection and Classification

Domainobserver: A Lightweight Solution For Detecting Malicious Domains Based On Dynamic Time Warping