Abstract:Uniform Resource Locator (URL) is a unique identifier composed of protocol and domain name used to locate and retrieve a resource on the Internet. Like any Internet service, URLs (also called websites) are vulnerable to compromise by attackers to develop Malicious URLs that can exploit/devastate the user's information and resources. Malicious URLs are usually designed with the intention of promoting cyber-attacks such as spam, phishing, malware, and defacement. These websites usually require action on the user's side and can reach users across emails, text messages, pop-ups, or devious advertisements. They have a potential impact that can reach, in some cases, to compromise the machine or network of the user, especially those arriving by email. Therefore, developing systems to detect malicious URLs is of great interest nowadays. This paper proposes a high-performance machine learning-based detection system to identify Malicious URLs. The proposed system provides two layers of detection. Firstly, we identify the URLs as either benign or malware using a binary classifier. Secondly, we classify the URL classes based on their feature into five classes: benign, spam, phishing, malware, and defacement. Specifically, we report on four ensemble learning approaches, viz. the ensemble of bagging trees (En_Bag) approach, the ensemble of k-nearest neighbor (En_kNN) approach, and the ensemble of boosted decision trees (En_Bos) approach, and the ensemble of subspace discriminator (En_Dsc) approach. The developed approaches have been evaluated on an inclusive and contemporary dataset for uniform resource locators (ISCX-URL2016). ISCX-URL2016 provides a lightweight dataset for detecting and categorizing malicious URLs according to their attack type and lexical analysis. Conventional machine learning evaluation measurements are used to evaluate the detection accuracy, precision, recall, F Score, and detection time. Our experiential assessment indicates that the ensemble of bagging trees (En_Bag) approach provides better performance rates than other ensemble methods. Alternatively, the ensemble of the k-nearest neighbor (En_kNN) approach provides the highest inference speed. We also contrast our En_Bag model with state-of-the-art solutions and show its superiority in binary classification and multi-classification with accuracy rates of 99.3% and 97.92%, respectively.

Classifier-Guided Topical Crawler: A Novel Method of Automatically Labeling the Positive URLs

A New Algorithm of Topical Crawler

Learnable Topical Crawler Through Online Semi-Supervised Clustering

UPCA: an Efficient URL-Pattern Based Algorithm for Accurate Web Page Classification.

UCrawler: A learning-based web crawler using a URL knowledge base

New algorithm of topic-oriented crawler

On-Line Topical Importance Estimation: an Effective Focused Crawling Algorithm Combining Link and Content Analysis

A Predication-Based Approach for Effective Resource Discovery in Topical Web

New Automatic Categorization Algorithm for Chinese Homepages

Can Chinese Web Pages Be Classified with English Data Source?

Web Page's Blocks Based Topical Crawler

A Web Site Classification Approach Based on Its Topological Structure.

An Approach to Extracting Central URLs on Catalog Page

Guide Focused Crawler Efficiently and Effectively Using On-Line Topical Importance Estimation

Automatic Corpora Construction for Text Classification.

Fine-grained Classification using Heterogeneous Web Data and Auxiliary Categories

Website Crawling for Specific Topics

An intelligent identification and classification system for malicious uniform resource locators (URLs)

SmartCrawler: A Two-Stage Crawler for Efficiently Harvesting Deep-Web Interfaces

A Semantic and Optimized Focused Crawler Based on Semantic Graph and Genetic Algorithm

An efficient clustering algorithm for large-scale topical web pages.