Abstract:Focused Crawler searches the internet for topic-specific web pages. Its effectiveness is determined on the multidimensional nature of the web pages. The main task of any Focused Crawler is to collect relevant web pages of predefined topics and neglecting the irrelevant web pages. Traditional Best-First based Focused Crawlers (FC) are based on Vector Space Model (VSM) which uses Term Frequency-Inverse Document Frequency (TF-IDF) that gives limited success rate on the web page classification. The major practical challenge associated with Focused Crawler is to correctly classify the web pages based on the given topic due to the unstructured data in web pages. The main objective of this work is to design an improved focused Crawling approach using web page classification. This work proposes a text classification model based on Long Short Term Memory (LSTM) and Convolutional Neural Network (CNN) with word embeddings to increase the accuracy of web page classification. The LSTM-CNN based text classification model is further used to guide the Focused Crawler for classification of web pages. The proposed text classification model is implemented by combining the LSTM with CNN. The validation of the proposed LSTM-CNN text classification model is carried out on different datasets and results are then compared with traditional supervised machine learning algorithms and different deep neural network (DNN) based approaches like CNN, RNN and RCNN.The suggested text classification model performs 8–12 percent better than typical supervised machine learning algorithms and 4–6 percent better than CNN, RNN, and RCNN, according to experimental results. Also, the improved focused crawling approach with LSTM-CNN based text classification model gives increasing harvest rate and target recall as compared to the Breadth-First Crawler, Best-First Crawler,CNN Crawler and DNN Crawler.

PDD Crawler: A focused web crawler using link and content analysis for relevance prediction

LEARNING-based Focused WEB Crawler

Analysis and Evaluation of the Link and Content Based Focused Treasure-Crawler

Optimized Focused Web Crawler with Natural Language Processing Based Relevance Measure in Bioinformatics Web Sources

Web Crawler and Web Crawler Algorithms: A Perspective

Comparative analysis of various web crawler algorithms

Focused Crawling Using Navigational Rank.

An Enhanced Semantic Focused Web Crawler Based on Hybrid String Matching Algorithm

Effective performance of information retrieval on web by using web crawling

Weakly supervised learning for an effective focused web crawler

Analysis of Statistical Hypothesis based Learning Mechanism for Faster Crawling

A Focused Crawler Combinatory Link and Content Model Based on T-Graph Principles

Analysis of a Statistical Hypothesis Based Learning Mechanism for Faster crawling

A Predication-Based Approach for Effective Resource Discovery in Topical Web

An efficient focused crawler using LSTM-CNN based deep learning

Towards A Quality-Oriented Real-Time Web Crawler

On-Line Topical Importance Estimation: an Effective Focused Crawling Algorithm Combining Link and Content Analysis

UCrawler: A learning-based web crawler using a URL knowledge base

Amelioration of linguistic semantic classifier with sentiment classifier manacle for the focused web crawler

A Focused Crawler Based on Correlation Analysis

A Semantic and Optimized Focused Crawler Based on Semantic Graph and Genetic Algorithm