Abstract:Focused Crawler searches the internet for topic-specific web pages. Its effectiveness is determined on the multidimensional nature of the web pages. The main task of any Focused Crawler is to collect relevant web pages of predefined topics and neglecting the irrelevant web pages. Traditional Best-First based Focused Crawlers (FC) are based on Vector Space Model (VSM) which uses Term Frequency-Inverse Document Frequency (TF-IDF) that gives limited success rate on the web page classification. The major practical challenge associated with Focused Crawler is to correctly classify the web pages based on the given topic due to the unstructured data in web pages. The main objective of this work is to design an improved focused Crawling approach using web page classification. This work proposes a text classification model based on Long Short Term Memory (LSTM) and Convolutional Neural Network (CNN) with word embeddings to increase the accuracy of web page classification. The LSTM-CNN based text classification model is further used to guide the Focused Crawler for classification of web pages. The proposed text classification model is implemented by combining the LSTM with CNN. The validation of the proposed LSTM-CNN text classification model is carried out on different datasets and results are then compared with traditional supervised machine learning algorithms and different deep neural network (DNN) based approaches like CNN, RNN and RCNN.The suggested text classification model performs 8–12 percent better than typical supervised machine learning algorithms and 4–6 percent better than CNN, RNN, and RCNN, according to experimental results. Also, the improved focused crawling approach with LSTM-CNN based text classification model gives increasing harvest rate and target recall as compared to the Breadth-First Crawler, Best-First Crawler,CNN Crawler and DNN Crawler.

Personalized Content Extraction and Text Classification Using Effective Web Scraping Techniques

Extracting Web Content by Exploiting Multi-Category Characteristics

Exploiting Multi-Category Characteristics and Unified Framework to Extract Web Content

Improving short text classification using public search engines

Web Scraping using Natural Language Processing: Exploiting Unstructured Text for Data Extraction and Analysis

An Efficient Information Extraction Mechanism with Page Ranking and a Classification Strategy based on Similarity Learning of Web Text Documents

Web Scraping - State of Art, Techniques and Approaches

An efficient focused crawler using LSTM-CNN based deep learning

Web Crawler: Design And Implementation For Extracting Article-Like Contents

NLP based intelligent news search engine using information extraction from e-newspapers

Web Scraping or Web Crawling: State of Art, Techniques, Approaches and Application

DOM-based Content Extraction of HTML Documents

Data Analysis by Web Scraping using Python

Eksplorasi Teknik Web Scraping pada Data Mining: Pendekatan Pencarian Data Berbasis Python

Using Structured Tokens To Identify Webpages For Data Extraction

Extracting Novel Features for E-Commerce Page Quality Classification.

Web Data Mining with Organized Contents Using Naive Bayes Algorithm

Scalable and Noise Tolerant Web Knowledge Extraction for Search Task Simplification.

PYTHON-POWERED DATA ANALYSIS THROUGH WEB SCRAPING

Similarity based Dynamic Web Data Extraction and Integration System from Search Engine Result Pages for Web Content Mining

Label-Efficient Self-Training for Attribute Extraction from Semi-Structured Web Documents