Abstract:Extracting web content is to obtain the required data embedded in web pages, usually including structured records, such as product information, and text content, such as news. Web pages use a large number of HTML tags to organize and to present various information. Both knowing little about the structures of web pages and mixing kinds of information in web pages are making the extraction process very challenging to guarantee extraction performance and extraction adaptability. This study proposes a unified web content extraction framework that can be applied in various web environments to extract both structured records and text content. First, we construct a characteristic container to hold kinds of characteristics related with extraction objectives, including visual text information, content semantics(instead of HTML tag semantics), web page structures, etc. Second, the above characteristics are integrated into an extraction framework for extraction decisions on different web sites. Especially, we put forward different strategies, path aggregation for extracting text content and HMM model for structured records, to locate the extraction area by exploiting both those extraction characteristics. Comparative experiments on multiple web sites with popular extraction methods, including CETR, CETD and CNBE, show that our proposed extraction method can provide better extraction precision and extraction adaptability.

Dom based extraction of topical information from web pages

DOM-Based Automatic Extraction of Topical Information from Web Pages

Web Information Segmentation Method Based on DOM Structure Tree

Analysis and Implementation of Extraction Algorithm of Web Hierarchy Structure

Exploiting Multi-Category Characteristics and Unified Framework to Extract Web Content

Automatic Extraction Of Commodity Attributes On Webpages Based On Hierarchical Structure

Adaptive Web Information Extraction Based on DOM Tree

Attributes extraction of Deep Web query interface based on DOM

An Algorithm on Web Article Automatic Extraction Based on DOM Structure

The Technology of Extracting Content Information from Web Page Based on DOM Tree

Combing Node Frequency and Semantic Feature for Webpage Informative Content Extraction

A Semantic DOM Approach for Webpage Information Extraction

An Approach of Information Extraction Based on Dom Tree and Weight Value

DOM-Based Multi-Factor Web Information Extraction Study

An Adaptive Web Information Extraction Approach Based on Stu-Dom Tree

Web News Pages Extraction Method Based on DOM and Decision Tree

DOM-based Content Extraction of HTML Documents

Simplified DOM Trees for Transferable Attribute Extraction from the Web

Tag Tree Template for Web Information and Schema Extraction.

Extraction of Content from Web Pages Based on Magnitude of Reduction of Information Quantity

Topic information extraction from Web pages based on tree comparison