Abstract:Extracting web content is to obtain the required data embedded in web pages, usually including structured records, such as product information, and text content, such as news. Web pages use a large number of HTML tags to organize and to present various information. Both knowing little about the structures of web pages and mixing kinds of information in web pages are making the extraction process very challenging to guarantee extraction performance and extraction adaptability. This study proposes a unified web content extraction framework that can be applied in various web environments to extract both structured records and text content. First, we construct a characteristic container to hold kinds of characteristics related with extraction objectives, including visual text information, content semantics(instead of HTML tag semantics), web page structures, etc. Second, the above characteristics are integrated into an extraction framework for extraction decisions on different web sites. Especially, we put forward different strategies, path aggregation for extracting text content and HMM model for structured records, to locate the extraction area by exploiting both those extraction characteristics. Comparative experiments on multiple web sites with popular extraction methods, including CETR, CETD and CNBE, show that our proposed extraction method can provide better extraction precision and extraction adaptability.

DOM-Based Automatic Extraction of Topical Information from Web Pages

Web Information Segmentation Method Based on DOM Structure Tree

Exploiting Multi-Category Characteristics and Unified Framework to Extract Web Content

Analysis and Implementation of Extraction Algorithm of Web Hierarchy Structure

Extracting Web Content by Exploiting Multi-Category Characteristics

The Technology of Extracting Content Information from Web Page Based on DOM Tree

Automatic Extraction Of Commodity Attributes On Webpages Based On Hierarchical Structure

Attributes extraction of Deep Web query interface based on DOM

Simplified DOM Trees for Transferable Attribute Extraction from the Web

DOM-based Content Extraction of HTML Documents

Tag Tree Template for Web Information and Schema Extraction.

Automatic Data Extraction from Web Discussion Forums

Ontology-Based Two-Phase Semi-Automatic Web Extracting

Solution for Automatic Web Review Extraction

A hybrid approach for content extraction with text density and visual importance of DOM nodes

Application and Design of Web Information Extraction System Based on Pattern Discovery

FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documents

Web Content Extraction & Its Data Management Method

Extracting Content For News Web Pages Based On Dom

Web Information Extraction Based on Repeated Pattern

Extracting Content Structure For Web Pages Based On Visual Representation