Abstract:Previous work shows that a web page can be partitioned into multiple segments or blocks, and often the importance of those blocks in a page is not equivalent. Also, it has been proven that differentiating noisy or unimportant blocks from pages can facilitate web mining, search and accessibility. However, no uniform approach and model has been presented to measure the importance of different segments in web pages. Through a user study, we found that people do have a consistent view about the importance of blocks in web pages. In this paper, we investigate how to find a model to automatically assign importance values to blocks in a web page. We define the block importance estimation as a learning problem. First, we use a vision-based page segmentation algorithm to partition a web page into semantic blocks with a hierarchical structure. Then spatial features (such as position and size) and content features (such as the number of images and links) are extracted to construct a feature vector for each block. Based on these features, learning algorithms are used to train a model to assign importance to different segments in the web page. In our experiments, the best model can achieve the performance with Micro-F1 79% and Micro-Accuracy 85.9%, which is quite close to a person's view.

Algorithm for Webpage Semantic Blocks Mining Using Tree Match Method

Web Information Segmentation Method Based on DOM Structure Tree

Web Object Block Mining Based on Tag Similarity

Block-based Web Search

Basic Semantic Units Based Web Page Content Extraction.

Content Extraction of Web Pages Based on Characteristic Symbols

Learning Important Models for Web Page Blocks Based on Layout and Content Analysis

Research of Semantic Web Mining

SFTM: Fast Comparison of Web Documents using Similarity-based Flexible Tree Matching

Layout-tree-based Approach for Identifying Visually Similar Blocks in a Web Page.

Constructing Novel Block Layouts for Webpage Analysis

Combing Node Frequency and Semantic Feature for Webpage Informative Content Extraction

A Parallel Pages Mining Approach: Combining URL Patterns and HTML Structures

Learning Block Importance Models for Web Pages

Similarity Evaluation of Tree-Structured Web Pages

E-commerce Webpage Recommendation Scheme Base on Semantic Mining and Neural Networks

Web Documents Mining

Chinese web page content extraction based on page content analysis

A Study on Combination of Block Importance and Relevance to Estimate Page Relevance

The construct and use of semantic tree of Web pages