A Novel Method for the Web Page Segmentation and Identification

Jing Wang,Zhijing Liu

DOI: https://doi.org/10.1109/ICCET.2009.149

2009-01-01

Abstract:Keyword extraction has been a very traditional topic in Natural Language Processing. However, most methods have been too complicated and slow to be applied in real applications, for example in web-based system. This paper proposes an approach which will ...

What problem does this paper attempt to address?

Web Information Segmentation Method Based on DOM Structure Tree

ZHOU Jian,TANG Jin,LUO Bin

DOI: https://doi.org/10.3969/j.issn.1006-2475.2013.10.056

2013-01-01

Abstract:Correct extraction and segmentation of Web information is significant to text information mining.The paper proposes and achieves a method which can get informative information from Web page and be able to follow the correct segmentation of the original text.The method first uses page layout tag < table > and < div > to build a DOM structure tree,and then uses the nested relations of the layout label,that the DOM structure tree reflects to choose the content blocks,extract text information correctly,and finally achieves information segment of the body through the manipulation of some special tags.The experimental results prove that this method is easy to realize and high efficiency and it can automatically extract informative message and section accurately.
A search-based Chinese word segmentation method.

Xin-Jing Wang,Yong Qin,Wen Liu

DOI: https://doi.org/10.1145/1242572.1242729

2007-01-01

Abstract:In this paper, we propose a novel Chinese word segmentation method which leverages the huge deposit of Web documents and search technology. It simultaneously solves ambiguous phrase boundary resolution and unknown word identification problems. Evaluations prove its effectiveness.
Research on Feature Extraction for Content-based Chinese Web Pages Analysis

Yizhong Zhang,Mingsheng Zhao,Jingnan Zhu

DOI: https://doi.org/10.3321/j.issn:1002-8331.2001.10.001

2001-01-01

Abstract:This paper presents a feature framework for content-based Chinese web page analysis and searching. The method for constructing segmentation keyword dictionary is introduced first. The keywords in the dictionary are these words that represent the contents and concepts of a certain are web pages. Then,feature extraction methods for text,tag information and hyperlink information are addressed. Experiments have shown that the proposed methods tested on Chinese travel web pages are worked very well.
Constructing Novel Block Layouts for Webpage Analysis

Zexun Jiang,Hao Yin,Yulei Wu,Yongqiang Lyu,Geyong Min,Xu Zhang

DOI: https://doi.org/10.1145/3326457

IF: 5.3

2019-01-01

ACM Transactions on Internet Technology

Abstract:Webpage segmentation is the basic building block for a wide range of webpage analysis methods. The rapid development of Web technologies results in more dynamic and complex webpages, which bring new challenges to this area. To improve the performance of webpage segmentation, we propose a two-stage segmentation method that can combine visual, logic, and semantic features of the contents on a webpage. Specifically, we devise a new model to measure the similarities of the elements on webpages based on both visual layout and logic organization in the first stage, and we propose a novel block regrouping method using semantic statistics and visual positions in the second stage. This two-stage method can effectively conduct webpage segmentation on complicated and dynamic webpages. The performance and accuracy of the method are verified by comparing with two existing webpage segmentation methods. The experiment results show that the proposed method significantly outperforms the existing state of the art in terms of higher precision, recall, and accuracy.
An Application Of Lexical Semantics Of Chinese In Webpage Keyword Extraction Algorithm

Chanjuan Wang,Bin Sun,Lu Zhang

2010-01-01

Abstract:The principles of Lexical Semantics provide guidance for webpage keyword extraction. This paper designs an algorithm that utilizes Chinese words' semantic features and leads to decent extraction effects. According to the semantic systematic principle and the mutual dependence among sememes, the algorithm further enhances its performance by introducing a set of "seed keywords", thereby the conceptual feature of the extracted keywords is emphasized, to which advertisers can attach more references.
Web Page Segmentation and Its Application for Web Information Crawling

Hanyang Feng,Wenzhe Zhang,Hesheng Wu,Chong-Jun Wang

DOI: https://doi.org/10.1109/ictai.2016.0097

2016-01-01

Abstract:Web page segmentation aims to break a page into sections that can reveal the information presentation structure and appear coherent to readers. In this paper, we propose a new web page segmentation framework based on the process of analyzing and understanding web page structure. After extracting the segmentation graph structure, we formulate the label assignment task which determines whether each boundary should segment current block or not on a graph as a structured learning problem. Computation of highest scoring label assignment relies on Viterbi algorithm and joint feature function captures the dependency among boundaries. To solve the learning of parameters, we adopt a learning model based on perceptron algorithm. Furthermore, utilizing the previous framework, we propose a web information crawling application framework which integrates web page segmentation and semantic block classification process.
A Web Page Segmentation Approach Using Visual Semantics.

Jun Zeng,Brendan Flanagan,Sachio Hirokawa,Eisuke Ito

DOI: https://doi.org/10.1587/transinf.e97.d.223

2014-01-01

IEICE Transactions on Information and Systems

Abstract:Web page segmentation has a variety of benefits and potential web applications. Early techniques of web page segmentation are mainly based on machine learning algorithms and rule-based heuristics, which cannot be used for large-scale page segmentation. In this paper, we propose a formulated page segmentation method using visual semantics. Instead of analyzing the visual cues of web pages, this method utilizes three measures to formulate the visual semantics: layout tree is used to recognize the visual similar blocks; seam degree is used to describe how neatly the blocks are arranged; content similarity is used to describe the content coherent degree between blocks. A comparison experiment was done using the VIPS algorithm as a baseline. Experiment results show that the proposed method can divide a Web page into appropriate semantic segments.
A Novel Iterative Algorithm to Text Segmentation for Web Born-Digital Images.

Zhigang Xu,Yuesheng Zhu,Ziqiang Sun,Zhen Liu

DOI: https://doi.org/10.1117/12.2197039

2015-01-01

Abstract:Since web born-digital images have low resolution and dense text atoms, text region over-merging and miss detection are still two open issues to be addressed. In this paper a novel iterative algorithm is proposed to locate and segment text regions. In each iteration, the candidate text regions are generated by detecting Maximally Stable Extremal Region (MSER) with diminishing thresholds, and categorized into different groups based on a new similarity graph, and the texted region groups are identified by applying several features and rules. With our proposed overlap checking method the final well-segmented text regions are selected from these groups in all iterations. Experiments have been carried out on the web born-digital image datasets used for robust reading competition in ICDAR 2011 and 2013, and the results demonstrate that our proposed scheme can significantly reduce both the number of over-merge regions and the lost rate of target atoms, and the overall performance outperforms the best compared with the methods shown in the two competitions in term of recall rate and f-score at the cost of slightly higher computational complexity.
Keywords extraction algorithm based on combined word and synset

JIANG Chang-jin,PENG Hong,CHEN Jian-chao,MA Qian-li,YAN Gui-duo

DOI: https://doi.org/10.3969/j.issn.1001-3695.2010.08.010

2010-01-01

Abstract:This paper presented a Chinese webpage keywords extraction algorithm after the study of existing techniques for keyword extraction. The presented approach could extremely improve the performance of Chinese word segmentation system. The modified Chinese word segmentation system could recognise most of new terms,phrases and non-login words in Chinese webpage and this is vitally important for Chinese keyword extraction. Moreover,constructed a synset database and used adding the frequencies of synonyms together,avoiding the co-occurrence of synonyms in output. Further more,created a eva-luation function to score candidate keyword based on its location,length,part-of-speech. The experiment results show that the proposed algorithm has better performance compared with the traditional keyword extraction algorithms.
Web Pages Information Retrieval Based on Keywords Cluster and Node Instance

DENG Jian-Shuang,ZHENG Qi-Lun,PENG Hong,LIN Xu-Dong

DOI: https://doi.org/10.3969/j.issn.1002-137X.2007.04.058

2007-01-01

Computer Science

Abstract:Many Web information retrieval methods are related to special Web sites, for example, the method based on extracting rules and the one based on training page samples. These methods can do well in a Web site but fail in the others without adding new rules or inputting new training pages manually. Furthermore, if the template of the Web site is changed, it has to redesign the extracting rules or re-inputting the training pages. It is hard to be maintained and used to extract information from large number of different Web sites. In the paper, there is a new method which can extract the useful information from the different sites automatically based on the keywords of a certain topic and the distance of the nodes. Experimental evaluation on a large of Web pages from different Web sites indicates that this method correctly and automatically extracts the information ignoring which Web sites the pages come from. This method has been applied to the system of intelligent searching and mining of electronic business successfully.
Chinese Word Segmentation Evaluation Methodology Based on Web Search Engines

WANG Hua-dong,RAO Pei-lun

DOI: https://doi.org/10.3969/j.issn.1007-7634.2007.01.022

IF: 8.1

2007-01-01

Information Sciences

Abstract:Chinese word segmentation is one of the determinants of result quality of Chinese search engines.Whether Chinese words are segmented effectively and correctly is vital to improving the relevance of the searching results and enhancing user satisfaction.The author first reviews the fundamental theories upon which Chinese segmentation evaluation methods are build,and then develops an integrated methodology measuring the quality of Chinese segmentation for web search engine.A set of methods and guidelines are proposed,addressing sampling issues,selection of evaluators,definition and selection of metrics,procedureof the evaluation,and etc.Then the methodology was applied in a real search engine evaluation in practice,and proved to be effective.The result of the evaluation was analyzed,and suggestions concerning evaluator screening and item rejection are provided,with the aim to get a better evaluation performance.
A Novel POS-Based Approach to Chinese News Topic Extraction from Internet

Xujian Zhao,Peiquan Jin,Lihua Yue

DOI: https://doi.org/10.1109/fgcns.2008.71

2008-01-01

Abstract:News topic extraction is very important for news search engine. The traditional methods are based on pattern matching and linguistic analysis, which mainly depend on the measurement of feature similarity. But due to two reasons, those methods are basically inefficient to process Chinese news topic extraction from Internet. The first reason is the difficulty of Natural Language Processing (NLP) for Chinese, and the other is the diversity and fast update speed of Internet news. At the present, some works utilizing news special structure (e.g. title) for Chinese news topic are presented. However, two problems still remain unsolved so far, which are (1) missing of some news topic and (2) irregular topic words produced. Aiming to solve these two problems, we propose a POS-based approach to news topic extraction. We first segment words and tag POS for news title, and then eliminate segmentation errors according to POS information and position relation. After that, topic words are associated and combined into bigger ones, and different topic weights are assigned to those bigger words. We conduct an experiment on 600 Chinese news Web pages to demonstrate our new approach. The experimental results show that our approach has a higher recall and precision rate of news topic extraction and reduces irregular topic words obviously.
A novel graph partition based page segmentation algorithm

Ye Yumming,Li Chunshan,Zhang Xiaofeng

2012-01-01

Information

Abstract:In mobile device browsing and web mining applications, segmenting web pages into small semantic modules is an essential and helpful technique. Traditional segmentation algorithms can only exploit some heuristic and explicit information of DOM tree, while the important semantic information hidden in sub tree structures of DOM trees is ignored. This results in poor performance in segmenting complicated web pages. In this paper, we cast page segmentation problem to a graph partition problem, where each DOM tree can be extracted as a weighted graph. Then, the graph partition based page segmentation algorithm (GPPS) is proposed to identify semantic modules by mining the sub tree structures of DOM trees. The proposed approach is evaluated by performing rigorous experiments on various web datasets as well as a large-scale dataset The experimental results demonstrate that GPPS is superior to the VIPS in terms of precision, recall, F-score and the time consumptions. ©2012 International Information Institute.
An Algorithm on Web Article Automatic Extraction Based on DOM Structure

Weiguo Shen,Xiaojian Zou

DOI: https://doi.org/10.14257/ijhit.2015.8.3.22

2015-01-01

International Journal of Hybrid Information Technology

Abstract:By analyzing a large number of web pages, we proposed a page segment algorithm which is based on DOM tree structural features and visual features. The algorithm segments the pages to the small particles. It produces basic processing units for recognition algorithms. After segmenting the pages, we extracted the structural and visual features of the pages, and proposed a method to identify the body of the web article. The method uses clustering algorithm and heuristic rules to produce an automatic wrapper. A testing experiment demonstrated the efficacy of the algorithm.
A segmentation method for web page analysis using shrinking and dividing

Jiuxin Cao,Bo Mao,Junzhou Luo

DOI: https://doi.org/10.1080/17445760802429585

2010-01-01

International Journal of Parallel Emergent and Distributed Systems

Abstract:On the basis of image processing technology and characteristics of web pages, a new web segmentation method-iterated shrinking and dividing is proposed in this paper. Dividing conditions and concept of dividing zone are introduced, based on which web page image is divided into visually consentaneous sub-images by shrinking and splitting iteratively. First, the web page is saved as image that is preprocessed by edge detection algorithm such as Canny. Then dividing zones are detected and the web image is segmented repeatedly until all blocks are indivisible. This method can be used to analyse the web pages such as detecting similar visual layout. Experiments show that the algorithm is suitable for web page segmentation, and does well in expansibility and performance.
Word Segmentation for Chinese Novels

Likun Qiu,Yue Zhang

DOI: https://doi.org/10.1609/aaai.v29i1.9523

2015-02-19

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Word segmentation is a necessary first step for automaticsyntactic analysis of Chinese text. Chinese segmentationis highly accurate on news data, but the accuraciesdrop significantly on other domains, such as science andliterature. For scientific domains, a significant portionof out-of-vocabulary words are domain-specific terms, and therefore lexicons can be used to improve segmentationsignificantly. For the literature domain, however,there is not a fixed set of domain terms. For example,each novel can contain a specifiac set of person, organizationand location names. We investigate a method forautomatically mining common noun entities for eachnovel using information extraction techniques, and usethe resulting entities to improve a state-of-the-art segmentationmodel for the novel. In particular, we designa novel double-propagation algorithm that mines nounentities together with common contextual patterns, anduse them as plug-in features to a model trained on thesource domain. An advantage of our method is that noretraining for the segmentation model is needed for eachnovel, and hence it can be applied efficiently given thehuge number of novels on the web. Results on five differentnovels show significantly improved accuracies,in particular for OOV words.
Page Segmentation of Chinese Newspapers

J Xi,JM Hu,LD Wu

DOI: https://doi.org/10.1016/s0031-3203(01)00248-5

IF: 8

2002-01-01

Pattern Recognition

Abstract:This paper describes a new bottom-up method for page segmentation of Chinese document images. Because of some special characteristics of Chinese newspaper documents, many traditional methods developed for English documents fail in segmenting them correctly. Based on run-length smoothing algorithm and minimal spanning tree clustering, the proposed method can resolve the problems of segmenting Chinese documents that differ from English documents.
DeSeA: A Page Segmentation Based Algorithm for Information Extraction

Juan He,Zhiqiang Gao,Hui Xu,Yuzhong Qu

DOI: https://doi.org/10.1109/skg.2005.62

2005-01-01

Abstract:The World Wide Web is becoming the most important source for surveying the state of research areas all over the world. DynamicView is such a Semantic Web portal implemented for researchers to query distribution of research areas in computer science. From millions of pages in free text available on thousands of web sites, finding the distribution of research areas is a hard task. To resolve this problem, we proposed a page segmentation algorithm for information extraction, which is called DeSeA (Delimiter based Segmentation Algorithm) as it divides a web page into coherent blocks using pre-defined delimiters and finds relevant blocks from them. DeSeA carries out on an Extended DOM Tree (EDT), transforming it into a block tree. Compared with existing algorithms, it considers delimiter's content splitting level, with higher-level delimiters having higher priority to segment a web page. Experiments show that DeSeA based information extraction algorithm fits commendably with DynamicView.
A Novel Chinese Web News Source Extraction Algorithm

Zhiming Liu -,Lu Liu -

DOI: https://doi.org/10.4156/jcit.vol6.issue8.12

2011-01-01

Journal of Convergence Information Technology

Abstract:Currently most of the information extraction algorithms are related to special web sites; it is difficult to meet the demand for extraction news sources from a number of different sites. According to characteristics of Chinese web news, this paper proposed an algorithm which can extract the news source from the different sites automatically based on rules and region recognition. Experimental evaluation indicates that this method has versatility and meets needs of practical applications.
Effective Page Segmentation Combining Pattern Analysis and Visual Separators for Browsing on Small Screens

Peifeng Xiang,Xin Yang,Yuanchun Shi

DOI: https://doi.org/10.1109/WI.2006.67

2006-01-01

Abstract:Page segmentation plays a key role in browsing on small screens. It breaks a large page into smaller segments according to their semantic relationships. Then, various approaches such as single column adaptation and thumbnail view with zooming links can be implemented based on these page segments. However, for current flexible Web pages, segmentation remains a challenging task. This paper proposes an effective automatic segmentation method which combining pattern analysis and visual separators. The basic idea is that a page's semantic structure is largely reflected by repeated continuous patterns and visual separators, which coincides with human's visual perception. The proposed method works in three steps: generating a refined tag tree from the DOM tree, recognizing and merging inexact patterns recursively, and segmenting the others by visual separators. Our experimental results show that the proposed method outperforms existing methods, especially for pages automatically generated from templates

A Novel Method for the Web Page Segmentation and Identification

Web Information Segmentation Method Based on DOM Structure Tree

A search-based Chinese word segmentation method.

Research on Feature Extraction for Content-based Chinese Web Pages Analysis

Constructing Novel Block Layouts for Webpage Analysis

An Application Of Lexical Semantics Of Chinese In Webpage Keyword Extraction Algorithm

Web Page Segmentation and Its Application for Web Information Crawling

A Web Page Segmentation Approach Using Visual Semantics.

A Novel Iterative Algorithm to Text Segmentation for Web Born-Digital Images.

Keywords extraction algorithm based on combined word and synset

Web Pages Information Retrieval Based on Keywords Cluster and Node Instance

Chinese Word Segmentation Evaluation Methodology Based on Web Search Engines

A Novel POS-Based Approach to Chinese News Topic Extraction from Internet

A novel graph partition based page segmentation algorithm

An Algorithm on Web Article Automatic Extraction Based on DOM Structure

A segmentation method for web page analysis using shrinking and dividing

Word Segmentation for Chinese Novels

Page Segmentation of Chinese Newspapers

DeSeA: A Page Segmentation Based Algorithm for Information Extraction

A Novel Chinese Web News Source Extraction Algorithm

Effective Page Segmentation Combining Pattern Analysis and Visual Separators for Browsing on Small Screens