Effective Metadata Extraction from Irregularly Structured Web Content

Baoyao Zhou,Wei Liu,Yu Yang,Weichun Wang,Ming Zhang

2008-01-01

Abstract:© Effective Metadata Extraction from Irregularly Structured Web Content Baoyao Zhou, Wei Liu, Yu Yang, Weichun Wang, Ming Zhang

What problem does this paper attempt to address?

Exploiting Multi-Category Characteristics and Unified Framework to Extract Web Content

Jingwei Zhang,Qian Wang,Qing Yang,Rui Zhou,Yanchun Zhang

DOI: https://doi.org/10.1007/s41019-018-0067-3

2018-01-01

Data Science and Engineering

Abstract:Extracting web content is to obtain the required data embedded in web pages, usually including structured records, such as product information, and text content, such as news. Web pages use a large number of HTML tags to organize and to present various information. Both knowing little about the structures of web pages and mixing kinds of information in web pages are making the extraction process very challenging to guarantee extraction performance and extraction adaptability. This study proposes a unified web content extraction framework that can be applied in various web environments to extract both structured records and text content. First, we construct a characteristic container to hold kinds of characteristics related with extraction objectives, including visual text information, content semantics(instead of HTML tag semantics), web page structures, etc. Second, the above characteristics are integrated into an extraction framework for extraction decisions on different web sites. Especially, we put forward different strategies, path aggregation for extracting text content and HMM model for structured records, to locate the extraction area by exploiting both those extraction characteristics. Comparative experiments on multiple web sites with popular extraction methods, including CETR, CETD and CNBE, show that our proposed extraction method can provide better extraction precision and extraction adaptability.
Extracting Web Content by Exploiting Multi-Category Characteristics

Qian Wang,Qing Yang,Jingwei Zhang,Rui Zhou,Yanchun Zhang

DOI: https://doi.org/10.1007/978-3-319-68786-5_19

2017-01-01

Abstract:Extracting web content aims at separating web content from web pages since web content is organized and presented by different HTML templates and is surrounded by various information. Knowing little about template structures and noise information before extraction, the variability of page templates, etc., make the extraction process very challenging to guarantee extraction precision and extraction adaptability. This study proposes an effective web content extraction method for various web environments. To ensure extraction performance, we exploited three kinds of characteristics, visual text information, content semantics(instead of HTML tag semantics) and web page structures. These characteristics are then integrated into an extraction framework for extraction decisions for different websites. Comparative experiments on multiple web sites with two popular extraction methods, CETR and CETD, show that our proposed extraction method outperforms CETR on precision when keeping the same advantage on recall, and also gains 4% improvement over CETD on the average F1-score; especially, our method can provide better extraction performance when facing short content than CETD, and presents a better extraction adaptability.
Web mining of relations from XML and construct database schema

Zhou Xu,Pan Xuezeng,Ren Yu

DOI: https://doi.org/10.1109/CIMCA.2006.233

2007-01-01

Abstract:Increasing amount of commercial data is presented in XML format for exchanging or publishing on the web. It is emerging as a new standard for information representation and exchanging over the Internet. How to retrieve valuable information from XML documents on the web is a new challenge to data mining research. Compared with relational database, XML data in documents is stored as file with tree logical structure inside, it results in lower efficiency and performance in directly querying data. So it is still necessary to transform data into database (warehouse) for data mining afterwards. In this paper, we present a scheme to analyze relation of elements in XML on the web, and construct relational database schema based on the analysis. During the process, there would be a worthy accessory product - a glossary, which can facilitate the process of data mining warehouse designing and building. © 2006 IEEE.
Web Information Segmentation Method Based on DOM Structure Tree

ZHOU Jian,TANG Jin,LUO Bin

DOI: https://doi.org/10.3969/j.issn.1006-2475.2013.10.056

2013-01-01

Abstract:Correct extraction and segmentation of Web information is significant to text information mining.The paper proposes and achieves a method which can get informative information from Web page and be able to follow the correct segmentation of the original text.The method first uses page layout tag < table > and < div > to build a DOM structure tree,and then uses the nested relations of the layout label,that the DOM structure tree reflects to choose the content blocks,extract text information correctly,and finally achieves information segment of the body through the manipulation of some special tags.The experimental results prove that this method is easy to realize and high efficiency and it can automatically extract informative message and section accurately.
Automatic Extraction Of Commodity Attributes On Webpages Based On Hierarchical Structure

Zhi Yu,Meiyan Li,Wei Wang,Can Wang

DOI: https://doi.org/10.1142/9789814689007_0046

2015-01-01

Abstract:Everygood in e-commerce websites contains a lot of commodity attributes; even some of them are hidden behind the user's dynamic interaction. Finding them out for further analysis is a challenging topic. How to extract the commodity attributes effectively has been an attracting research topic in this area recently. Previous works cost too much calculation to deal this problem effectively, since they involve complex operations such as semantic analysis to extract hidden attributes. In this article, we analyze the relationship between different URL terms and commodity attributes, discuss with the hierarchical structure of the query results and propose a new Commodity Attributes Extraction algorithm based on Single-Layer method to find hidden attributes denoted by the URL terms. Using these URL terms, we can find out hidden attributes from a new query result page easily by analyzing its URL. Experiment result demonstrates the effectiveness of our algorithm.
Building Web Page Logical Structure Model towards Effective Metadata Extraction

Baoyao Zhou,Ming Zhang

DOI: https://doi.org/10.1109/APWeb.2010.81

2010-01-01

Abstract:Web pages are typical semi-structure data. Some tree-based models have been proposed to describe the semantic content structure of web pages in order to facilitate further content analysis. However, most existing models only present the segmentation hierarchy of content blocks rather than the semantic relationships among them. In this work, we propose a novel web page semantic structure model, called Logical Structure Model. It can present more comprehensive structure information of web pages. Based on this model, the hidden patterns in web content can be revealed easier. The proposed model has been used to facilitate identifying course metadata in our Online Course Organization project, which aims to build an online course portal to serve the course information obtained from the Web.
Attributes extraction of Deep Web query interface based on DOM

Shi Long,Qiang Baohua,He Qian,Wu Chunming,Chen Chao

DOI: https://doi.org/10.3969/j.issn.1673-808X.2012.06.010

2012-01-01

Abstract:Query interface schema extraction is the precondition of Deep Web data integration.Generally query interface schema consists of a set of domain-related attributes,and one attribute is formed by a single element or a combination of multi-elements.The current researches on attribute extraction are mostly based on the single element fashion,and those multi-elements based are few.Aiming at the case of multi-elements attribute extraction,a DOM-based method for query interface schema extraction is proposed.This method parses query interface to become a DOM and extracts the form elements base on the corresponding DOM nodes.The method employs two-phase clustering algorithms to cluster the form elements,mines the combination relationship of them and combines elements to realize attributes extraction.This method has a favorable performance at both single-element and multi-elements attribute extraction.The experimental result shows that this method is effective.
An Agent based Approach towards Metadata Extraction, Modelling and Information Retrieval over the Web

Zeeshan Ahmed,Detlef Gerhard

DOI: https://doi.org/10.48550/arXiv.1008.1333

2010-08-07

Abstract:Web development is a challenging research area for its creativity and complexity. The existing raised key challenge in web technology technologic development is the presentation of data in machine read and process able format to take advantage in knowledge based information extraction and maintenance. Currently it is not possible to search and extract optimized results using full text queries because there is no such mechanism exists which can fully extract the semantic from full text queries and then look for particular knowledge based information.

Artificial Intelligence
Web Content Extraction & Its Data Management Method

ZHANG Cheng-hong,XIAO Jun-jian,ZHANG Cheng

DOI: https://doi.org/10.3969/j.issn.0427-7104.2001.02.012

2001-01-01

Abstract:With the development of Internet and its relative technology, the WWW has become the largest information area. For the enterprise or the individual, Web becomes the main information source gradually. However, because of too many web sites and the information overflow resulting from this, it is more and more difficult to obtain useful information. Search engines only provide the scope of the searching information, and the concrete information must be looked up carefully by oneself. Because Web information is non-strutured or semi-structured, the analysis tool can't be used to analyze it directly. So it is necessary to advance a method of extracting the Web content automatically and structuring the Web data to simplify the process of obtaining information and facilitate the information analysis. This paper will describe this in detail.
A Rule-Based Information Extraction System for Human-Readable Semi-Structured Scientific Documents

Gang Chen,Baoran An,Sifeng Zeng

DOI: https://doi.org/10.1109/iccsnt.2015.7490711

2015-01-01

Abstract:The explosion of data over the past twenty years has fostered a huge amount of research in processing semi-structured documents like HTML and XML documents on Web. Nevertheless, the explosion of semi-structured documents that originate from outside the Web domain is more challenging. The data of semi-structured documents are everywhere: in scientific research reports, official journals, electronic health records and any records from other domains. Different from HTML and XML, semi-structured record is usually human-readable and has its own internal schema. On account of the incredible increase of data volumes, the traditional methods of studying on these data cannot meet the needs of high performance and flexible access. Relational database technique is a good alternative technique for managing and organizing data. In this paper, we aim at (1) revealing the semantic structure in human-readable scientific records (HSR), and (2) presenting a transformation system that converts HSR into structured relational database. Our approach identifies the relevant logical sub-structures and objects in HSR according to specific templates. We don't target "the best" methods but provide an available framework for facilitating the semi-structured HSR analysis tasks. The evaluation to the system shows its ability to enable the efficient transformation of HSR and therefore contributes to the increasing corpus of semi-structured HSR documents.
Military Web Information Extraction Incorporating Resource Metadata Distribution

Hui Li,Jiahao Zhou,Wei Sun

DOI: https://doi.org/10.1109/insai56792.2022.00019

2022-01-01

Abstract:We analyze the relationship between the metadata distribution of military resources and the distribution of the web content in news, events, and resources in three major sections of military web stations, and thus we propose a method for extracting information from military web pages by integrating the metadata distribution and the basic features of military web pages to increase the difference between the content density and the noisy text density, to extract the content of military web pages. To verify the practical effectiveness of the proposed method in extracting information from military web pages, 69,379 web pages in the three major sections were randomly selected for the experiments in this work. The experimental results show that compared with the CETR, VIPS, SRV, and WNISK algorithms, the proposed method in this paper achieves better content extraction performance, with F-values of 98.21%, 97.11%, and 97.46% for the extraction of web page information in the three major sections respectively.
Extracting Objects from the Web.

Zaiqing Nie,Fei Wu,Ji-Rong Wen,Wei-Ying Ma

DOI: https://doi.org/10.1109/ICDE.2006.69

2006-01-01

Abstract:Extracting and integrating object information from the Web is of great significance for Web data management. The existing Web information extraction techniques cannot provide satisfactory solution to the Web object extraction task since objects of the same type are distributed in diverse Web sources, whose structures are highly heterogeneous. In this paper, we propose a novel approach called Object-Level Information Extraction (OLIE) to extract Web objects. This approach extends a classic information extraction algorithm, Conditional Random Fields (CRF), by adding Web-specific information. The experimental results show OLIE can significantly improve the Web object extraction accuracy.
EGA:An Algorithm for Automatic Semi-structured Web Documents Extraction

Liyu Li,Shiwei Tang,Dongqing Yang,Tengjiao Wang,Zhihua Su

DOI: https://doi.org/10.1007/978-3-540-24571-1_69

2004-01-01

Abstract:With the fast expansion of World Wide Web, more and more semi-structured web documents appear on the web. In this paper, we study how to extract information from the semi-structured web documents by automatically generated wrappers. To automate the wrapper generation and the data extraction process, we develop a novel algorithm EGA (EPattern Generation Algorithm) to conduct the extraction pattern based on the local structural context features of the web documents. These optimal or near optimal extraction patterns are described in XPath language. Experimental results on RISE and our own data sets confirm the feasibility of our approach.
Data extraction from web pages based on structural-semantic entropy.

Xiaoqing Zheng,Yiling Gu,Yinsheng Li

DOI: https://doi.org/10.1145/2187980.2187991

2012-01-01

Abstract:Most of today's web content is designed for human consumption, which makes it difficult for software tools to access them readily. Even web content that is automatically generated from back-end databases is usually presented without the original structural information. In this paper, we present an automated information extraction algorithm that can extract the relevant attribute-value pairs from product descriptions across different sites. A notion, called structural-semantic entropy, is used to locate the data of interest on web pages, which measures the density of occurrence of relevant information on the DOM tree representation of web pages. Our approach is less labor-intensive and insensitive to changes in web-page format. Experimental results on a large number of real-life web page collections are encouraging and confirm the feasibility of the approach, which has been successfully applied to detect false drug advertisements on the web due to its capacity in associating the attributes of records with their respective values.
Statistical Web Object Extraction

Jun Zhu,Zaiqing Nie,Bo Zhang

DOI: https://doi.org/10.4018/978-1-60566-010-3.ch283

2009-01-01

Abstract:The World Wide Web is a vast and rapidly growing repository of information. There are various kinds of objects, such as products, people, conferences, and so on, embedded in both statically and dynamically generated Web pages. Extracting the information about real-world objects is a key technique for Web mining systems. For example, the object-level search engines, such as Libra (http://libra.msra.cn) and Rexa (http://rexa.info), which help researchers find academic information like papers, conferences and researcher’s personal information, completely rely on structured Web object information. However, how to extract the object information from diverse Web pages is a challenging problem. Traditional methods are mainly template-dependent and thus not scalable to the huge number of Web pages. Furthermore, many methods are based on heuristic rules. So they are not robust enough. Recent developments in statistical machine learning make it possible to develop advanced statistical Web object extraction models. One key difference of Web object extraction from traditional information extraction from natural language text documents is that Web pages have plenty of structure information, such as two-dimensional spatial layouts and hierarchical vision tree representation. Statistical Web object extraction models can effectively leverage this information with properly designed statistical models. Another challenge of Web object extraction is that many text contents on Web pages are not regular natural language sentences. They have some structures but are lack of natural language grammars. Thus, existing natural language processing (NLP) techniques are not directly applicable. Fortunately, statistical Web object extraction models can easily merge with statistical NLP methods which have been the theme in the field of natural language processing during the last decades. Thus, the structure information on Web pages can be leveraged to help process text contents, and traditional NLP methods can be used to extract more features. Finally, the Web object extraction from diverse and large-scale Web pages provides a valuable and challenging problem for machine learning researchers. To nicely solve the problem, new learning methodology and new models (Zhu et al., 2007b) have to be developed.
Automating the extraction of data from HTML tables with unknown structure

David W. Embley,Cui Tao,Stephen W. Liddle

DOI: https://doi.org/10.1016/j.datak.2004.10.004

2005-07-01

Abstract:Data on the Web in HTML tables is mostly structured, but we usually do not know the structure in advance. Thus, we cannot directly query for data of interest. We propose a solution to this problem based on document-independent extraction ontologies. Our solution entails elements of table understanding, data integration, and wrapper creation. Table understanding allows us to find tables of interest within a Web page, recognize attributes and values within the table, pair attributes with values, and form records. Data-integration techniques allow us to match source records with a target schema. Ontologically specified wrappers allow us to extract data from source records into a target schema. Experimental results show that we can successfully locate data of interest in tables and map the data from source HTML tables with unknown structure to a given target database schema. We can thus “directly” query source data with unknown structure through a known target schema.

computer science, information systems, artificial intelligence
Statistical Entity Extraction From the Web.

Zaiqing Nie,Ji-Rong Wen,Wei-Ying Ma

DOI: https://doi.org/10.1109/JPROC.2012.2191369

IF: 20.6

2012-01-01

Proceedings of the IEEE

Abstract:There are various kinds of valuable semantic information about real-world entities embedded in webpages and databases. Extracting and integrating these entity information from the Web is of great significance. Comparing to traditional information extraction problems, web entity extraction needs to solve several new challenges to fully take advantage of the unique characteristic of the Web. In this...
Automatic Extraction and Cluster Analysis of Natural Disaster Metadata Based on the Unified Metadata Framework

Zongmin Wang,Xujie Shi,Haibo Yang,Bo Yu,Yingchun Cai

DOI: https://doi.org/10.3390/ijgi13060201

IF: 3.4

2024-06-15

ISPRS International Journal of Geo-Information

Abstract:The development of information technology has led to massive, multidimensional, and heterogeneously sourced disaster data. However, there's currently no universal metadata standard for managing natural disasters. Common pre-training models for information extraction requiring extensive training data show somewhat limited effectiveness, with limited annotated resources. This study establishes a unified natural disaster metadata standard, utilizes self-trained universal information extraction (UIE) models and Python libraries to extract metadata stored in both structured and unstructured forms, and analyzes the results using the Word2vec-Kmeans cluster algorithm. The results show that (1) the self-trained UIE model, with a learning rate of 3 × 10-4 and a batch_size of 32, significantly improves extraction results for various natural disasters by over 50%. Our optimized UIE model outperforms many other extraction methods in terms of precision, recall, and F1 scores. (2) The quality assessments of consistency, completeness, and accuracy for ten tables all exceed 0.80, with variances between the three dimensions being 0.04, 0.03, and 0.05. The overall evaluation of data items of tables also exceeds 0.80, consistent with the results at the table level. The metadata model framework constructed in this study demonstrates high-quality stability. (3) Taking the flood dataset as an example, clustering reveals five main themes with high similarity within clusters, and the differences between clusters are deemed significant relative to the differences within clusters at a significance level of 0.01. Overall, this experiment supports effective sharing of disaster data resources and enhances natural disaster emergency response efficiency.

geography, physical,remote sensing,computer science, information systems
An Analysis of Structured Data on the Web

Nilesh Dalvi,Ashwin Machanavajjhala,Bo Pang

DOI: https://doi.org/10.48550/arXiv.1203.6406

2012-03-29

Abstract:In this paper, we analyze the nature and distribution of structured data on the Web. Web-scale information extraction, or the problem of creating structured tables using extraction from the entire web, is gathering lots of research interest. We perform a study to understand and quantify the value of Web-scale extraction, and how structured information is distributed amongst top aggregator websites and tail sites for various interesting domains. We believe this is the first study of its kind, and gives us new insights for information extraction over the Web.

Databases
Exploiting Collective Hidden Structures In Webpage Titles For Open Domain Entity Extraction

Wei Song,Shiqi Zhao,Chao Zhang,Hua Wu,Haifeng Wang,Lizhen Liu,Hanshi Wang

DOI: https://doi.org/10.1145/2736277.2741107

2015-01-01

Abstract:We present a novel method for open domain named entity extraction by exploiting the collective hidden structures in webpage titles. Our method uncovers the hidden textual structures shared by sets of webpage titles based on generalized URL patterns and a multiple sequence alignment technique. The highlights of our method include: 1) The boundaries of entities can be identified automatically in a collective way without any manually designed pattern, seed or class name. 2) The connections between entities are also discovered naturally based on the hidden structures, which makes it easy to incorporate distant or weak supervision. The experiments show that our method can harvest large scale of open domain entities with high precision. A large ratio of the extracted entities are long-tailed and complex and cover diverse topics. Given the extracted entities and their connections, we further show the effectiveness of our method in a weakly supervised setting. Our method can produce better domain specific entities in both precision and recall compared with the state-of-the-art approaches.

Effective Metadata Extraction from Irregularly Structured Web Content

Exploiting Multi-Category Characteristics and Unified Framework to Extract Web Content

Extracting Web Content by Exploiting Multi-Category Characteristics

Web mining of relations from XML and construct database schema

Web Information Segmentation Method Based on DOM Structure Tree

Automatic Extraction Of Commodity Attributes On Webpages Based On Hierarchical Structure

Building Web Page Logical Structure Model towards Effective Metadata Extraction

Attributes extraction of Deep Web query interface based on DOM

An Agent based Approach towards Metadata Extraction, Modelling and Information Retrieval over the Web

Web Content Extraction & Its Data Management Method

A Rule-Based Information Extraction System for Human-Readable Semi-Structured Scientific Documents

Military Web Information Extraction Incorporating Resource Metadata Distribution

Extracting Objects from the Web.

EGA:An Algorithm for Automatic Semi-structured Web Documents Extraction

Data extraction from web pages based on structural-semantic entropy.

Statistical Web Object Extraction

Automating the extraction of data from HTML tables with unknown structure

Statistical Entity Extraction From the Web.

Automatic Extraction and Cluster Analysis of Natural Disaster Metadata Based on the Unified Metadata Framework

An Analysis of Structured Data on the Web

Exploiting Collective Hidden Structures In Webpage Titles For Open Domain Entity Extraction