Abstract:In big data era, information integration often requires abundant data extracted from massive data sources. Due to a large number of data sources, data source selection plays a crucial role in information integration, since it is costly and even impossible to access all data sources. Data Source selection should consider both efficiency and effectiveness issues. For efficiency, the approach should scale to large data source amount. From effectiveness aspect, data quality and overlapping of sources are to be considered. In this paper, we study source selection problem in Big Data and propose methods which can scale to datasets with up to millions of data sources and guarantee the quality of results. Motivated by this, we propose a new metric taking the expected number of true values a source can provide as a criteria to evaluate the contribution of a data source. Based on our proposed index, we present a scalable algorithm and two pruning strategies to improve the efficiency without sacrificing precision. Experimental results on both real world and synthetic data sets show that our methods can select sources providing a large proportion of true values efficiently and can scale to massive data sources.

Data source selection with similar theme in Deep Web integrated system

DEEP WEB DATA SOURCES CLASSIFICATION BASED ON TEXT VSM OF QUERY INTERFACE

Research on Deep Web Classification Based on Domain Feature Text

The Ranking of Deep Web Sources Based on Data Quality

Effective Approach to Deep Web Entries Identification

Ontology-Assisted Deep Web Source Selection

A Survey of Deep Web Data Integration

Approximate Content Summary For Database Selection In Deep Web Data Integration

Method for Database Selection of Deep Web Based on the Dynamic Learning Algorithm

Domain-oriented Deep Web Data Sources' Discovery and Identification

Pattern Matching Method for Deep Web Interface Integration

Data source selection for information integration in big data era

Domain-Specific Deep Web Sources Discovery

A Machine Learning Approach Classification of Deep Web Sources

THE STUDY OF INTRANET SEARCH ENGINE BASED ON LUCENE

The Method of Extracting Refined Semantic Similarity among Objects of Heterogeneous Information Sources

Research on detecting and classifying Deep Web interfaces

Efficient Quality-Driven Source Selection from Massive Data Sources

A Deep Web Data Integration System for Job Search

A New Web Mining Data Integration Model Based on XML

A Holistic Solution for Duplicate Entity Identification in Deep Web Data Integration