A Review of Data Integration

Chen Yue

2004-01-01

Abstract:Firstly, the concepts and difficulties of data integration are introduced. Secondly, three typical data integration approaches: Schema Integration, Data Replication and Hybrid Approach are discussed and compared-Then, heterogeneity problem, a main difficulty of data integration, is explained in detail. Finally, further research fields are suggested.

What problem does this paper attempt to address?

Heterogeneous data integration: Challenges and opportunities

I Made Putrama,Péter Martinek

DOI: https://doi.org/10.1016/j.dib.2024.110853

2024-08-29

Abstract:Integrating multiple data source technologies is essential for organizations to respond to highly dynamic market needs. Although physical data integration systems have been considered to have better query processing systems, they pose higher implementation and maintenance costs. Meanwhile, virtual data integration has become an alternative topic that is increasingly attracting the attention of researchers in the current era of big data. Various data integration methodologies have been developed and used in various domains, processing heterogeneous data using various approaches. This review article aims to provide an overview of heterogeneous data integration research focusing on methodology and approaches. It surveys existing publications, highlighting key trends, challenges, and open research topics. The main findings are: (i) Research has been conducted in various domains. However, most focus on big data rather than specific study domains; (ii) researchers primarily focus on semantics challenges, and (iii) gaps still need to be addressed and related to integration issues involving semantics and unstructured data formats that must be thoroughly addressed. Furthermore, considering elements of cutting-edge technology, such as machine learning and data integration, about privacy concerns provides a chance for additional investigation. Finally, we provide insight into the potential for a broader review of integration challenges based on case studies.
Research on Key Problems in Data Integration System

LIU Min-chao,LIU Wei-dong

2006-01-01

Abstract:The notion of data integration was introduced and the aim of building data integration systems was presented.Some key problems in building data integration system were explained.They were building semantic relation,query rewriting,query optimizing and executing.Current researches on these problems were described in detail,and some thoughts for resolving them were proposed.
Metadata Management Technology in Data Warehouse

Li Zhao,Chunjie Yang,Bailin Hang

DOI: https://doi.org/10.1109/wcica.2004.1342307

2004-01-01

Abstract:Data integration is the kernel matter of information systems. Metadata describes the data structure and object of the data warehouse. The paper systemically analyzes the functions of metadata management in founding a data warehouse and implementing information integration. The some software tools and technique of metadata management are summarized and discussed. A new design approach and implementation of the metadata management system are presented. Finally, a sort of configuration of information integration systems is proposed.
IMPLEMENT DATA SHARING OVER NETWORK HETEROGENEOUS DATABASES BY DATA DIMENSION REDUCTION METHOD

Hua JIANG,Qi CHEN,Rui-Zhao YU

2000-01-01

Journal of Computer Research and Development

Abstract:The variety of databases structures as well as multiple data correlation on database index sets are major barriers on data sharing between heterogeneous databases. Most research work currently centralized on coherence access methods over heterogeneous databases, however, those methods can't establish a system in good scalability. In this paper, a new resolution is presented, which firstly find out general characteristics between many heterogeneous databases, and then the complexity of each database is reduced by using dimension reduction methods, so that data\|sharing becomes more flexible with a good scalability. In addition, the principal component views(PCV) reduce data correlation between shared\|databases.
Practicability of Dataspace Systems

Hamid Turab Mirza,Ling Chen,Gencai Chen

DOI: https://doi.org/10.4156/jdcta.vol4.issue3.23

2010-01-01

International Journal of Digital Content Technology and its Applications

Abstract:Nowadays there is rarely a scenario where all the data can be fit nicely into one relational database management system or any other single data model. In acknowledgement of this fact a new concept of Dataspaces was introduced, according to which a dataspace system is assumed to be a hybrid of a search engine, a databases management system, an information integration system and a data sharing system. However the concept was presented in a visionary way and its implementation in the real world scenario has opened up many complex research challenges. Moreover so far the efforts put forth by the research community are quite disparate and it is highly desirable to have a unified effort which would hopefully enable rapid progress. Furthermore due to very high end-user expectations of such systems there are a lot of challenges and problems that need to be resolved and much scope for future work remains.
Survey on Data Integration Technologies for Relational Data and Knowledge Graph

GAO Yun-Jun,GE Cong-Cong,GUO Yu-Xiang,CHEN Lu

DOI: https://doi.org/10.13328/j.cnki.jos.006808

2023-01-01

Journal of Software

Abstract:Recently, big data is considered a critical strategic resource by many countries and regions. However, difficult data circulation and insufficient data regulation commonly exist in the big data era, thereby leading to the serious phenomenon of data silos, poor data quality, and difficulty in unleashing the potential of data elements. This provokes researchers to explore data integration techniques for breaking data barriers, enabling data sharing, improving data quality, and activating the potential of data elements. Relational data and knowledge graphs, as two significant forms of data organization and storage, have been widely applied in real life. To this end, this study focuses on relational data and knowledge graphs to summarize and analyze the key technologies of data integration, including entity resolution, data fusion, and data cleaning. Finally, it prospects future research directions.
Research of Matching Technology in Data Integration

XU Feng,MAN Zhenmei,WANG Zhijian

DOI: https://doi.org/10.3969/j.issn.1000-3428.2006.06.014

2006-01-01

Abstract:With the development of network technology,it is necessary for information processing to adopt an unity accessing method and form for vast networks of large scale,heterogeneous and distributed computing resource,which is data integration’s work.Schema matching is a basic problem in the field of data integration;the main aspect of it concerns a match between global schema and local schema.This paper proposes solutions to solve these conflicts and integrate the schemas into a unified one.
Research on the Framework and Key Technology in Heterogeneous Data Integration

史万江,吴健

DOI: https://doi.org/10.3969/j.issn.1002-2279.2008.04.054

2008-01-01

Microprocessors

Abstract:With the development of information and management technology,more and more existing system and data need to be integrated to new applications.This Paper analyzes the method of heterogeneous data integration and the advantage of XML in data representation.Then a model of XML-based heterogeneous data integration is introduced,system architecture and implementation techniques are discussed in detail.
Some Theoretical Issues in Integration Studies

Li Guangjian

DOI: https://doi.org/10.3969/j.issn.0252-3116.2005.10.001

2005-01-01

Abstract:The paper reviews some theoretical issues in integration studies abroad. After the recapitulation of the three development stages of integration studies, which are resource organization-oriented integration, heterogeneous data sources-oriented integration and application-oriented integration, the paper summarizes the different viewpoints about forms and levels of integration, including dichotomies, trichotomies and quartation of integration. In addition to the brief introduction of approaches to integration: GAV,LAV,GLAV, BAV, and P2P, the paper discusses a generalized architecture for integration.
XML Based Exchange and Integration Techniques of Heterogeneous Data with Implementation

谢芳华,任午令,唐任仲

DOI: https://doi.org/10.3969/j.issn.1009-0134.2004.04.001

2004-01-01

Abstract:An XML technique based data exchange model was proposed, with using XML documentsas the base of data communication carrier, the information communication platform underInternet/Intranet environment was established and the software component was developed,which provides an effective low-cost method and tool to integrate and share different formatdata between heterogeneous databases. The method and tool has been applied to thedata integration and exchange of a supply chain management system.
Research on interoperability of enterprise system integration and application in distributed parts library

XU Xiang-rong,GU Xin-jian,YE Zuo-liang

DOI: https://doi.org/10.3969/j.issn.1009-0134.2006.05.004

2006-01-01

Abstract:Interoperability is the key problem needed to resolve in enterprise distributed computing and system integration.The demands of enterprise system integration and the Interoperability problem of heterogeneous distributed object technology were analysed.The method of using web services to transform the old application program and implement system integration was presented.The technology basic of web services interoperability and problems which result in web services interoperability were discussed.The limitation of web service in enterprise system integration was pointed.At last,an platform-independent web services interoperability example on system integration was provided about a heterogeneous distributed parts library system integration.
Review of Dempster-Shafer Method for Data Fusion

Cong-fu XU,Wei-dong GENG,Yun-he PAN

DOI: https://doi.org/10.3321/j.issn:0372-2112.2001.03.027

2001-01-01

Abstract:As one of the most important fusion methods,Dempster-Shafer(DS)method has been used widely in all kinds of data fusion systems.In this paper,the fusion model based on DS method,typical realization approaches of this method,and DS method′s extension are discussed comprehensively.Finally,commentaries on this fusion emthod from the viewpoint of data fusion are given,and it is pointed out that combining DS method with other related theories is an important trend.
Semantic Web meets Integrative Biology: a survey.

Huajun Chen,Tong Yu,Jake Y Chen

DOI: https://doi.org/10.1093/bib/bbs014

IF: 9.5

2012-01-01

Briefings in Bioinformatics

Abstract:Integrative Biology (IB) uses experimental or computational quantitative technologies to characterize biological systems at the molecular, cellular, tissue and population levels. IB typically involves the integration of the data, knowledge and capabilities across disciplinary boundaries in order to solve complex problems. We identify a series of bioinformatics problems posed by interdisciplinary integration: (i) data integration that interconnects structured data across related biomedical domains; (ii) ontology integration that brings jargons, terminologies and taxonomies from various disciplines into a unified network of ontologies; (iii) knowledge integration that integrates disparate knowledge elements from multiple sources; (iv) service integration that build applications out of services provided by different vendors. We argue that IB can benefit significantly from the integration solutions enabled by Semantic Web (SW) technologies. The SW enables scientists to share content beyond the boundaries of applications and websites, resulting into a web of data that is meaningful and understandable to any computers. In this review, we provide insight into how SW technologies can be used to build open, standardized and interoperable solutions for interdisciplinary integration on a global basis. We present a rich set of case studies in system biology, integrative neuroscience, bio-pharmaceutics and translational medicine, to highlight the technical features and benefits of SW applications in IB.
The Research On Customer Data Integration Technology Based On Web Services

Js Su,Xj Gu,Zg Bao

2005-01-01

Abstract:Along with the popularization of the automobiles, customer management and supply chain management of automobile parts is becoming more and more important. Customer data is an important resource of automobile parts company,it should be shared and integrated for the forecast of customer and company decision-making. The problem of distributed data transmission can be solved by using XML and Web services. It provides a cooperative tool for different application system in different places. J2EE can be used to solve the problem of data integration. In this paper, the transmission of customer data through network is investigated, the method for data collection is analyzed, the architecture system of the data integration is designed and analyzed, and the data collection is investigated.
Research On Data Analysis System Based On Xml (Id : 3-007)

Chen Guohai,Chen Jixi,Gu Xinjian,Zhan Hongfei

2006-01-01

Abstract:Historically, data analysis and data analysis system is the research hotspot of the manufacturing industry, and also the key issue of the informatization of modern enterprises. However, most of the data analysis systems used recently can't resolve the transformation and integration of the data sources efficiently, especially the isomerous data sources. The data expressing also seems lack of diversity. In the view of data transformation, integration and data expressing combining with the characteristic of the XML, a new data analysis system based on XML is proposed. Then the framework, hierarchical structure and the crisis problems of the system are analyzed in detail and that it has a promise future is revealed.
Human-in-the-loop Data Integration

Guoliang Li

DOI: https://doi.org/10.14778/3137765.3137833

IF: 2.5

2017-01-01

Proceedings of the VLDB Endowment

Abstract:Data integration aims to integrate data in different sources and provide users with a unified view. However, data integration cannot be completely addressed by purely automated methods. We propose a hybrid human-machine data integration framework that harnesses human ability to address this problem, and apply it initially to the problem of entity matching. The framework first uses rule-based algorithms to identify possible matching pairs and then utilizes the crowd to refine these candidate pairs in order to compute actual matching pairs. In the first step, we propose similarity-based rules and knowledge-based rules to obtain some candidate matching pairs, and develop effective algorithms to learn these rules based on some given positive and negative examples. We build a distributed in-memory system DIMA to efficiently apply these rules. In the second step, we propose a selection-inference-refine framework that uses the crowd to verify the candidate pairs. We first select some "beneficial" tasks to ask the crowd and then use transitivity and partial order to infer the answers of unasked tasks based on the crowdsourcing results of the asked tasks. Next we refine the inferred answers with high uncertainty due to the disagreement from the crowd. We develop a crowd-powered database system CDB and deploy it on real crowdsourcing platforms. CDB allows users to utilize a SQL-like language for processing crowd-based queries. Lastly, we provide emerging challenges in human-in-the-loop data integration.
Semantic data integration and querying: a survey and challenges

Maroua Masmoudi,Sana Ben Abdallah Ben Lamine,Mohamed Hedi Karray,Bernard Archimede,Hajer Baazaoui Zghal

DOI: https://doi.org/10.1145/3653317

IF: 16.6

2024-03-23

ACM Computing Surveys

Abstract:Digital revolution produces massive, heterogeneous and isolated data. These latter remain underutilized, unsuitable for integrated querying and knowledge discovering. Hence the importance of this survey on data integration which identifies challenging issues and trends. First, an overview of the different generations and basics of data integration is given. Then, semantic data integration is focused, since it semantically links data allowing wider insights and decision-making. More than thirty works are reviewed. The goal is to help analysts to identify relevant criteria to compare then choose among semantic data integration approaches, focusing on the category (materialized, virtual or hybrid) and querying techniques.

computer science, theory & methods
Research on Enterprise Application Integration Based on Web Services

XU Xiangrong,GU Xinjian,YE Zuoliang

DOI: https://doi.org/10.3969/j.issn.1005-2402.2006.09.013

2006-01-01

Abstract:The sharing and inner operating between different applications have become increasingly insistent demands in the process of implementing enterprise informationization. Firstly, the current situation of the existed information systems in the enterprise was analyzed. Secondly, the limitations of the traditional integration techniques were discussed. Thirdly, the advantage of Web services was described and an enterprise application integration framework based on Web services was presented. Finally, an integration example was provided about system integration between product configuration and parts library.
A Survey of Heterogeneous CAD System Integration

Shuming Gao,Fazhi He

2009-01-01

Abstract:The integration of heterogeneous CAD systems plays a key role in realizing collaborative product development. Firstly, the integration methods based on geometry data exchange are discussed, and the inherent problems and shortages compared with other methods are analyzed. Secondly, the integration approaches based on parametric feature information exchange are summarized, with focusing on the key technical issues, typical research works and commercial tools. Thirdly, the online integration technique based on operation commands exchange is introduced. Finally, future trends of heterogeneous CAD system integration are prospected.
A Review of Design Intelligence: Progress, Problems, and Challenges

Yong-chuan Tang,Jiang-jie Huang,Meng-ting Yao,Jia Wei,Wei Li,Yong-xing He,Ze-jian Li

DOI: https://doi.org/10.1631/fitee.1900398

2019-01-01

Abstract:Design intelligence is an important branch of artificial intelligence (AI), focusing on the intelligent models and algorithms in creativity and design. In the context of AI 2.0, studies on design intelligence have developed rapidly. We summarize mainly the current emerging framework of design intelligence and review the state-of-the-art techniques of related topics, including user needs analysis, ideation, content generation, and design evaluation. Specifically, the models and methods of intelligence-generated content are reviewed in detail. Finally, we discuss some open problems and challenges for future research in design intelligence.

A Review of Data Integration

Heterogeneous data integration: Challenges and opportunities

Research on Key Problems in Data Integration System

Metadata Management Technology in Data Warehouse

IMPLEMENT DATA SHARING OVER NETWORK HETEROGENEOUS DATABASES BY DATA DIMENSION REDUCTION METHOD

Practicability of Dataspace Systems

Survey on Data Integration Technologies for Relational Data and Knowledge Graph

Research of Matching Technology in Data Integration

Research on the Framework and Key Technology in Heterogeneous Data Integration

Some Theoretical Issues in Integration Studies

XML Based Exchange and Integration Techniques of Heterogeneous Data with Implementation

Research on interoperability of enterprise system integration and application in distributed parts library

Review of Dempster-Shafer Method for Data Fusion

Semantic Web meets Integrative Biology: a survey.

The Research On Customer Data Integration Technology Based On Web Services

Research On Data Analysis System Based On Xml (Id : 3-007)

Human-in-the-loop Data Integration

Semantic data integration and querying: a survey and challenges

Research on Enterprise Application Integration Based on Web Services

A Survey of Heterogeneous CAD System Integration

A Review of Design Intelligence: Progress, Problems, and Challenges