Abstract:The growth of information technology has opened the gate for the organizations to maintain their data in various forms and at various volumes. This increases the volume and dimension of data being maintained. However, they store their data in their data servers or in cloud environment. Such data have been used to generate various intelligence to support various problems. To support such analysis process, different data have been used and the big data comes to play in this part. Optimizing techniques to help improve the process of ETL could greatly help in real-time analysis of data. ETL optimization could be achieved through several factors simplest being increasing the frequency of the process. Other ways to achieve optimization are through the use of various architectures, programming models, intelligence in transformation and security. To improve the performance of ETL, an efficient dynamic multi-variant relational intelligent ETL framework has been presented in this article. The distributed approach maintains various ontology's and data dictionaries which have been dynamically updated by different threads of ETL process. Initially, the process is start by applying the extraction process which extracts the data from different sources and finds set of dimensions and their characteristics. Such data extracted have been verified over the data dictionary. Further, the relational score has been measured for each data source with the existing one. Similarly, the method computes the value of multi-variant relational similarity (MVRS) for the data obtained from a single source. This will be performed by different threads of ETL process. According to the value of MVRS, the method performs map reduce and merging of data. According to the value of MVRS the method selects the data node and merges the data to store in the data warehouse. The threads of ETL are capable of reading the changes in data dictionaries and ontology's to iterate the process of transformation and loading. The method improves the performance of ETL with least time complexity and higher performance.

ETL-based Solution for Integrating Data in Financial Domain

Synthetic Integration Solution for Enterprise Application

EXPRESS Driven XML Data Used in CAx/PDM Integration Framework

Data Integration Platform for Small-Medium Enterprise Process Integration

ETC-Based Methodology on Data Processing Validation

Model Mapping Techniques for STEP and Its Application in Open Environment

Dynamic Cross-System Workflow Integration Based on MAS

A novel agent-based parallel ETL system for massive data

Data integration from traditional to big data: main features and comparisons of ETL approaches

Integrating IDEF3 and discrete event approach for business process modeling and simulation

The service architecture of Integrated Financial Enterprise

Complex Instant Flow Integration Model and Its Application in Insurance

XML Based Exchange and Integration Techniques of Heterogeneous Data with Implementation

Open structure model of data integration for CAPP system

Integrated Product Modeling Based On Web Services In Distributed Environment

Research on technology of product data exchange for internet-based distributed integrated manufacturing system

A Novel Xml-Based Power Resource Modeling Framework for Power System Heterogeneous Data Integrating

METL: a modern ETL pipeline with a dynamic mapping matrix

Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Data Extraction, Transformation, and Loading Process Automation for Algorithmic Trading Machine Learning Modelling and Performance Optimization

Extraction Transformation and Loading (ETL) of Data Using ETL Tools