Abstract:The large-scale dimension table connection technology in the distributed environment is one of the key technologies in online big data analysis, which is widely used in real-time recommendation, real-time analysis and other fields. The dimension table connection indicates that stream data and dimension tables stored offline will be connected to be processed accordingly. Firstly, this paper studies the existing dimension table connection technology and surveys the design of relevant optimization technologies and mainstream distributed engines. The traditional way of improving performance is optimizing dimension table data query. Traditional optimization is limited to the scale of the dimension table and data stream rate. Secondly, in terms of the inefficient usage of existent optimization technologies’ consideration for the whole cluster in distributed environment, this paper puts forward a computing model suitable for hybrid calculation of offline batch data and real-time stream data. This paper proposes a method of dimension table associated data cache, which reads dimension table data from a single node and distributes and calculates the data after it is segmented. This paper also optimizes the computing logic of dimension table connection so that a higher-level scale of the dimension table is applied, and the data connection limitation is overcome. Finally, the dimension table connection technology in this paper and the traditional dimension table connection technology have been implemented in Apache Flink. The optimization for dimension table connection of distributed stream computing in this paper has been verified via the experiment of comparing throughput and latency based on dataset from Double 11 Shopping Carnival of Alibaba Group.

OPTIMIZATION OF DIMENSION TABLE SCHEMA IN DATA WAREHOUSE

Requirement-Based Data Cube Schema Design

Data Warehouse Native Feature Based OLAP Querying with Keywords

MODELING AND QUERY OF DIMENSION FOR DATA WAREHOUSES

Simple and economical remedy for OLAP systems that can not effectively deal with ragged dimensions

Optimized Design of Materialized Views in a Real-Life Data Warehousing Environment

A Clustered Dwarf Structure to Speed Up Queries on Data Cubes

Generating Multidimensional Schemata from Relational Aggregation Queries

On Improving GDSS Data Warehouse

An Adaptive Approach to Schema Classification for Data Warehouse Modeling

Based on the Data Warehouse Dimensional Modeling Optimization

Development of Multiply Dimensions in Data Warehouse

Optimization for Large-Scale Dimension Table Connection Technology in Distributed Environment

Partitioned Dimension: Modeling the Numerical Dimension in Data Warehouse.

Cost-Based Optimization Of Logical Partitions For A Query Workload In A Hadoop Data Warehouse

Research on Data Cube Technology of Dwarf Based Semantic OLAP

A Distributed Multidimensional Data Model Of Data Warehouse

Opti Zation Of Spark S O Age Solutions

An Improved Storage Algorithm for Multidimensional Data Cube

Meta Galaxy: A Flexible and Efficient Cube Model for Data Retrieval in OLAP

Cost-Driven Storage Schema Selection for XML