Abstract:With the rapid development of the Internet and communication technology,massive data has been accumulated in many applications,for example,Internet information security management,storage and transaction of large scale scientific compute middle results(nuclear simulation,meteorology analysis),real-time monitoring system information(sensor network,astronomy meteorology monitoring system,secondary planet monitoring system),and analysis and research based on Internet information.Increasing data volumes and real time data-loading requirement pose enormous challenges to data-loading techniques.A data loading system in real time,the IMIL(Internet monitoring information loader) is presented which is used in real-time Internet monitoring information system.IMIL consists of an extensible fault-tolerant hardware architecture,an efficient algorithm for bulk data loading using SQL*Loader and an exchange partition mechanism,optimized parallelism,and guidelines for system tuning.Performance studies show the positive effects of these techniques with loading speed of every cluster,increasing from 220 million records per day to 1.2 billion per day,and achieving the top loading speed of 6TB data when 10 clusters are in parallel.This framework offers a promising approach for loading other large and complex databases.

Research of Distribute Massive Text Data Index and Retrieval System

Research of Massive Internet Text Data Real-Time Loading and Index System

Implementation of large-scale distributed information retrieval system

Architecture Design of Wide-area Distributed Real-time Database in Grid Dispatching Control System

The Design and Implementation of an Intelligent Distributed Text Retrieval System

Research on Mass Process-Planning Data Storing and Parallel Accessing in Large-Scale Manufacturing Enterprise.

Research on Mass Process-Planning Data Storing and Parallel Accessing in Large-Scale Manufacturing Enterprise

A Two-Tier Distributed Full-Text Indexing System

Research and Implementation of the Muti-Task-Parallel Scheduling Loading Technology of Massive Text Data

Distributed Retrieval System for Large Scale Text Collection

Research and Implementation on Compression, Encryption, Storage and Retrieval System for Massive Data

An Efficient Strategy of Building Distributed Index Based on Lucene.

Toward a distributed terabyte text retrieval system in China-US million book digital library.

Research on Distributed Index Mechanism for Large Dataset

A Distributed Text Mining System for Online Web Textual Data Analysis

Research and Implementation of a Full-text Retrieval System

TIIS: A Two-Level Inverted-Index Scheme for Large-Scale Data Processing in the Parallel Database System

A Cache-Based Distributed Terabyte Text Retrieval System in CADAL

A PEER-TO-PEER TEXT SHARING AND RETRIEVAL SYSTEM

Research and implement of real-time data loading system IMIL

Research and Implementation of Multi-Terabyte Data Real-Time Loading Technology