Abstract:To accommodate the explosively increasing amount of data in many areas such as scientific computing and e-Business, physical storage devices and control components have been separated from traditional computing systems to become a scalable, intelligent storage subsystem that, when appropriately designed, should provide transparent storage interface, effective data allocation, flexible and efficient storage management, and other impressive features. The design goals and desirable features of such a storage subsystem include high performance, high scalability, high availability, high reliability and high security. Extensive research has been conducted in this field by researchers all over the world, yet many issues still remain open and challenging. This paper studies five different online massive storage systems and one offline storage system that we have developed with the research grant support from China. The storage pool with multiple network-attached RAIDs avoids expensive store-and-forward data copying between the server and storage system, improving data transfer rate by a factor of 2–3 over a traditional disk array. Two types of high performance distributed storage systems for local-area network storage are introduced in the paper. One of them is the Virtual Interface Storage Architecture (VISA) where VI as a communication protocol replaces the TCP/IP protocol in the system. VISA’s performance is shown to achieve better than that of IP SAN by designing and implementing the vSCSI (VI-attached SCSI) protocol to support SCSI commands in the VI network. The other is a fault-tolerant parallel virtual file system that is designed and implemented to provide high I/O performance and high reliability. A global distributed storage system for wide-area network storage is discussed in detail in the paper, where a Storage Service Provider is added to provide storage service and plays the role of user agent for the storage system. Object based Storage Systems not only store data but also adopt the attributes and methods of objects that encapsulate the data. The adaptive policy triggering mechanism (APTM), which borrows proven machine learning techniques to improve the scalability of object storage systems, is the embodiment of the idea about smart storage device and facilitates the self-management of massive storage systems. A typical offline massive storage system is used to backup data or store documents, for which the tape virtualization technology is discussed. Finally, a domain-based storage management framework for different types of storage systems is presented in the paper.

Research of Massive Internet Text Data Real-Time Loading and Index System

Research and Implementation of Multi-Terabyte Data Real-Time Loading Technology

Research and implement of real-time data loading system IMIL

Research of Distribute Massive Text Data Index and Retrieval System

Implementation of large-scale distributed information retrieval system

Research on Mass Process-Planning Data Storing and Parallel Accessing in Large-Scale Manufacturing Enterprise.

Research on Mass Process-Planning Data Storing and Parallel Accessing in Large-Scale Manufacturing Enterprise

Research and Implementation of the Muti-Task-Parallel Scheduling Loading Technology of Massive Text Data

The Design and Implementation of an Intelligent Distributed Text Retrieval System

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

An Information Retrieval Algorithm for Massive and Real-time Data

A Two-Tier Distributed Full-Text Indexing System

Load shedding in data stream system

TIIS: A Two-Level Inverted-Index Scheme for Large-Scale Data Processing in the Parallel Database System

An Efficient Strategy of Building Distributed Index Based on Lucene.

A Distributed Text Mining System for Online Web Textual Data Analysis

Research and Application of Massive Data Processing Technology

Research and Implementation on Compression, Encryption, Storage and Retrieval System for Massive Data

Massive Storage Systems

Toward a distributed terabyte text retrieval system in China-US million book digital library.

The Study and Implementation of an Effective Massive Data Storage Technique to OLAP