Abstract:To accommodate the explosively increasing amount of data in many areas such as scientific computing and e-Business, physical storage devices and control components have been separated from traditional computing systems to become a scalable, intelligent storage subsystem that, when appropriately designed, should provide transparent storage interface, effective data allocation, flexible and efficient storage management, and other impressive features. The design goals and desirable features of such a storage subsystem include high performance, high scalability, high availability, high reliability and high security. Extensive research has been conducted in this field by researchers all over the world, yet many issues still remain open and challenging. This paper studies five different online massive storage systems and one offline storage system that we have developed with the research grant support from China. The storage pool with multiple network-attached RAIDs avoids expensive store-and-forward data copying between the server and storage system, improving data transfer rate by a factor of 2–3 over a traditional disk array. Two types of high performance distributed storage systems for local-area network storage are introduced in the paper. One of them is the Virtual Interface Storage Architecture (VISA) where VI as a communication protocol replaces the TCP/IP protocol in the system. VISA’s performance is shown to achieve better than that of IP SAN by designing and implementing the vSCSI (VI-attached SCSI) protocol to support SCSI commands in the VI network. The other is a fault-tolerant parallel virtual file system that is designed and implemented to provide high I/O performance and high reliability. A global distributed storage system for wide-area network storage is discussed in detail in the paper, where a Storage Service Provider is added to provide storage service and plays the role of user agent for the storage system. Object based Storage Systems not only store data but also adopt the attributes and methods of objects that encapsulate the data. The adaptive policy triggering mechanism (APTM), which borrows proven machine learning techniques to improve the scalability of object storage systems, is the embodiment of the idea about smart storage device and facilitates the self-management of massive storage systems. A typical offline massive storage system is used to backup data or store documents, for which the tape virtualization technology is discussed. Finally, a domain-based storage management framework for different types of storage systems is presented in the paper.

InfoMall: A Large-Scale Storage System for Web Archiving

A New Data Storage and Service Model of China Web InfoMall 1

Engineering Of Web Infomall: The Chinese Web Archive

A digital library architecture supporting massive small files and efficient replica maintenance.

From Web Archive to WebDigest: Concept and Examples

An Enterprise-Wide Digital Archive Management System Infrastructure

Wih - The Web Information Collecting System

Building a Story Tracer out of a Web Archive

A shared fragments analysis system for large collections of web pages

Distributed Archiving Storage System Based on CAS

Large-Scale Web Service Collection for Service Archive Management

Archiving the Relaxed Consistency Web

Massive Storage Systems

Architectural Design and Evaluation of an Efficient Web-Crawling System

Searching for Historical Events on a Large-Scale Web Archive.

Reconstructing Websites for the Lazy Webmaster

The Many Shapes of Archive-It

Future Web Growth and its Consequences for Web Search Architectures

Creating Structure in Web Archives With Collections: Different Concepts From Web Archivists

ColdStore: A Storage System for Archival Data

Chinese Web Infrastructure Building: Challenges and Our Roadmap