Design and Development of the Mass Data Storage Platform Based on Hadoop

Cui Jie,Li Taoshen,Lan Hongxing
2012-01-01
Journal of Computer Research and Development
Abstract:With the development and utilization of BeiBu Bay Marine ecological resources, mass marine science data rapidly emerge in large numbers and it is very important to use a mass data storage platform to manage and store these science data reasonable. This paper puts forward the management and storage the mass marine science data methods based on the distributed computing technology, builds the mass marine science data storage platform solutions, designs and develops a mass data storage platform based on Hadoop by using Linux cluster technology. This system which consists of five modules includes system management module, parallel loading storage module, parallel query module, data dictionary module, backup and recovery module and it can achieve to store massive amounts of marine science data. The system module achieving result shows that this system enjoys good safety, reliability, easy maintenance and good expansibility.
What problem does this paper attempt to address?