Experimental performance study of a user intensive and large-scale digital library framework

Xiangwen Liao,Binxing Fang,Weihua Luo,Bin Wang

DOI: https://doi.org/10.1109/DIAL.2006.18

2006-01-01

Abstract:In digital libraries a challenge at the system level is how to design retrieval engines that can effectively process an increasing massive number of documents, while handling a considerable number of queries simultaneously. However, traditional database retrieval system has problems of the stability and performance and commercial search engines have also a long list of problems such as the high quality retrieval and long-term availability (Lossau, 2004). This paper presents a high-performance digital library architecture, which processes large-scale data and tries to use the character of digital libraries and exploit resource efficiently to provide sub-second service for user queries. We focus on system architecture, optimization of index structure and each component, and the integration of each component with platform. The experimental results show that (1) our system achieves a throughput of 283.3 replies per second with 2.0G metadata index on one node of Dawning 4000H[20] platform; (2)The throughout of our system decreases sub-linearly with the increase of index size and increases sub-linearly with the increase of search server's number