ScaMMDB: Facing Challenge of Mass Data Processing with MMDB

Yansong Zhang,Yanqin Xiao,Zhanwei Wang,Xiaodong Ji,Yunkui Huang,Shan Wang
DOI: https://doi.org/10.1007/978-3-642-03996-6_1
2009-01-01
Abstract:Main memory database(MMDB) has much higher performance than disk resident database(DRDB), but the architecture of hardware limits the scalability of memory capacity. In OLAP applications, comparing with data volume, main memory capacity is not big enough and it is hard to extend. In this paper, ScaMMDB prototype is proposed towards the scalability of MMDB. A multi-node structure is established to enable system to adjust total main memory capacity dynamically when new nodes enter the system or some nodes leave the system. ScaMMDB is based on open source MonetDB which is a typical column storage model MMDB, column data transmission module, column data distribution module and query execution plan re-writing module are developed directly in MonetDB. Any node in ScaMMDB can response user's requirements and SQL statements are transformed automatically into extended column operating commands including local commands and remote call commands. Operation upon certain column is pushed into the node where column is stored, current node acts as temporarily mediator to call remote commands and assembles the results of each column operations. ScaMMDB is a test bed for scalability of MMDB, it can extend to MMDB cluster, MMDB replication server, even peer-to-peer OLAP server for further applications.
What problem does this paper attempt to address?