A large-scale rendering system based on hadoop

Weifeng Liu,Bin Gong,Yi Hu

DOI: https://doi.org/10.1109/ICPCA.2011.6106549

2011-01-01

Abstract:3D rendering is a kind of application which is not only data intensive but also computation intensive and we can render different frames in parallel. For a 3D rendering system, the I/O performance and the ability to handle the failures of both hardware and software are very important. Traditional method uses OpenPBS to schedule the rendering tasks and puts the data on a NFS data server. But NFS data server can very easily be the bottleneck of the system and OpenPBS can't reschedule the failing tasks, so new method is required to solve those problems. MapReduce is a programming model serving for processing large scale data sets in a parallel manner and hadoop is an open source implementation of the MapReduce programming model. In this paper we propose a method which uses hadoop to do the 3D rendering work in parallel. The proposed method is based on hadoop-0.20.2, evaluation on our method and the traditional method shows that for a scene with 40 frames, our method can reduce the execution time more significantly than traditional method as the number of rendering nodes increases, and our method can also handle failures of both the rendering node and rendering program.