The Development and Deployment of Large-File Upload Services
Huan Chen,Liangjie Zhang,Bo Hu,Lihui Luo,Chunxiao Xing
DOI: https://doi.org/10.29268/stbd.2015.2.3.4
2015-01-01
Services Transactions on Big Data
Abstract:The popularity of enterprise cloud storage is rapidly growing. A number of Internet service vendors and providers, such as Google, Baidu and Microsoft, entered this emerging market and released a variety of cloud storage services. These services allow people to access work documents and files all over the world at anytime. Interestingly, with the prevalence of mobile Internet, rich media becomes regular and popular. More and more people use cloud storage for keeping their personal photos, music and movies. Nevertheless, the size of the media files is often beyond the upper limit that normal form-based file upload service allows hence dedicated large-file upload services are required to be developed and deployed. Although many cloud vendors offer versatile cloud storage services, very little is known about the detailed development and deployment of the large-file upload services. This paper proposes a complete solution of large-file upload service, with the contributions in manifold: Firstly, we do not limit the maximum size of a large file that can be uploaded. This is extremely practical for storing huge database resource files generated from ERP tools. Secondly, we developed large-file upload service APIs that have very strict verification of correctness, to reduce the risk of data inconsistency, which has better safety. Thirdly, we extend the service developed recently for team collaboration with the capability of handling large files. Fourthly, this paper is arguably the first one that formalizes the testing and deployment procedures of large-file upload services with the help of Docker. In general, most largefile upload services are exposed to the public, facing security and performance issues, which brings much concern. With the proposed Docker-based deployment strategy, we can replicate the large-file upload service agilely and locally, to satisfy massive private or local deployment of KDrive. Finally, we evaluate and analyze the proposed strategies and technologies in accordance to the experimental results. This paper is an extension version of the SCC 2015 conference paper: On Developing and Deploying Large-File Upload Services of Personal Cloud Storage.