Applied Erasure Coding in Networks and Distributed Storage

Katina Kralevska
DOI: https://doi.org/10.48550/arXiv.1803.01358
2018-03-04
Abstract:The amount of digital data is rapidly growing. There is an increasing use of a wide range of computer systems, from mobile devices to large-scale data centers, and important for reliable operation of all computer systems is mitigating the occurrence and the impact of errors in digital data. The demand for new ultra-fast and highly reliable coding techniques for data at rest and for data in transit is a major research challenge. Reliability is one of the most important design requirements. The simplest way of providing a degree of reliability is by using data replication techniques. However, replication is highly inefficient in terms of capacity utilization. Erasure coding has therefore become a viable alternative to replication since it provides the same level of reliability as replication with significantly less storage overhead. The present thesis investigates efficient constructions of erasure codes for different applications. Methods from both coding and information theory have been applied to network coding, Optical Packet Switching (OPS) networks and distributed storage systems. The following four issues are addressed: - Construction of binary and non-binary erasure codes; - Reduction of the header overhead due to the encoding coefficients in network coding; - Construction and implementation of new erasure codes for large-scale distributed storage systems that provide savings in the storage and network resources compared to state-of-the-art codes; and - Provision of a unified view on Quality of Service (QoS) in OPS networks when erasure codes are used, with the focus on Packet Loss Rate (PLR), survivability and secrecy.
Information Theory,Distributed, Parallel, and Cluster Computing
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is how to efficiently apply erasure - coding techniques in different networks and distributed storage systems with the rapid growth of digital data volume, so as to improve data reliability and transmission efficiency. Specifically, the paper focuses on the following aspects: 1. **Constructing binary and non - binary erasure codes**: Research on how to design efficient binary and non - binary erasure codes to adapt to different application scenarios. 2. **Reducing header overhead in network coding**: Explore how to reduce the header overhead caused by coding coefficients, thereby improving the efficiency of network coding. 3. **Constructing new erasure codes suitable for large - scale distributed storage systems**: Propose new erasure code construction methods, aiming to reduce the consumption of storage and network resources and provide higher efficiency compared with existing technologies. 4. **Applying erasure codes in optical packet - switched networks to improve quality of service**: Research on how to use erasure codes in optical packet - switched networks, with a focus on quality of service (QoS) aspects such as packet loss rate (PLR), survivability and security. Through these studies, the paper aims to provide more reliable and efficient solutions for network communication and distributed storage systems. In particular, the paper proposes two new erasure code construction methods: HashTag erasure codes (HTECs) and balanced locally repairable codes (BLRCs), which are optimized for different application scenarios respectively.