Efficient Data Management for IPFS dApps

Vero Estrada-Galiñanes,Ahmad ElRouby,Léo Marc-André Theytaz

2024-04-25

Abstract:Inefficient data management has been the Achilles heel of blockchain-based decentralized applications (dApps). An off-chain storage layer, which lies between the application and the blockchain layers, can improve space efficiency and data availability with erasure codes and decentralized maintenance. This paper presents two fundamental components of such storage layer designed and implemented for the IPFS network. The IPFS Community is a component built on top of the IPFS network that encodes and decodes data before uploading to the network. Since data is encoded with alpha entanglement codes, the solution requires less storage space than the native IPFS solution which replicates data by pinning content with the IPFS Cluster. To detect and repair failures in a timely manner, we introduce the monitoring and repair component. This novel component is activated by any node and distributes the load of repairs among various nodes. These two components are implemented as pluggable modules, and can, therefore, be easily migrated to other distributed file systems by adjusting the connector component.

Networking and Internet Architecture,Distributed, Parallel, and Cluster Computing

What problem does this paper attempt to address?

The paper primarily proposes a solution to the data management issues in decentralized applications (dApps) based on blockchain. Specifically, the paper focuses on how to improve the data management efficiency of applications based on IPFS (InterPlanetary File System).

The main issues addressed in the paper include:
1. **Low data management efficiency**: Current blockchain-based decentralized applications have deficiencies in data management, especially in the design of the storage layer, leading to low space efficiency and poor data availability.
2. **High storage costs**: The cost of storing data on the blockchain is high and unstable. For example, storing data on Ethereum requires paying high gas fees.
3. **High resource consumption**: Typical blockchain solutions require a large amount of network resources, such as computing power and memory, which further exacerbates resource management issues and may increase the risk of centralization.
4. **Unreliable files**: Due to the lack of effective fault detection and repair mechanisms, the reliability of files in decentralized storage networks is relatively low.

To address the above issues, the paper proposes the following two core components:
1. **IPFS Community**: This is a component built on the IPFS network that uses Alpha Entanglement Codes to encode and decode data. Compared to IPFS's native replication mechanism, this method can reduce the required storage space.
2. **Monitoring and Repair Component**: This is an innovative component that can be activated by any node and distributes the repair workload across multiple nodes, ensuring timely detection and repair of faults.

These two components are implemented as pluggable modules, making them easy to migrate to other distributed file systems with only adjustments to the connector components needed.

Through these methods, the paper aims to improve the data management efficiency of IPFS-based applications, enhance data reliability and availability, and reduce storage costs and resource consumption.