Abstract:Content-Based Multimedia Retrieval (CBMR) has become very popular in several applications, driven by the growing routine use of multimedia data. Since the datasets used in real-world applications are very large and descriptor's dimensionality is high, querying is an expensive, albeit important functionality. Further, exact search is prohibitive in most cases, motivating the use of Approximate Nearest Neighbour Search (ANNS) algorithms, trading accuracy for performance. These have been mainly developed targeting a sequential execution in a single node. However, the large and increasing datasets used and the high query loads submitted to those systems typically surpass the memory and computing resources available in a single node. This motivated the development of parallel distributed memory ANNS solutions to meet the computing capabilities required by those applications. A common problem that must be handled when using distributed memory systems is data partitioning and its impact on load imbalance. Several data partitioning approaches have already been proposed, including elaborated spatial-aware strategies. However, little effort has been put into carefully analyzing the performance of those strategies at scale. Here, we evaluated the commonly used data partitioning strategies in ANNS and identified their limitations to propose a novel class of partitioning algorithms that can minimize load imbalance while improving data locality to attain high performance on the distributed memory search. Experimentally, we found that our proposed algorithms (SABBS and SABBSR) improved search performance by up to 1.64× compared to the best previous solution. In a distributed memory weak scaling evaluation, with up to 12 billion 128-dimensional descriptors and 60 compute nodes, the gains were maintained as the system scaled with our novel approaches. These results demonstrate the efficiency of our new algorithms for billion-scale ANNS and the importance of considering not only data locality but also data and load imbalance in the data partitioning.

Storage Balancing in Self-organizing Multimedia Delivery Systems

Adaptive data dissemination in wireless sensor networks

Spatial-Aware and Load Balancing Distributed Data Partitioning Strategies for Content-Based Multimedia Retrieval

Peer-to-peer Based Multimedia Distribution Service

Teleport: Load Balancing with Dynamic Adjustment Strategy Based on OSD Storage System

A Novel Content Updating Policy Based on Dynamic Storage Balancing for Clustered Streaming Media System

An Architectural Design for Brokered Collaborative Content Delivery System

Babel Storage: Uncoordinated Content Delivery from Multiple Coded Storage Systems

An Event-Driven Dynamic Load Balancing Strategy For Streaming Media Clustered Server Systems

Dynamic File Grouping for Load Balancing in Streaming Media Clustered Server Systems

Accelerate Data Sharing In A Wide-Area Networked File Storage System

A scheme for improving the performance of multimedia storage system

Service-Oriented Multimedia Delivery in Pervasive Space

A genetic load balancing algorithm to improve the QoS metrics for software defined networking for multimedia applications

A Cluster Based Replication Architecture for Load Balancing in Peer-to-Peer Content Distribution

BufferBank Storage: an Economic, Scalable and Universally Usable In-Network Storage Model for Streaming Data Applications

Approach to Allocate Storage Space Dynamically in Network Storage System

Joint Storage Assignment for D2d Offloading Systems

DeepFloat: Resource-Efficient Dynamic Management of Vehicular Floating Content

Content replication and placement in mobile networks

A Blockchain-Based Storage System With Financial Incentives for Load-balancing