Abstract:Motivated by increasing powerful edge devices with data-intensive computing and limited storage size, we study a MapReduce-based wireless distributed computing framework by allocating a portion of files in the remote data center to the network edge and utilizing computation and memory resources at the edge. Our framework is composed of three step phases: 1) Map; 2) Shuffle; and 3) Reduce. However, in the data shuffling stage, shuffling many data accounts for a large amount of the total running time over wireless interference networks will degrade its performance. Moreover, data shuffling between pervasive edge devices with limited spectrum bandwidth is very challenging. Today, many devices focus on computing functions rather than collecting all the individual wireless data centers. Therefore, we can use over-the-air computation (AirComp) technology to reliably compute multiple target functions by harnessing interference in the multiple-access channel with a higher computation efficiency than the traditional orthogonal multiaccess scheme that combats interference. We study a mixed-timescale optimization of the transmitting–receiving (Tx-Rx) policy and file allocation to minimize the averaged computation mean-squared error (MSE) under the power constraint of each device. File allocation control is adaptive to the long-term statistical channel state information (CSI), while the Tx-Rx policy is adaptive to the CSI and file allocation strategy. We decompose the problem into a short-term Tx-Rx policy and a long-term file allocation control problem to tackle the joint nonconvex optimization. Simulation results indicate the effectiveness of our proposed two-timescale algorithm and the advantages of our computation framework over the state-of-the-art baselines.

Cormorant: Running Analytic Queries on MapReduce with Collaborative Software-Defined Networking

Distributed Shortest Path Query Processing on Dynamic Road Networks

Sdn Based Vxlan Optimization In Cloud Computing Networks

Network Map Reduce

Hybrid Coded MapReduce for Latency-Constrained Tasks with Straggling Servers

OEHadoop: Accelerate Hadoop Applications by Co-Designing Hadoop with Data Center Network

Cost-effective and performant virtual WANs with CORNIFER

Demonstration of Application-Driven Network Slicing and Orchestration in Optical/packet Domains: On-Demand Vdc Expansion for Hadoop MapReduce Optimization.

BigDataSDNSim: A Simulator for Analyzing Big Data Applications in Software-Defined Cloud Data Centers

CABSR: Congestion Agent Based Source Routing for Network-on-Chip

Cora: Accelerating Stateful Network Applications with SmartNICs

A Distributed Computation of the Shortest Path in Large-Scale Road Network

Rapier: Integrating routing and scheduling for coflow-aware data center networks

Over-the-Air Computation of Large Scale Nomographic Functions in MapReduce over the Edge Cloud Network

SDPMN: Privacy Preserving MapReduce Network Using SDN

CORA: Conflict Razor for Policies in SDN.

Availability and Network-Aware MapReduce Task Scheduling over the Internet

Bandwidth-Aware Scheduling with SDN in Hadoop: A New Trend for Big Data

A Deadline-Aware Coflow Scheduling Approach for Big Data Applications.

RDMA-driven MongoDB: an Approach of RDMA Enhanced NoSQL Paradigm for Large-Scale Data Processing

Content-Centric and Software-Defined Networking with Big Data