Abstract:Efficiently analyzing geo-distributed datasets is emerging as a major demand in a cloud-edge system. Since the datasets are often generated in closer proximity to end users, traditional works mainly focus on offloading proper tasks from those hotspot edges to the datacenter to decrease the overall completion time of submitted jobs in a one-shot manner. However, optimizing the completion time of <italic>current job</italic> alone is insufficient in a long-term scope since some datasets would be used multiple times. Instead, optimizing the data distribution is much more efficient and could directly benefit forthcoming jobs, although it may postpone the execution of current one. Unfortunately, due to the throwaway feature of data fetcher, existing data analytics systems fail to re-distribute corresponding data out of hotspot edges after the execution of data analytics. In order to minimize the overall completion time for a <italic>sequence</italic> of jobs as well as to guarantee the performance of current one, we propose to re-distribute the data along with task offloading, and formulate corresponding <inline-formula><tex-math notation="LaTeX">$\varepsilon$</tex-math> <alternatives><mml:math><mml:mi>ɛ</mml:mi></mml:math><inline-graphic xlink:href="qian-ieq2-3086274.gif"/></alternatives></inline-formula>-bounded data-driven task scheduling problem over wide area network under the consideration of edge heterogeneity. We design an online schema <inline-formula><tex-math notation="LaTeX">$run$</tex-math> <alternatives><mml:math><mml:mrow><mml:mi>r</mml:mi><mml:mi>u</mml:mi><mml:mi>n</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="qian-ieq3-3086274.gif"/></alternatives></inline-formula>Data, which offloads proper tasks and related data via piggybacking to the datacenter based on delicately calculated probabilities. Through rigorous theoretical analysis, <inline-formula><tex-math notation="LaTeX">$run$</tex-math> <alternatives><mml:math><mml:mrow><mml:mi>r</mml:mi><mml:mi>u</mml:mi><mml:mi>n</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="qian-ieq4-3086274.gif"/></alternatives></inline-formula>Data is proved concentrated on its optimum with high probability. We implement <inline-formula><tex-math notation="LaTeX">$run$</tex-math> <alternatives><mml:math><mml:mrow><mml:mi>r</mml:mi><mml:mi>u</mml:mi><mml:mi>n</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="qian-ieq5-3086274.gif"/></alternatives></inline-formula>Data based on Spark and HDFS. Both testbed results and trace-driven simulations show that <inline-formula><tex-math notation="LaTeX">$run$</tex-math> <alternatives><mml:math><mml:mrow><mml:mi>r</mml:mi><mml:mi>u</mml:mi><mml:mi>n</mml:mi></mml:mrow></mml:math><inline-graphic xlink:href="qian-ieq6-3086274.gif"/></alternatives></inline-formula>Data re-distributes proper data via piggybacking and achieves up to 37 percent reduction on average response time compared with state-of-the-art schemas.

Speeding Up NGB with Distributed File Streaming Framework

A Public Grid Computing Framework Based on a Hierarchical Combination of Middleware

A Deadline And Budget Constrained Cost-Time Optimization Algorithm For Scheduling Dependent Tasks In Grid Computing

Multi-discipline computations and collaborative visualization on the campus grid

Globus service enhancements for exascale applications and facilities

Block-Based Concurrent and Storage-Aware Data Streaming for Grid Applications with Lots of Small Files

Distributed File Streamer: A Framework for Distributed Application Data Coupling

Scheduling On-demand Data Streaming for Cyberinfrastructure Applications with Constraints of Storage and Bandwidth

NO2: Speeding Up Parallel Processing of Massive Compute-Intensive Tasks

Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments

Building A High Performance Parallel File System Using Grid Datafarm and ROOT I/O

A Hierarchical Grid Algorithm for Accelerating High-Performance Conjugate Gradient Benchmark on Sunway Many-Core Processor

Concurrent and Storage-Aware Data Streaming for Data Processing Workflows in Grid Environments

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

A novel deadline and budget constrained scheduling heuristics for computational grids

An Integrated Resource Management and Scheduling System for Grid Data Streaming Applications

A Novel Acceleration Method for DGTD Algorithm on Sunway TaihuLight

A Time-driven Data Placement Strategy for a Scientific Workflow Combining Edge Computing and Cloud Computing

High-performance Genomic Analysis Framework with In-Memory Computing

Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration