Abstract:Intermediate data structures are a common cause of inefficiency in functional programming. Fusion attempts to eliminate intermediate data structures by combining adjacent data traversals into one; existing fusion techniques, however, are based on predefined rewrite rules and hence are limited in expressiveness. In this work we explore a different approach to eliminating intermediate data structures, based on inductive program synthesis. We dub this approach superfusion (by analogy with superoptimization , which uses inductive synthesis for program optimization). Starting from a reference program annotated with data structures to be eliminated, superfusion first generates a sketch where program fragments operating on those data structures are replaced with holes; it then fills the holes with constant-time expressions such that the resulting program is equivalent to the reference. The main technical challenge here is scalability because optimized programs are often complex, making the search space intractably large for naive enumeration. To address this challenge, our key insight is to first synthesize a ghost function that describes the relationship between the original intermediate data structure and its compressed version; this function, although not used in the final program, serves to decompose the joint sketch filling problem into independent simpler problems for each hole. We implement superfusion in a tool called SuFu and evaluate it on a dataset of 290 tasks collected from prior work on deductive fusion and program restructuring. The results show that SuFu solves 264 out of 290 tasks, exceeding the capabilities of rewriting-based fusion systems and achieving comparable performance with specialized approaches to program restructuring on their respective domains.

Filter-embedding Semiring Fusion for Programming with MapReduce

Generate, test, and aggregate: a calculation-based framework for systematic parallel programming with mapreduce

Calculational Parallel Programming (Parallel Programming With Homomorphism And Mapreduce)

Towards Systematic Parallel Programming Over Mapreduce

Accumulative Computation on MapReduce

Parallelization in Calculational Forms.

A Semantic++ MapReduce Parallel Programming Model.

Towards co-designed optimizations in parallel frameworks: A MapReduce case study

Parallelization of Ontology Construction and Fusion Based on MapReduce

Fusing Gathers with Integer Linear Programming

Evaluating Large Graph Processing in MapReduce Based on Message Passing

Query optimization for massively parallel data processing.

Superfusion: Eliminating Intermediate Data Structures Via Inductive Synthesis

Formal Derivation of Efficient Parallel Programs by Construction of List Homomorphisms.

Program optimizations and transformations in calculation form

Optimization and reconstruction shuffle in MapReduce

A Semantic++ MapReduce: A Preliminary Report

Distance-function design and fusion for sequence data.

Monoidify! Monoids as a Design Principle for Efficient MapReduce Algorithms

ConMR: Concurrent MapReduce Programming Model for Large Scale Shared-Data Applications

Towards Formalizing of MapReduce.