Abstract:Cross-matching operation, which is to find corresponding data for the same celestial object or region from multiple catalogues,is indispensable to astronomical data analysis and research. Due to the large amount of astronomical catalogues generated by the ongoing and next-generation large-scale sky surveys, the time complexity of the cross-matching is increasing dramatically. Heterogeneous computing environments provide a theoretical possibility to accelerate the cross-matching, but the performance advantages of heterogeneous computing resources have not been fully utilized. To meet the challenge of cross-matching for substantial increasing amount of astronomical observation data, this paper proposes Heterogeneous-computing-enabled Large Catalogue Cross-matcher (HLC2), a high-performance cross-matching framework based on spherical position deviation on CPU-GPU heterogeneous computing platforms. It supports scalable and flexible cross-matching and can be directly applied to the fusion of large astronomical cataloguesfrom survey missions and astronomical data centres. A performance estimation model is proposed to locate the performance bottlenecks and guide the optimizations. A two-level partitioning strategy is designed to generate an optimized data placement according to the positions of celestial objects to increase throughput. To make HLC2 a more adaptive solution, the architecture-aware task splitting, thread parallelization, and concurrent scheduling strategies are designed and integrated. Moreover, a novel quad-direction strategy is proposed for the boundary problem to effectively balance performance and completeness. We have experimentally evaluated HLC2 using public released catalogue data. Experiments demonstrate that HLC2 scales well on different sizes of catalogues and the cross-matching speed is significantly improved compared to the state-of-the-art cross-matchers.

MapReduce for astronomical cross-matching

Join Query Optimization Based on MapReduce under Skewed Data

Astronomical Data Fusion Tool Based on PostgreSQL

The performance of MapReduce: an in-depth study

The Performance of MapReduce

HLC2: a highly efficient cross-matching framework for large astronomical catalogues on heterogeneous computing environments

Parallelization of BLAST with MapReduce for Long Sequence Alignment.

Distributed data management using MapReduce

Research on the Analysis and Statistic of Geographical Conditions Based on the Strategy of "grid Index + MapReduce"

Query optimization for massively parallel data processing.

Progressive Image Retrieval with Quality Guarantee under MapReduce Framework

A Command-line Cross-matching tool for modern astrophysical pipelines

Xml Structural Similarity Search Using Mapreduce

Adapting Skyline Computation To The Mapreduce Framework: Algorithms And Experiments

Massive Image Data Management Using Hbase And Mapreduce

A Survey on Map-Matching Algorithms

M2M: A Simple Matlab-to-MapReduce Translator for Cloud Computing

Skyline Computation under MapReduce Framework

TT-HEALpix: A New Data Indexing Strategy for Efficient Cross-match of Large-scale Astronomical Catalogs

Fast coordinate cross-match tool for large astronomical catalogue

Cloud Based Short Read Mapping Service