The Petabyte Project

Evan F. Lewis,Sarah Burke-Spolaor,Maura McLaughlin,Duncan Lorimer,Kshitij Aggarwal,Devansh Agarwal,Joseph Kania,Nate Garver-Daniels,Joseph P. Glaser

2023-08-24

Abstract:Transient radio sources, such as fast radio bursts, intermittent pulsars, and rotating radio transients, can offer a wealth of information regarding extreme emission physics as well as the intervening interstellar and/or intergalactic medium. Vital steps towards understanding these objects include characterizing their source populations and estimating their event rates across observing frequencies. However, previous efforts have been undertaken mostly by individual survey teams at disparate observing frequencies and telescopes, and with non-uniform algorithms for searching and characterization. The Petabyte Project (TPP) aims to address these issues by uniformly reprocessing data from several petabytes of radio transient surveys covering two decades of observing frequency (300 MHz-20 GHz). The TPP will provide robust event rate analyses, in-depth assessment of survey and pipeline completeness, as well as revealing discoveries from archival and ongoing radio surveys. We present an overview of TPP's processing pipeline, scope, and our potential to make new discoveries.

Instrumentation and Methods for Astrophysics,High Energy Astrophysical Phenomena

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to improve the understanding of radio transient sources (such as fast radio bursts, intermittent pulsars, and rotating radio transients) by uniformly processing and re - analyzing a large amount of observational data of these sources. Specifically, this research aims to solve the following key issues:

1. **Inconsistent data processing methods**: Most of the past radio transient source searches were carried out by different observational teams on different frequencies and telescopes, using non - uniform algorithms for searching and characterization. This has led to the results of different observations being difficult to compare directly.

2. **Accuracy of event rate estimation**: Due to the lack of tracking of observational completeness in past studies and the possibility that some types of fast radio bursts (FRB) may not have been detected, the estimation of FRB event rates may be biased. An accurate event rate is crucial for predicting the FRB detection rate of new instruments or new observations.

3. **Omission of high - DM and low - DM sources**: Many traditional searches have failed to fully de - disperse data to detect fast radio bursts with high dispersion measures (DM), or misidentified low - DM candidates as man - made radio - frequency interference (RFI). This may lead to some important FRBs being missed.

4. **Limitations of manual classification**: Previous searches usually involved manual inspection of thousands of candidates. This method is not only time - consuming but also error - prone.

To solve these problems, the paper introduces "The Petabyte Project" (TPP), which aims to provide more accurate event rate analysis and new discoveries by uniformly re - processing petabytes of radio transient source data from multiple telescopes and observational frequencies. The key objectives of TPP include:

- **Uniform data processing**: A unified reader (your) has been developed that can read data in multiple formats and convert it into a standard format, ensuring that all data can be processed efficiently.

- **Automated classification**: Use the deep - learning tool FETCH to automatically classify candidates, reducing human intervention and improving the accuracy and efficiency of classification.

- **Complete event rate evaluation**: Use the BARB tool to calculate the FRB event rate, taking into account the influence of different observational conditions, thereby providing a more reliable event rate estimate.

Through these improvements, TPP is expected to reveal more new FRBs and other transient sources hidden in the archival data and provide an important reference for future FRB research.