Abstract:Accurate classification of transients obtained from spectroscopic data are important to understand their nature and discover new classes of astronomical objects. For supernovae (SNe), SNID, NGSF (a Python version of SuperFit), and DASH are widely used in the community. Each tool provides its own metric to help determine classification, such as rlap of SNID, chi2/dof of NGSF, and Probability of DASH. However, we do not know how accurate these tools are, and they have not been tested with a large homogeneous dataset. Thus, in this work, we study the accuracy of these spectral classification tools using 4,646 SEDMachine spectra, which have accurate classifications obtained from the Zwicky Transient Facility Bright Transient Survey (BTS). Comparing our classifications with those from BTS, we have tested the classification accuracy in various ways. We find that NGSF has the best performance (overall Accuracy 87.6% when samples are split into SNe Ia and Non-Ia types), while SNID and DASH have similar performance with overall Accuracy of 79.3% and 76.2%, respectively. Specifically for SNe Ia, SNID can accurately classify them when rlap > 15 without contamination from other types, such as Ibc, II, SLSN, and other objects that are not SNe (Purity > 98%). For other types, determining their classification is often uncertain. We conclude that it is difficult to obtain an accurate classification from these tools alone. This results in additional human visual inspection effort being required in order to confirm the classification. To reduce this human visual inspection and to support the classification process for future large-scale surveys, this work provides supporting information, such as the accuracy of each tool as a function of its metric.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the accuracy of existing spectral classification tools for transient sources. Specifically, the author used 4,646 spectral data from SEDMachine (Spectral Energy Distribution Machine), which have accurate classifications obtained from the Zwicky Transient Survey (ZTF Bright Transient Survey, BTS), to test the classification accuracy of three commonly - used spectral classification tools: SNID, NGSF (Next Generation SuperFit) and DASH. ### Research Background In modern time - domain astronomy, with the sharp increase in the amount of observational data, it has become crucial to quickly and accurately classify newly discovered transient sources. Currently, the main methods for transient source classification include template - matching - based techniques (such as SNID, SuperFit, GELATO, etc.) and deep - learning - based methods (such as DASH, SNIascore, etc.). However, the performance of these tools has not been rigorously tested on large - scale homogeneous data sets, so their classification accuracy remains unclear. ### Main Problems 1. **Evaluate the accuracy of existing spectral classification tools**: - By using a large amount of spectral data with known classifications (4,646 SEDMachine spectra), the researchers hope to evaluate the classification accuracy of the three tools SNID, NGSF and DASH in different categories (such as type Ia supernovae and other types). 2. **Provide supporting information to improve future large - scale surveys**: - By comprehensively evaluating the performance of these tools, the researchers hope to be able to provide useful supporting information for future spectral classification work, such as the accuracy of each tool under different metrics, thereby reducing the need for manual visual inspection. ### Method Overview - **Sample Selection**: From August 2018 to June 2022, SEDMachine obtained a total of 10,822 spectra. After a series of screening steps (such as removing cosmic - ray contamination, signal - to - noise ratio greater than 3, etc.), 4,646 spectra with accurate BTS classifications were finally obtained as test samples. - **Classification Tools and Parameter Settings**: Run SNID, NGSF and DASH respectively, set the same wavelength range (4000 Å to 8000 Å), and classify according to the default templates of each tool. For NGSF, the extinction law parameter was adjusted; for DASH, the default template was used. - **Performance Evaluation**: By comparing with the accurate classification results provided by BTS, calculate evaluation metrics such as the overall accuracy, purity, true positive rate (TPR) and false positive rate (FPR) for each tool. ### Conclusions The research shows that when the samples are divided into two categories (type Ia supernovae and non - type Ia), NGSF shows the best overall accuracy (87.6%), while the accuracies of SNID and DASH are 79.3% and 76.2% respectively. For type Ia supernovae, when the rlap value of SNID is greater than 15, a purity of more than 98% can be achieved. However, for other types of transient sources, there is still a large degree of uncertainty in classification, and additional manual visual inspection is required to confirm the classification results. In summary, this study provides an important reference for the performance of spectral classification tools for transient sources and points out the direction for future improvement.

How accurate are transient spectral classification tools? -- A study using 4,646 SEDMachine spectra

A transient search using combined human and machine classifications

The SED Machine: a robotic spectrograph for fast transient classification

CCSNscore: A multi-input deep learning tool for classification of core-collapse supernovae using SED-Machine spectra

Photometric Classification of Early-Time Supernova Lightcurves with SCONE

Photometric classification of HSC transients using machine learning

Photometrically-Classified Superluminous Supernovae from the Pan-STARRS1 Medium Deep Survey: A Case Study for Science with Machine Learning-Based Classification

The Zwicky Transient Facility Bright Transient Survey. II. A Public Statistical Sample for Exploring Supernova Demographics

Nonparametric Transient Classification using Adaptive Wavelets

First Impressions: Early-Time Classification of Supernovae using Host Galaxy Information and Shallow Learning

Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era

Picture Perfect: Photometric Transient Classification Using the ParSNIP Model with Roman Hourglass Simulations

Superphot+: Real-time Fitting and Classification of Supernova Light Curves

Superphot+: Realtime Fitting and Classification of Supernova Light Curves

Towards Precision Photometric Type Ia Supernova Cosmology with Machine Learning

Testing the accuracy of SED modeling techniques using the NIHAO-SKIRT-Catalog

Towards Rate Estimation for Transient Surveys I: Assessing Transient Detectability and Volume Sensitivity for iPTF

Multifilter UV to Near-infrared Data-driven Light-curve Templates for Stripped-envelope Supernovae

Probabilistic Classification of Infrared-selected targets for SPHEREx mission: In search of YSOs

Testing the Accuracy of Spectral Energy Distribution Modeling Techniques Using the NIHAO-SKIRT-Catalog

Avocado: Photometric Classification of Astronomical Transients with Gaussian Process Augmentation