The Online Observation Quality System Implementation for the ASTRI Mini-Array Project

L. Castaldini,N. Parmiggiani,A. Bulgarelli,L. Baroncelli,V. Fioretti,A. Di Piano,I. Abu,M. Capalbi,O. Catalano,V. Conforti,M. Fiori,F. Gianotti,F. Lucarelli,M. C. Maccarone,T. Mineo,S. Lombardi,V. Pastore,F. Russo,P. Sangiorgi,S. Scuderi,G. Tosti,M. Trifoglio,L. Zampieri
2024-04-03
Abstract:The ASTRI Mini-Array project, led by the Italian National Institute for Astrophysics, aims to construct and operate nine Imaging Atmospheric Cherenkov Telescopes for high-energy gamma-ray source study and stellar intensity interferometry. Located at the Teide Astronomical Observatory in Tenerife, the project's software is essential for remote operation, emphasizing the need for prompt feedback on observations. This contribution introduces the Online Observation Quality System (OOQS) as part of the Supervisory Control And Data Acquisition (SCADA) software. OOQS performs real-time data quality checks on data from Cherenkov cameras and Intensity Interferometry instruments. It provides feedback to SCADA and operators, highlighting abnormal conditions and ensuring quick corrective actions for optimal observations. Results are archived for operator visualization and further analysis. The OOQS data quality pipeline prototype utilizes a distributed application with three main components to handle the maximum array data rate of 1.15 Gb/s. The first is a Kafka consumer that manages the data stream from the Array Data Acquisition System through Apache Kafka, handling the data serialization and deserialization involved in the transmission. The data stream is divided into batches of data written in files. The second component monitors new files and conducts analyses using the Slurm workload scheduler, leveraging its parallel processing capabilities and scalability. Finally, the process results are collected by the last component and stored in the Quality Archive.
Instrumentation and Methods for Astrophysics
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: How to ensure the data quality of high - energy gamma - ray source observations and stellar intensity interferometry in the ASTRI Mini - Array project, especially providing timely feedback and quickly correcting abnormal situations during remote operations. ### Specific Problem Description 1. **Real - time Data Quality Monitoring**: - The ASTRI Mini - Array project needs to conduct real - time quality checks on data from Cherenkov cameras and intensity interferometers. These data have an extremely high flow rate (up to 1.15 Gb/s at most), so an efficient Online Observation Quality System (OOQS) is required to process and analyze these data. 2. **Automated Corrective Measures**: - When the observed data does not meet the quality requirements, the system needs to be able to automatically take corrective measures and notify the night - time operator. This requires that the OOQS can not only detect abnormal situations but also quickly feed back to the SCADA system and other subsystems in order to adjust the observation parameters in a timely manner. 3. **Efficient Data Processing and Storage**: - In order to deal with large - scale data streams, the OOQS must have efficient data - processing capabilities. Specifically, it needs to be able to quickly receive, decode, analyze, and store a large amount of observational data. At the same time, all data quality results need to be permanently archived for subsequent analysis and visualization. ### Solution Overview The paper introduces the implementation of the OOQS and its integration in the SCADA software. The OOQS solves the above problems in the following ways: - **Data Stream Management**: Use Apache Kafka and Avro protocol to manage and transmit data streams. - **Distributed Processing**: Adopt the Slurm workload scheduler for parallel processing to improve data - processing efficiency. - **Real - time Analysis**: Conduct real - time data analysis and aggregation through two applications, DQ - Analysis and DQ - Aggregator. - **Efficient Storage**: Store the analysis results in the Quality Archive database to ensure the long - term availability and traceability of data. ### Summary The introduction of the OOQS has significantly improved the observation quality and operation efficiency of the ASTRI Mini - Array project, ensuring the smooth progress of high - energy gamma - ray source research.