The Evolution of Raw Data Archiving and the Growth of Its Importance in Crystallography

John R. Helliwell,James R. Hester,Loes Kroon-Batenburg,Brian McMahon,Selina L. S. Storm
2024-02-23
Abstract:The hardware for data archiving has expanded capacities for digital storage enormously in the past decade or more. This article charts the efforts of IUCr to facilitate discussions and plans relating to raw data archiving and reuse within the various communities of crystallography, diffraction, and scattering.
Other Quantitative Biology
What problem does this paper attempt to address?
This paper focuses on the importance and development of archiving and reusing raw data in crystallography. The International Union of Crystallography (IUCr) has been discussing this issue through workshops and working groups over the past decade to facilitate discussions and implementation strategies for archiving raw data. The paper points out that with the significant improvement in digital storage hardware capabilities, archiving raw data ensures the reproducibility of research results, i.e., "fundamental truth." IUCr has also introduced a "Raw Data Letters" section in the journal IUCrData for publishing partially or uninterpreted diffraction pattern data. The paper highlights several key points: 1. The current status of archiving and sharing raw data, best practices, and IUCr-sponsored workshops to promote community discussions and implementation methods. 2. The role of peer review in validating raw data is discussed, emphasizing the importance of archiving raw data for maximizing future data sharing and reuse potential. 3. The standardization and interoperability of data in different scientific fields, such as the FAIR (Findable, Accessible, Interoperable, Reusable) principles, and the focus on data quality. 4. The progress in experimental hardware, such as improvements in light sources and detectors, their impact on data acquisition rates and volumes, and the resulting data management challenges. The paper also covers the current situation in different branches of crystallography, such as the differing stances of the biomolecular and structural chemistry communities towards archiving diffraction images, and mentions related work in powder diffraction and small-angle scattering communities. Finally, the article addresses the issues of physical storage and long-term archiving strategies, as well as the influence of environmental and economic factors. In conclusion, this paper aims to provide an overview of the development of archiving and reusing raw data in the field of crystallography, highlighting its importance, and discussing the successes and challenges in achieving this goal.