Abstract:Modern high-throughput structural biology laboratories produce vast amounts of raw experimental data. The traditional method of data reduction is very simple-results are summarized in peer-reviewed publications, which are hopefully published in high-impact journals. By their nature, publications include only the most important results derived from experiments that may have been performed over the course of many years. The main content of the published paper is a concise compilation of these data, an interpretation of the experimental results, and a comparison of these results with those obtained by other scientists.Due to an avalanche of structural biology manuscripts submitted to scientific journals, in many recent cases descriptions of experimental methodology (and sometimes even experimental results) are pushed to supplementary materials that are only published online and sometimes may not be reviewed as thoroughly as the main body of a manuscript. Trouble may arise when experimental results are contradicting the results obtained by other scientists, which requires (in the best case) the reexamination of the original raw data or independent repetition of the experiment according to the published description of the experiment. There are reports that a significant fraction of experiments obtained in academic laboratories cannot be repeated in an industrial environment (Begley CG & Ellis LM, Nature 483(7391):531-3, 2012). This is not an indication of scientific fraud but rather reflects the inadequate description of experiments performed on different equipment and on biological samples that were produced with disparate methods. For that reason the goal of a modern data management system is not only the simple replacement of the laboratory notebook by an electronic one but also the creation of a sophisticated, internally consistent, scalable data management system that will combine data obtained by a variety of experiments performed by various individuals on diverse equipment. All data should be stored in a core database that can be used by custom applications to prepare internal reports, statistics, and perform other functions that are specific to the research that is pursued in a particular laboratory.This chapter presents a general overview of the methods of data management and analysis used by structural genomics (SG) programs. In addition to a review of the existing literature on the subject, also presented is experience in the development of two SG data management systems, UniTrack and LabDB. The description is targeted to a general audience, as some technical details have been (or will be) published elsewhere. The focus is on "data management," meaning the process of gathering, organizing, and storing data, but also briefly discussed is "data mining," the process of analysis ideally leading to an understanding of the data. In other words, data mining is the conversion of data into information. Clearly, effective data management is a precondition for any useful data mining. If done properly, gathering details on millions of experiments on thousands of proteins and making them publicly available for analysis-even after the projects themselves have ended-may turn out to be one of the most important benefits of SG programs.

Protein simulation data in the relational model

DCMS: A data analytics and management system for molecular simulation

Protein Construction-Based Data Partitioning Scheme for Alignment of Protein Macromolecular Structures Through Distributed Querying in Federated Databases

High dimensional biological data retrieval optimization with NoSQL technology

Modeling and In-Database Management of Relational, Data-Aware Processes (Extended Version)

Array Requirements for Scientific Applications and an Implementation for Microsoft SQL Server

MDRepo - an open environment for data warehousing and knowledge discovery from molecular dynamics simulations

Mesh Data Management in Large-Scale Scientific Computing

An optimized relational database for querying structural patterns in proteins

Bio-Strings: A Relational Database Data-Type for Dealing with Large Biosequences

Parallel data analysis directly on scientific file formats

Scientific data organization and management in numerical simulations

Extensible Database Simulator for Fast Prototyping In-Database Algorithms

Running and analyzing massively parallel molecular simulations

D4M: Bringing associative arrays to database engines

Data storage architectures to accelerate chemical discovery: data accessibility for individual laboratories and the community

Scalable Linear Algebra on a Relational Database System

A protein relational database and protein family knowledge bases to facilitate structure-based design analyses

Solving the problem of storing trajectories of molecular dynamics in a DBMS

SQL for SRL: Structure Learning Inside a Database System

Data management in the modern structural biology and biomedical research environment