Abstract:Synthetic biology is a complex discipline that involves creating detailed, purpose-built designs from genetic parts. This process is often phrased as a Design-Build-Test-Learn loop, where iterative design improvements can be made, implemented, measured, and analyzed. Automation can potentially improve both the end-to-end duration of the process and the utility of data produced by the process. One of the most important considerations for the development of effective automation and quality data is a rigorous description of implicit knowledge encoded as a formal knowledge representation. The development of knowledge representation for the process poses a number of challenges, including developing effective human-machine interfaces, protecting against and repairing user error, providing flexibility for terminological mismatches, and supporting extensibility to new experimental types. We address these challenges with the DARPA SD2 Round Trip software architecture. The Round Trip is an open architecture that automates many of the key steps in the Test and Learn phases of a Design-Build-Test-Learn loop for high-throughput laboratory science. The primary contribution of the Round Trip is to assist with and otherwise automate metadata creation, curation, standardization, and linkage with experimental data. The Round Trip's focus on metadata supports fast, automated, and replicable analysis of experiments as well as experimental situational awareness and experimental interpretability. We highlight the major software components and data representations that enable the Round Trip to speed up the design and analysis of experiments by 2 orders of magnitude over prior ad hoc methods. These contributions support a number of experimental protocols and experimental types, demonstrating the Round Trip's breadth and extensibility. We describe both an illustrative use case using the Round Trip for an on-the-loop experimental campaign and overall contributions to reducing experimental analysis time and increasing data product volume in the SD2 program.

MLDev: Data Science Experiment Automation and Reproducibility Software

Machine learning experiment management tools: a mixed-methods empirical study

MLXP: A Framework for Conducting Replicable Experiments in Python

A Backend Platform for Supporting the Reproducibility of Computational Experiments

Enabling Reproducibility and Meta-learning Through a Lifelong Database of Experiments (LDE)

SysML'19 demo: customizable and reusable Collective Knowledge pipelines to automate and reproduce machine learning experiments

Deriva-ML: A Continuous FAIRness Approach to Reproducible Machine Learning Models

Case Studies and Challenges in Reproducibility in the Computational Sciences

Promoting open science in test-driven software experiments

Round Trip: An Automated Pipeline for Experimental Design, Execution, and Analysis

dagger: A Python Framework for Reproducible Machine Learning Experiment Orchestration

SystemDS: A Declarative Machine Learning System for the End-to-End Data Science Lifecycle

Experimenting with Multi-Agent Software Development: Towards a Unified Platform

BEAT: An Open-Source Web-Based Open-Science Platform

MLExchange: A web-based platform enabling exchangeable machine learning workflows for scientific studies

AutoDS: Towards Human-Centered Automation of Data Science

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows

OpenML: An R Package to Connect to the Machine Learning Platform OpenML

Autonomous experiments using active learning and AI

Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

Wanted: standards for automatic reproducibility of computational experiments