Abstract:The advancements in artificial intelligence in recent years, such as Large Language Models (LLMs), have fueled expectations for breakthroughs in genomic foundation models (GFMs). The code of nature, hidden in diverse genomes since the very beginning of life's evolution, holds immense potential for impacting humans and ecosystems through genome modeling. Recent breakthroughs in GFMs, such as Evo, have attracted significant investment and attention to genomic modeling, as they address long-standing challenges and transform in-silico genomic studies into automated, reliable, and efficient paradigms. In the context of this flourishing era of consecutive technological revolutions in genomics, GFM studies face two major challenges: the lack of GFM benchmarking tools and the absence of open-source software for diverse genomics. These challenges hinder the rapid evolution of GFMs and their wide application in tasks such as understanding and synthesizing genomes, problems that have persisted for decades. To address these challenges, we introduce GFMBench, a framework dedicated to GFM-oriented benchmarking. GFMBench standardizes benchmark suites and automates benchmarking for a wide range of open-source GFMs. It integrates millions of genomic sequences across hundreds of genomic tasks from four large-scale benchmarks, democratizing GFMs for a wide range of in-silico genomic applications. Additionally, GFMBench is released as open-source software, offering user-friendly interfaces and diverse tutorials, applicable for AutoBench and complex tasks like RNA design and structure prediction. To facilitate further advancements in genome modeling, we have launched a public leaderboard showcasing the benchmark performance derived from AutoBench. GFMBench represents a step toward standardizing GFM benchmarking and democratizing GFM applications.

Building a continuous benchmarking ecosystem in bioinformatics

Omnibenchmark (alpha) for continuous and open benchmarking in bioinformatics

Codabench: Flexible, Easy-to-use, and Reproducible Meta-Benchmark Platform

AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite

A Continuous Benchmarking Infrastructure for High-Performance Computing Applications

Aibench: an industry standard ai benchmark suite

Challenges and best practices in omics benchmarking

Reproducible biomedical benchmarking in the cloud: lessons from crowd-sourced data challenges

BinBencher: Fast, flexible and meaningful benchmarking suite for metagenomic binning

Benchmark Data Repositories for Better Benchmarking

Benchmark Frameworks and Τbench

NetBenchmark: a bioconductor package for reproducible benchmarks of gene regulatory network inference

A BenchCouncil View on Benchmarking Emerging and Future Computing

Benchopt: Reproducible, efficient and collaborative optimization benchmarks

Defining and benchmarking open problems in single-cell analysis

NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems

AI Competitions and Benchmarks: The life cycle of challenges and benchmarks

OmniGenBench: Automating Large-scale in-silico Benchmarking for Genomic Foundation Models

From Standardized Data Formats to Standardized Tools for Optimization Algorithm Benchmarking

Guidelines for benchmarking of optimization approaches for fitting mathematical models

Reference environments: A universal tool for reproducibility in computational biology