Abstract:This paper investigates the application of the transformer architecture in protein folding, as exemplified by DeepMind's AlphaFold project, and its implications for the understanding of large language models as models of language. The prevailing discourse often assumes a ready-made analogy between proteins -- encoded as sequences of amino acids -- and natural language -- encoded as sequences of discrete symbols. Instead of assuming as given the linguistic structure of proteins, we critically evaluate this analogy to assess the kind of knowledge-making afforded by the transformer architecture. We first trace the analogy's emergence and historical development, carving out the influence of structural linguistics on structural biology beginning in the mid-20th century. We then examine three often overlooked pre-processing steps essential to the transformer architecture, including subword tokenization, word embedding, and positional encoding, to demonstrate its regime of representation based on continuous, high-dimensional vector spaces, which departs from the discrete, semantically demarcated symbols of language. The successful deployment of transformers in protein folding, we argue, discloses what we consider a non-linguistic approach to token processing intrinsic to the architecture. We contend that through this non-linguistic processing, the transformer architecture carves out unique epistemological territory and produces a new class of knowledge, distinct from established domains. We contend that our search for intelligent machines has to begin with the shape, rather than the place, of intelligence. Consequently, the emerging field of critical AI studies should take methodological inspiration from the history of science in its quest to conceptualize the contributions of artificial intelligence to knowledge-making, within and beyond the domain-specific sciences.

TRILL: ORCHESTRATING MODULAR DEEP-LEARNING WORKFLOWS FOR DEMOCRATIZED, SCALABLE PROTEIN ANALYSIS AND ENGINEERING

ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing

Benchmarking Protein Language Models for Protein Crystallization

Synthesizing Proteins on the Graphics Card. Protein Folding and the Limits of Critical AI Studies

Modeling Protein Using Large-scale Pretrain Language Model

Transformer-based deep learning for predicting protein properties in the life sciences

OPUS-Fold3: a Gradient-Based Protein All-Atom Folding and Docking Framework on TensorFlow

Deep Learning in Protein Structural Modeling and Design

xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein

Prop3D: A flexible, Python-based platform for machine learning with protein structural properties and biophysical data

When Geometric Deep Learning Meets Pretrained Protein Language Models.

DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning

SaprotHub: Making Protein Modeling Accessible to All Biologists

From sequence to function through structure: deep learning for protein design

Design Proteins Using Large Language Models: Enhancements and Comparative Analyses

AMPL: A Data-Driven Modeling Pipeline for Drug Discovery

Protein identification with deep learning: from abc to xyz

How the technologies behind self‐driving cars, social networks, ChatGPT, and DALL‐E2 are changing structural biology

Structure-based protein design with deep learning

Machine learning for functional protein design