Abstract:Expressive large-scale neural networks enable training powerful models for prediction tasks. However, in many engineering and science domains, such models are intended to be used not just for prediction, but for design -- e.g., creating new proteins that serve as effective therapeutics, or creating new materials or chemicals that maximize a downstream performance measure. Thus, researchers have recently grown an interest in building deep learning methods that solve offline \emph{model-based optimization} (MBO) problems, in which design candidates are optimized with respect to surrogate models learned from offline data. However, straightforward application of predictive models that are effective at predicting in-distribution properties of a design are not necessarily the best suited for use in creating new designs. Thus, the most successful algorithms that tackle MBO draw inspiration from reinforcement learning and generative modeling to meet the in-distribution constraints. Meanwhile, recent theoretical works have observed that exploiting the structure of the target black-box function is an effective strategy for solving MBO from offline data. Unfortunately, discovering such structure remains an open problem. In this paper, following first principles, we develop a model that learns the structure of an MBO task and empirically leads to improved designs. To this end, we introduce \emph{Cliqueformer} -- a scalable transformer-based architecture that learns the black-box function's structure in the form of its \emph{functional graphical model} (FGM), thus bypassing the problem of distribution shift, previously tackled by conservative approaches. We evaluate Cliqueformer on various tasks, ranging from high-dimensional black-box functions from MBO literature to real-world tasks of chemical and genetic design, consistently demonstrating its state-of-the-art performance.

Optformer: Beyond Transformer for Black-box Optimization

Evolution Transformer: In-Context Evolutionary Optimization

EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm

Transformer-Based Learned Optimization

B2Opt: Learning to Optimize Black-box Optimization with Little Budget

Evolutionary Transfer Optimization - A New Frontier in Evolutionary Computation Research

Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model

TransOpt: Transformer-based Representation Learning for Optimization Problem Classification

Optimization Based on Nonlinear Transformation in Decision Space

OPT-GAN: A Broad-Spectrum Global Optimizer for Black-box Problems by Learning Distribution

Cliqueformer: Model-Based Optimization with Structured Transformers

CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention

Xformer: Hybrid X-Shaped Transformer for Image Denoising

DECN: Evolution Inspired Deep Convolution Network for Black-box Optimization

Transformers from an Optimization Perspective

AutoFormer: Searching Transformers for Visual Recognition

Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization

Visformer: The Vision-friendly Transformer

Evolutionary Dual-Stream Transformer.

ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning

How Well Can Transformers Emulate In-context Newton's Method?