OmniPred: Language Models as Universal Regressors

Xingyou Song,Oscar Li,Chansoo Lee,Bangding Yang,Daiyi Peng,Sagi Perel,Yutian Chen

2024-03-05

Abstract:Over the broad landscape of experimental design, regression has been a powerful tool to accurately predict the outcome metrics of a system or model given a set of parameters, but has been traditionally restricted to methods which are only applicable to a specific task. In this paper, we propose OmniPred, a framework for training language models as universal end-to-end regressors over $(x,y)$ evaluation data from diverse real world experiments. Using data sourced from Google Vizier, one of the largest blackbox optimization databases in the world, our extensive experiments demonstrate that through only textual representations of mathematical parameters and values, language models are capable of very precise numerical regression, and if given the opportunity to train over multiple tasks, can significantly outperform traditional regression models.

Machine Learning,Artificial Intelligence,Computation and Language,Databases

What problem does this paper attempt to address?

The paper mainly discusses how to use language models as universal regressors (OmniPred) to solve regression problems in experimental design. Traditional regression methods are usually limited to specific tasks, while OmniPred proposes a text-based framework that can handle diverse real-world experimental data from different domains. By training on a large amount of black-box optimization data obtained from Google Vizier, the language model can achieve precise numerical regression and outperform traditional regression models such as multilayer perceptron and boosting trees in multitask learning.

The core contributions of the paper include:
1. Proposing OmniPred, a scalable and simple text-based indicator prediction framework applicable to general input spaces.
2. With only text and token representations, OmniPred can achieve high-precision indicator prediction on experimental design data.
3. By simultaneously learning multiple different input spaces and targets, OmniPred surpasses traditional regression models in many cases.
4. Even on unseen new tasks, OmniPred can maintain the benefits of transfer learning with local fine-tuning using a small amount of new evaluation data.

The research found that language models can handle variable-length inputs and contextual metadata, which has been verified in reward modeling (e.g., reinforcement learning). However, the accuracy of their numerical processing in complex and numerical-based data in experimental design has not been fully demonstrated. The paper addresses this technical challenge and demonstrates that language models can achieve high-precision numerical prediction through token representations.

The experimental section showcases the model's simultaneous regression capability on multiple tasks and different input spaces, as well as the improvements in prediction accuracy through multitask training and fine-tuning. Additionally, the paper discusses the impact of pretraining and fine-tuning strategies on adaptability to new tasks.