Abstract:Offline model-based optimization (MBO) aims to identify a design that maximizes a black-box function using only a fixed, pre-collected dataset of designs and their corresponding scores. A common approach in offline MBO is to train a regression-based surrogate model by minimizing mean squared error (MSE) and then find the best design within this surrogate model by different optimizers (e.g., gradient ascent). However, a critical challenge is the risk of out-of-distribution errors, i.e., the surrogate model may typically overestimate the scores and mislead the optimizers into suboptimal regions. Prior works have attempted to address this issue in various ways, such as using regularization techniques and ensemble learning to enhance the robustness of the model, but it still remains. In this paper, we argue that regression models trained with MSE are not well-aligned with the primary goal of offline MBO, which is to select promising designs rather than to predict their scores precisely. Notably, if a surrogate model can maintain the order of candidate designs based on their relative score relationships, it can produce the best designs even without precise predictions. To validate it, we conduct experiments to compare the relationship between the quality of the final designs and MSE, finding that the correlation is really very weak. In contrast, a metric that measures order-maintaining quality shows a significantly stronger correlation. Based on this observation, we propose learning a ranking-based model that leverages learning to rank techniques to prioritize promising designs based on their relative scores. We show that the generalization error on ranking loss can be well bounded. Empirical results across diverse tasks demonstrate the superior performance of our proposed ranking-based models than twenty existing methods.

How to Select the Appropriate One from the Trained Models for Model-Based OPE.

Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

Beyond Reward: Offline Preference-guided Policy Optimization

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators

MOPO: Model-based Offline Policy Optimization

Off-Policy Evaluation via Off-Policy Classification

When is Offline Policy Selection Sample Efficient for Reinforcement Learning?

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

Automated Off-Policy Estimator Selection via Supervised Learning

Model-Based Offline Weighted Policy Optimization (Student Abstract)

Mild Policy Evaluation for Offline Actor–Critic

Offline Model-Based Adaptable Policy Learning for Decision-Making in Out-of-Support Regions

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

Probabilistic Offline Policy Ranking with Approximate Bayesian Computation

When is Off-Policy Evaluation (Reward Modeling) Useful in Contextual Bandits? A Data-Centric Perspective

Off-Policy Evaluation in Doubly Inhomogeneous Environments

Offline Model-Based Optimization by Learning to Rank

Policy Optimization with Model-based Explorations

Offline Policy Evaluation in Large Action Spaces Via Outcome-Oriented Action Grouping

Optimistic Model Rollouts for Pessimistic Offline Policy Optimization