Abstract:Latent Space Bayesian Optimization (LSBO) combines generative models, typically Variational Autoencoders (VAE), with Bayesian Optimization (BO) to generate de-novo objects of interest. However, LSBO faces challenges due to the mismatch between the objectives of BO and VAE, resulting in poor exploration capabilities. In this paper, we propose novel contributions to enhance LSBO efficiency and overcome this challenge. We first introduce the concept of latent consistency/inconsistency as a crucial problem in LSBO, arising from the VAE-BO mismatch. To address this, we propose the Latent Consistent Aware-Acquisition Function (LCA-AF) that leverages consistent points in LSBO. Additionally, we present LCA-VAE, a novel VAE method that creates a latent space with increased consistent points through data augmentation in latent space and penalization of latent inconsistencies. Combining LCA-VAE and LCA-AF, we develop LCA-LSBO. Our approach achieves high sample-efficiency and effective exploration, emphasizing the significance of addressing latent consistency through the novel incorporation of data augmentation in latent space within LCA-VAE in LSBO. We showcase the performance of our proposal via de-novo image generation and de-novo chemical design tasks.
What problem does this paper attempt to address?
The problems that this paper attempts to solve are two key issues existing in the existing Latent Space Bayesian Optimization (LSBO) methods:
1. **Latent Consistency/Inconsistency**: In LSBO, the mismatch between the objectives of the Variational Auto - Encoder (VAE) and Bayesian Optimization (BO) leads to the latent space consistency problem. Specifically, when a latent variable \(z\) is re - encoded back into the latent space after decoding, if the resulting point is different from the original point, then this point is called a latent inconsistent point. This inconsistency will cause the black - box function values obtained during the optimization process to not match the actual evaluation points, thus affecting the optimization effect.
2. **Limited Latent Consistent Points**: In low - density regions (i.e., regions with fewer training instances), the number of latent consistent points is very limited, which restricts the exploration ability of LSBO. Since exploration usually occurs in low - density regions, and these regions often have latent inconsistencies, it is difficult for LSBO to effectively explore new, unseen regions.
To solve these problems, the author proposes the following methods:
- **Latent Consistent Aware - Acquisition Function (LCA - AF)**: By modifying the Acquisition Function (AF) to make it only focus on latent consistent points, thus avoiding the influence of latent inconsistency. LCA - AF judges whether the latent variable converges to a consistent point through multiple rounds of encoding and decoding, and calculates the AF values of these consistent points.
- **Latent Consistent Aware - VAE (LCA - VAE)**: By introducing the Latent Consistency Loss (LCL), a penalty term is added to the objective function of VAE to increase the number of latent consistent points, especially in low - density regions. LCA - VAE achieves this through data augmentation in the latent space.
Combining LCA - AF and LCA - VAE, the author develops a new LSBO method, called Latent Consistency Aware - LSBO (LCA - LSBO). The experimental results in image generation and de novo chemical design tasks show that LCA - LSBO can explore samples more efficiently and improve the optimization performance.
### Summary of Mathematical Formulas
- **Latent Consistency Loss (LCL)**:
\[
LCL(\hat{z})=\|\hat{z}-\hat{z}_{1}\|^{2}
\]
where \(\hat{z}_{1}=f_{\text{enc}}^{\phi}(f_{\text{dec}}^{\theta}(\hat{z}))\).
- **LCA - VAE Objective Function**:
\[
J_{\text{LCA - VAE}}(\phi, \theta)=J_{\text{VAE}}(\phi, \theta)-\gamma \mathbb{E}_{\hat{z} \sim p_{\text{ref}}}[LCL(\hat{z})]
\]
where \(J_{\text{VAE}}(\phi, \theta)\) is the objective function of the standard VAE, \(\gamma>0\) is a hyperparameter used to balance the trade - off between the two terms.
Through these improvements, LCA - LSBO can explore and optimize more effectively in the latent space, especially in low - density regions, thus improving the overall performance.