Unstructured High-Dimensional Bayesian Optimization

Report | Slides | GitHub repository

Motivation

I completed this credit-bearing undergraduate research project (MA3288) with Prof. Jonathan Scarlett in the summer of 2024. The project began with the dimensionality-scaled lengthscale prior (DSP) proposed by Hvarfner et al. For two points sampled uniformly from a $D$-dimensional hypercube, the mean Euclidean distance grows on the order of $\sqrt{D}$. A stationary kernel with a fixed lengthscale therefore makes points appear progressively less correlated as dimension increases. DSP scales the prior mean of the Gaussian-process lengthscales by $\sqrt{D}$, imposing a low-complexity prior without assuming locality or a known active subspace.

The following figure from Hvarfner et al. shows the connection between lengthscale and model complexity:

Lengthscale evolution

I investigated whether DSP could be improved by addressing unknown hyperparameters. In low-dimensional Bayesian optimization, methods such as adaptive GP-UCB gradually shrink lengthscales to avoid persistent underfitting. Since DSP begins with a strong low-complexity assumption, a similar cooldown strategy appears natural.

The difficulty is that a global cooldown preserves the initial relative ordering of automatic-relevance-determination (ARD) lengthscales. In high dimensions, those early estimates can be unreliable, and distinguishing active from inactive dimensions is itself part of the problem. I therefore tested meta-strategies that evolve either the prior or posterior lengthscale distribution while allowing the relative importance of dimensions to change.

Soft Winsorization

I also explored simplifying the objective observations. Multiplying all observations by a constant does not help when BoTorch standardizes them. Instead, I applied a sigmoid transformation after standardization so that extreme observations are compressed toward the center. I called this soft Winsorization because it smoothly approximates the clipping used by ordinary Winsorization. In regression experiments, it sometimes made a deliberately low-complexity GP less sensitive to extreme observations.

What I learned

None of these methods consistently outperformed DSP across the benchmark suite. Initially, I attributed this to insufficient tuning. A later project on UMAP, however, suggested a more structural concern: although pairwise distance grows like $\sqrt{D}$, its spread grows much more slowly. Dividing distances by $\sqrt{D}$ therefore compresses their relative variation, so that points can become nearly indistinguishable to the kernel.

Why, then, does vanilla Bayesian optimization work surprisingly well with DSP? My interpretation is that numerical stability deserves substantial credit: the prior supplies useful gradients for lengthscale estimation, and larger lengthscales also make acquisition functions easier to optimize. The low-complexity interpretation remains valuable, but it may not be the only—or even the dominant—mechanism. This project was a useful reminder that a method can work for reasons that are subtler than its first theoretical explanation.