# Algorithm Development for Black-Box Global Optimization

## Motivation

Optimization problems arise in many science applications, e.g.

• Environment: clean up contaminated groundwater at minimal cost
• Renewable energies: maximize the power generated
• Simulation: minimize the error between the simulation model and observations

• ## Optimization problem characteristics

• Objective function values are computed by a time consuming black-box simulation model

• Derivative information is not available

• Problems are multimodal, i.e., there are several local optima

• Constraints may be computationally cheap, computed by an expensive black-box simulation model, or simple box constraints

• The simulation model may not return a function value (not a bug in the simulation model, e.g., may be some exception case in the simulation model)

• Uncertainty in observation data and stochasticity in forward model evaluation

• Variables can be continuous, mixed-integer, integer, binary

## Challenge

• Gradient-based methods are not applicable because we do not have derivative information

• Evolutionary algorithms require too many expensive function evaluations

• Increasing compute power allows us to increase the accuracy of the simulation models, which leads to increasing compute time for function evaluations

• ## Approach

We use a computationally cheap approximation s(x) (the surrogate model) of the expensive function f(x) in order to predict function values at unsampled points and guide the search for improved solutions:

A general surrogate model optimization algorithm works as follows:

• Step 1: Create an initial experimental design. Evaluate f(x) at the selected points. Fit the surrogate s(x) to the data.

• Step 2: Solve a computationally cheap auxiliary optimization problem on s(x) to select a new evaluation point xnew and compute f(xnew).

• Step 3: If the stopping criterion is not satisfied, update s(x) with the new data and go to Step 2. Otherwise, stop.

• ## Selected applications

Combustion simulations: more to come..

Cosmology: more to come...

Event generator tuning: more to come...

Co-optimization of fuels and engines: more to come...

## Ongoing work

• Development of new algorithms for problems with uncertainty (from noise in observation data, from stochasticity in forward simulation, from model fidelity)

• Development of algorithms for large-scale problems

• For more detailed project descriptions, see LBL Optimization

• ## Contact

For more information, contact Juliane Mueller

## Related publications

T. Takhtaganov, Z. Lukic, J. Mueller, D. Morozov, "Cosmic Inference: Constraining Parameters with Observations and a Highly Limited Number of Simulations", The Astrophysical Journal., 2021. [link].

J. Mueller, J. Park, R. Sahu, C. Varadharajan, B. Arora, B. Faybishenko, D. Agarwal, "Surrogate optimization of deep neural networks for groundwater predictions", Journal of Global Optimization., 2020. [link].

J. Mueller, "An algorithmic framework for the optimization of computationally expensive bi-fidelity black-box problems", INFOR: Information Systems and Operational Research., 2019. [link].

W. Langhans, J. Mueller, and W. Collins, "Optimization of the Eddy-Diffusivity/Mass-Flux shallow cumulus and boundary-layer parameterization using surrogate models", Journal of Advances in Modeling Earth Systems., 2019. [link].

J. Mueller and M. Day, "Surrogate optimization of computationally expensive black-box problems with hidden constraints", INFORMS Journal on Computing., 2019. [link].

T. Takhtaganov, J. Mueller "Adaptive Gaussian process surrogates for Bayesian inference", preprint, 2018 [link].

O. Karslioglu, M. Gehlmann, J. Mueller, S. Nemsak, J. Sethian, A. Kaduwela, H. Bluhm, C.S. Fadley "An Efficient Algorithm for Automatic Structure Optimization in X-ray Standing-Wave Experiments", Journal of Electron Spectroscopy and Related Phenomena, 2018 [link].

G. Conti, S Nemsak, C.-T. Kuo, M. Gehlmann, C. Conlon, A. Keqi, A. Rattanachata, O. Karslioglu, J. Mueller, J. Sethian, H. Bluhm, J.E. Rault, J.P. Rueff, H. Fang, A. Javey, C.S. Fadley "Characterization of free standing InAs quantum membranes by standing wave hard x-ray photoemission spectroscopy", APL Materials, 2018 [link].

J. Mueller, J. Woodbury "GOSAC: Global Optimization with Surrogate Approximation of Constraints", Journal of Global Optimization, 2017 [link].

J. Mueller "SOCEMO: Surrogate Optimization of Computationally Expensive Multi-Objective Problems", INFORMS Journal on Computing, 2017 [link].

J. Mueller, "MISO: Mixed-Integer Surrogate Optimization Framework", Optimization and Engineering, to appear, 2015 [link].

J. Mueller, R. Paudel, N. Mahowald, C. Shoemaker, "CH4 Parameter Estimation in CLM4.5bgc Using Surrogate Global Optimization", Geoscientific Model Development, 8:141-207, 2015 [pdf].