GPyOpt: The tool for Bayesian Optimization

Written by Javier Gonzalez, Amazon Reseach Cambridge, UK.

Reference Manual index

Last updated Monday, 22 May 2017.

=====================================================================================================

  1. What is GPyOpt?

  2. Installation and setup

  3. First steps with GPyOpt and Bayesian Optimization

  4. Alternative GPyOpt interfaces: Standard, Modular and Spearmint

  5. What can I do with GPyOpt?

    1. Bayesian optimization with arbitrary restrictions.
    2. Parallel Bayesian optimization.
    3. Mixing different types of variables.
    4. Armed bandits problems.
    5. Tuning Scikit-learn models.
    6. Integrating model hyperparameters.
    7. Input warping.
    8. Using various cost evaluations functions.
    9. Contextual variables.
    10. External objective evaluation.
  6. Currently supported models, acquisitions and initial designs

    1. Supported initial designs.
    2. Implementing new models.
    3. Implementing new acquisistion.

=====================================================================================================

1. What is GPyOpt?

GPyOpt is a tool for optimization (minimization) of black-box functions using Gaussian processes. It has been implemented in Python by the group of Machine Learning (at SITraN) of the University of Sheffield.

GPyOpt is based on GPy, a library for Gaussian process modeling in Python. Here you can also find some notebooks about GPy functionalities. GPyOpt is a tool for Bayesian Optimization but we also use it for academic dissemination in Gaussian Processes Summer Schools, where you can find some extra labs and a variety of talks on Gaussian processes and Bayesian optimization.

The purpose of this manual is to provide a guide to use GPyOpt. The framework is BSD-3 licensed and we welcome collaborators to develop new functionalities. If you have any question or suggestions about the notebooks, please write an issue in the GitHub repository.

2. Installation and setup

The simplest way to install GPyOpt is using pip. Ubuntu users can do:

sudo apt-get install python-pip
pip install gpyopt

If you'd like to install from source, or want to contribute to the project (e.g. by sending pull requests via github), read on. Clone the repository in GitHub and add it to your $PYTHONPATH.

git clone [email protected]:SheffieldML/GPyOpt.git ~/SheffieldML
echo 'PYTHONPATH=$PYTHONPATH:~/SheffieldML' >> ~/.bashrc

There are a number of dependencies that you may need to install. Three of them are needed to ensure the good behaviour of the package. These are, GPy, numpy and scipy. Other dependencies, such as DIRECT, cma and pyDOE are optional and only are required for in some options of the module. All of them are pip installable.

3. First steps with GPyOpt and Bayesian Optimization

The tutorial Introduction to Bayesian Optimization with GPyOpt reviews some basic concepts on Bayesian optimization and shows some basic GPyOpt functionalities. It is a manual for beginners who want to start using the package.

Drawing

4. Alternative GPyOpt interfaces: Standard, Modular and Spearmint

GPyOpt has different interfaces oriented to different types of users. Apart from the general interface (detailed in the introductory manual) you can use GPyOpt in a modular way: you can implement and use your some elements of the optimization process, such us a new model or acquisition function, but still use the main backbone of the package. You can check the GPyOpt: Modular Bayesian Optimization notebook if you are interested on using GPyOpt this way.

Also, we have developed and GPyOpt interface with Spearmint but this only covers some general features that are available in GPyOpt.

5. What can I do with GPyOpt?

There are several options implemented in GPyOpt that allows to cover a wide range of specific optimization problems. We have implemented a collection of notebooks to explain these functionalities separately but they can be easily combined.

5.1. Bayesian optimization with arbitrary restrictions

With GPyOpt you can solve optimization problems with arbitrary non trivial restrictions. Have a look to the notebook GPyOpt: Bayesian Optimization with fixed constraints if you want to know more about how to use GPyOpt in these type of problems.

5.2 Parallel Bayesian optimization

The main bottleneck when using Bayesian optimization is the cost of evaluating the objective function. In the notebook GPyOpt: parallel Bayesian Optimization you can learn more about the different parallel methods currently implemented in GPyOpt.

5.3 Mixing different types of variables

In GPyOpt you can easily combine different types of variables in the optimization. Currently you can use discrete an continuous variables. The way GPyOpt handles discrete variables is by marginally optimizing the acquisition functions over combinations of feasible values. This may slow down the optimization if many discrete variables are used but it avoids rounding errors. See the notebook entitled GPyOpt: mixing different types of variables for further details.

5.4 Armed bandits problems

Armed bandits optimization problems are a particular case of Bayesian Optimization that appear when the domain of the function to optimize is entirely discrete. This has several advantages with respect to optimize in continuous domains. The most remarkable is that the optimization of the acquisition function can be done by taking the $arg min$ of all candidate points while the rest of the BO theory applies. In the notebook GPyOpt: armed bandits optimization you can check how to use GPyOpt in these types of problems.

5.5 Tuning scikit-learn models

Scikit-learn is a very popular library with a large variety of useful methods in Machine Learning. Have a look to the notebook GPyOpt: configuring Scikit-learn methods to learn how learn the parameters of Scikit-learn methods using GPyOpt. You will learn how to automatically tune the parameters of a Support Vector Regression.

5.6 Integrating the model hyper parameters

Maximum Likelihood estimation can be a very instable choice to tune the surrogate model hyper parameters, especially in the fist steps of the optimization. When using a GP model as a surrogate of your function you can integrate the most common acquisition functions with respect to the parameters of the model. Check the notebook GPyOpt: integrating model hyperparameters to check how to use this option.

5.7 Input warping

TODO

5.8 Using various cost evaluation functions

The cost of evaluating the objective can be a crucial factor in the optimization process. Check the notebook GPyOpt: dealing with cost functions to learn how to use arbitrary cost functions, including the objective evaluation time.

5.9 Contextual variables

During the optimization phase, you may want to fix the value of some of the variables. These variables are called context as they are part of the objective but are fixed when the aquisition is optimized, you can learn how to use them in this notebook.

5.10 External objective evaluation

If you cannot define your objective function in Python, you have an option of evaluating it externally, and calling GPyOpt to suggest the next locations to evaluate. This approach is illustrated here.

6. Currently supported models and acquisitions.

Currently, you can initialize your model with three types of initial designs:

  • Random,
  • Latin Hypercubes and
  • Sobol sequences.

Check this notebook to check how these designs look in a small example.

6.1 Implemeting new models

The currently available models in GPyOpt are:

  • Standard GPs (with MLE and HMC inference)
  • Sparse GPs
  • Warperd GPs (both for the input and the output)
  • Random Forrests

On top of this, if you want to implement your own model you can learn how to that in this notebook.

6.2 Implementing new acquisitions

The currently available acquisition functions in GPyOpt are:

  • Expected Improvement.
  • Maximum Probability of Improvement.
  • Lower Confidence Bound.

On top of this, if you want to implement your own model you can learn how to that in this notebook.