The surrogate model is what the optimizer thinks the scoring function looks like. It is part of the mechanism used to choose optimal configurations.
You can generate predictions from the surrogate model (effectively asking OPTaaS to guess what the scoring function may be at a certain point) at any set of arbitrary configuration points.
from mindfoundry.optaas.client.client import OPTaaSClient
client = OPTaaSClient('https://optaas.mindfoundry.ai', '<Your OPTaaS API key>')
from mindfoundry.optaas.client.parameter import FloatParameter
from mindfoundry.optaas.client.client import Goal
task = client.create_task(
title='Basic 2D Example',
parameters=[
FloatParameter('x', minimum=-3, maximum=1),
FloatParameter('y', minimum=-6, maximum=21)
],
goal=Goal.min,
)
def scoring_function(x, y):
''' A simple well with min at 0, 0'''
score = x**2 + y**2
return score
best_result = task.run(scoring_function, max_iterations=20)
print("Best Result:", best_result)
Running task "Basic 2D Example" for 20 iterations (no score threshold set) Iteration: 0 Score: 57.25 Configuration: {'x': -1.0, 'y': 7.5} Iteration: 1 Score: 207.0625 Configuration: {'x': -2.0, 'y': 14.25} Iteration: 2 Score: 0.5625 Configuration: {'x': 0.0, 'y': 0.75} Iteration: 3 Score: 19.265625 Configuration: {'x': -1.5, 'y': 4.125} Iteration: 4 Score: 310.890625 Configuration: {'x': 0.5, 'y': 17.625} Iteration: 5 Score: 124.515625 Configuration: {'x': -2.5, 'y': 10.875} Iteration: 6 Score: 7.140625 Configuration: {'x': -0.5, 'y': -2.625} Iteration: 7 Score: 3.94140625 Configuration: {'x': -1.75, 'y': -0.9375} Iteration: 8 Score: 157.87890625 Configuration: {'x': 0.25, 'y': 12.5625} Iteration: 9 Score: 380.53515625 Configuration: {'x': -2.75, 'y': 19.3125} Iteration: 10 Score: 1.0128486056914057 Configuration: {'x': 0.99999998, 'y': 0.11335186673101204} Iteration: 11 Score: 0.00927530250495255 Configuration: {'x': 0.03888422866549611, 'y': 0.0881097001813192} Iteration: 12 Score: 0.0012117009114401805 Configuration: {'x': 0.0050892493390089985, 'y': 0.03443545342529378} Iteration: 13 Score: 0.0006806662021513174 Configuration: {'x': 0.006729942231381237, 'y': 0.025206627694191637} Iteration: 14 Score: 0.0006364356966787191 Configuration: {'x': 0.0010303479825333319, 'y': 0.02520662769419203} Iteration: 15 Score: 0.00022768157398298682 Configuration: {'x': -0.0005074972550283101, 'y': 0.015080584223402142} Iteration: 16 Score: 0.00024225926376497916 Configuration: {'x': 0.0005289767480935436, 'y': 0.015555688585368233} Iteration: 17 Score: 0.0002342686143728285 Configuration: {'x': 6.452081614416779e-05, 'y': 0.015305699965604729} Iteration: 18 Score: 0.00021932618498741224 Configuration: {'x': -0.0006874418906318507, 'y': 0.014793701654231666} Iteration: 19 Score: 0.00021932619083313206 Configuration: {'x': -0.0006874425416874602, 'y': 0.014793701821552655} Task Completed Best Result: { 'configuration': { 'type': 'exploitation', 'values': {'x': -0.0006874418906318507, 'y': 0.014793701654231666}}, 'score': 0.00021932618498741224, 'user_defined_data': None}
The surrogate model should predict a fairly low score with high confidence, since it has been exploring the vicinity of this point.
interesting_configs = [{'x': 0.0, 'y': 0.0}]
predictions = task.get_surrogate_predictions(interesting_configs)
[(p.mean, p.variance) for p in predictions]
[(0.0047830747120087835, 0.00012424807215192135)]
The surrogate model should be significantly less confident, as there were no evaluations near this point.
far_away_points = [{'x': 1.0, 'y': 20.0}, {'x': -1.0, 'y': -6.0}]
predictions = task.get_surrogate_predictions(far_away_points)
[(p.mean, p.variance) for p in predictions]
[(399.7652778647905, 0.04206749064962412), (35.073995608525436, 0.0782837406899546)]
The predictions are quite accurate, as the function is quite simple, so the surrogate is able to learn it fairly quickly. The increased variance reflect the lower certainty, as expected.
Here's an article we wrote on how the surrogate works: https://towardsdatascience.com/the-intuitions-behind-bayesian-optimization-with-gaussian-processes-7e00fcc898a0