Better optimizer for image generation

This notebook is based on neural-style.ipynb plus one key observation: the L-BFGS-B optimization algorithm used always gets stuck in local minima, and the local minima seems to always have a characteristic pattern of artifacts with a high spatial frequency in the generated image. Can we suppress those artifacts by simple blurring?

Setup - same as original notebook

In [1]:
%matplotlib inline
import importlib
import utils2; importlib.reload(utils2)
from utils2 import *

from scipy.optimize import fmin_l_bfgs_b
from scipy.misc import imsave
from keras import metrics

from vgg16_avg import VGG16_Avg
Using TensorFlow backend.
/home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
In [2]:
limit_mem()
In [3]:
img = Image.open('/home/ubuntu/www.platform.ai/data/imagenet-sample-train/train/n01558993/n01558993_9684.JPEG'); img
Out[3]:
In [4]:
rn_mean = np.array([123.68, 116.779, 103.939], dtype=np.float32)
preproc = lambda x: (x - rn_mean)[:, :, :, ::-1]
In [5]:
img_arr = preproc(np.expand_dims(np.array(img), 0))
shp = img_arr.shape
In [6]:
deproc = lambda x,s: np.clip(x.reshape(s)[:, :, :, ::-1] + rn_mean, 0, 255)
In [7]:
model = VGG16_Avg(include_top=False)
In [8]:
layer = model.get_layer('block5_conv1').output
In [9]:
layer_model = Model(model.input, layer)
targ = K.variable(layer_model.predict(img_arr))
In [10]:
class Evaluator(object):
    def __init__(self, f, shp): self.f, self.shp = f, shp
        
    def loss(self, x):
        loss_, self.grad_values = self.f([x.reshape(self.shp)])
        return loss_.astype(np.float64)

    def grads(self, x): return self.grad_values.flatten().astype(np.float64)
In [11]:
loss = metrics.mse(layer, targ)
grads = K.gradients(loss, model.input)
fn = K.function([model.input], [loss]+grads)
evaluator = Evaluator(fn, shp)

Modified code after this point

Let's see if it's possible to recreate the input image from the block5_conv1 activations, but with a twist: blur the image whenever the optimizer gets stuck.

First, a function that blurs an image in-place (channel by channel) using simple gaussian blur with sigma = 1

In [113]:
import scipy.ndimage.filters

def blurify(x):
    x = x.reshape(shp)
    for k in range(3):
        x[:,:,:,k] = scipy.ndimage.filters.gaussian_filter(x[:,:,:,k], 1)
    return x

Then, modify solve_image to detect when a local minimum has been reached (very small change per step < 0.001), and blur the image then continue. This may need to be tuned, it is possible that it detects local minima somewhat too early (could try change per step < 0.0001 instead).

In [12]:
def solve_image(eval_obj, niter, x):
    last_min_val = 1000 # Start from some arbitrary very high number # TODO init with None
    loss_history = {}
    for i in range(niter):
        x, min_val, info = fmin_l_bfgs_b(eval_obj.loss, x.flatten(),
                                         fprime=eval_obj.grads, maxfun=20)
        x = np.clip(x, -127,127)
        print('Iter: %d  Loss: %f' %(i, min_val))
        if abs(last_min_val - min_val) < 0.001:
            x = blurify(x)
            print('Blurify')
        last_min_val = min_val
        loss_history[i] = min_val
        imsave('/home/ubuntu/results/res_at_iteration_%d.png' %(i), (deproc(x.copy(), shp)[0]))
    return x, loss_history

Start from all zeros. Noise vs all zeros doesn't seem to make any difference in practice...

In [13]:
rand_img = lambda shape: np.random.uniform(-2.5, 2.5, shape)/100
# x = rand_img(shp)
x = np.zeros(shp)
plt.imshow(x[0]);

Run for a ridiculous number of iterations

In [21]:
iterations=1000
In [116]:
x, loss_history = solve_image(evaluator, iterations, x)
Iter: 0  Loss: 56.332458
Iter: 1  Loss: 21.307999
Iter: 2  Loss: 11.739264
Iter: 3  Loss: 8.211292
Iter: 4  Loss: 6.507470
Iter: 5  Loss: 5.483242
Iter: 6  Loss: 4.860817
Iter: 7  Loss: 4.388783
Iter: 8  Loss: 3.996233
Iter: 9  Loss: 3.699281
Iter: 10  Loss: 3.469527
Iter: 11  Loss: 3.264268
Iter: 12  Loss: 3.202618
Iter: 13  Loss: 3.190215
Iter: 14  Loss: 3.190377
Blurify
Iter: 15  Loss: 5.273498
Iter: 16  Loss: 3.830165
Iter: 17  Loss: 3.301814
Iter: 18  Loss: 2.958852
Iter: 19  Loss: 2.727372
Iter: 20  Loss: 2.563292
Iter: 21  Loss: 2.436050
Iter: 22  Loss: 2.337002
Iter: 23  Loss: 2.315290
Iter: 24  Loss: 2.316147
Blurify
Iter: 25  Loss: 3.951824
Iter: 26  Loss: 2.967965
Iter: 27  Loss: 2.604438
Iter: 28  Loss: 2.388319
Iter: 29  Loss: 2.237290
Iter: 30  Loss: 2.131394
Iter: 31  Loss: 2.109755
Iter: 32  Loss: 2.108187
Iter: 33  Loss: 2.108195
Blurify
Iter: 34  Loss: 3.451382
Iter: 35  Loss: 2.599714
Iter: 36  Loss: 2.289659
Iter: 37  Loss: 2.129872
Iter: 38  Loss: 2.020915
Iter: 39  Loss: 2.001437
Iter: 40  Loss: 2.001369
Blurify
Iter: 41  Loss: 3.246094
Iter: 42  Loss: 2.457810
Iter: 43  Loss: 2.199605
Iter: 44  Loss: 2.031650
Iter: 45  Loss: 1.950836
Iter: 46  Loss: 1.951602
Blurify
Iter: 47  Loss: 3.095697
Iter: 48  Loss: 2.364367
Iter: 49  Loss: 2.109860
Iter: 50  Loss: 1.964509
Iter: 51  Loss: 1.935248
Iter: 52  Loss: 1.935365
Blurify
Iter: 53  Loss: 2.983280
Iter: 54  Loss: 2.290123
Iter: 55  Loss: 2.053647
Iter: 56  Loss: 1.899210
Iter: 57  Loss: 1.874540
Iter: 58  Loss: 1.874610
Blurify
Iter: 59  Loss: 2.853730
Iter: 60  Loss: 2.220968
Iter: 61  Loss: 2.010096
Iter: 62  Loss: 1.853112
Iter: 63  Loss: 1.801871
Iter: 64  Loss: 1.802014
Blurify
Iter: 65  Loss: 2.734373
Iter: 66  Loss: 2.126369
Iter: 67  Loss: 1.905844
Iter: 68  Loss: 1.902967
Iter: 69  Loss: 1.902974
Blurify
Iter: 70  Loss: 2.715861
Iter: 71  Loss: 2.146672
Iter: 72  Loss: 1.923933
Iter: 73  Loss: 1.792043
Iter: 74  Loss: 1.789842
Iter: 75  Loss: 1.789849
Blurify
Iter: 76  Loss: 2.611001
Iter: 77  Loss: 2.051578
Iter: 78  Loss: 1.843392
Iter: 79  Loss: 1.729595
Iter: 80  Loss: 1.706310
Iter: 81  Loss: 1.706330
Blurify
Iter: 82  Loss: 2.605799
Iter: 83  Loss: 2.003242
Iter: 84  Loss: 1.791754
Iter: 85  Loss: 1.788433
Iter: 86  Loss: 1.788436
Blurify
Iter: 87  Loss: 2.556896
Iter: 88  Loss: 1.965723
Iter: 89  Loss: 1.770293
Iter: 90  Loss: 1.712831
Iter: 91  Loss: 1.712929
Blurify
Iter: 92  Loss: 2.514152
Iter: 93  Loss: 1.947668
Iter: 94  Loss: 1.755485
Iter: 95  Loss: 1.752536
Iter: 96  Loss: 1.752539
Blurify
Iter: 97  Loss: 2.599280
Iter: 98  Loss: 1.963046
Iter: 99  Loss: 1.756613
Iter: 100  Loss: 1.631467
Iter: 101  Loss: 1.628476
Iter: 102  Loss: 1.628479
Blurify
Iter: 103  Loss: 2.485286
Iter: 104  Loss: 1.907258
Iter: 105  Loss: 1.698251
Iter: 106  Loss: 1.694783
Iter: 107  Loss: 1.694786
Blurify
Iter: 108  Loss: 2.429407
Iter: 109  Loss: 1.886185
Iter: 110  Loss: 1.691889
Iter: 111  Loss: 1.656286
Iter: 112  Loss: 1.656301
Blurify
Iter: 113  Loss: 2.426243
Iter: 114  Loss: 1.861527
Iter: 115  Loss: 1.649160
Iter: 116  Loss: 1.530549
Iter: 117  Loss: 1.447638
Iter: 118  Loss: 1.377305
Iter: 119  Loss: 1.374920
Iter: 120  Loss: 1.374924
Blurify
Iter: 121  Loss: 2.228023
Iter: 122  Loss: 1.703411
Iter: 123  Loss: 1.537968
Iter: 124  Loss: 1.457610
Iter: 125  Loss: 1.457708
Blurify
Iter: 126  Loss: 2.265303
Iter: 127  Loss: 1.748284
Iter: 128  Loss: 1.544124
Iter: 129  Loss: 1.493667
Iter: 130  Loss: 1.493702
Blurify
Iter: 131  Loss: 2.348393
Iter: 132  Loss: 1.742352
Iter: 133  Loss: 1.549081
Iter: 134  Loss: 1.431310
Iter: 135  Loss: 1.428494
Iter: 136  Loss: 1.428496
Blurify
Iter: 137  Loss: 2.333052
Iter: 138  Loss: 1.710642
Iter: 139  Loss: 1.509532
Iter: 140  Loss: 1.505881
Iter: 141  Loss: 1.505883
Blurify
Iter: 142  Loss: 2.297161
Iter: 143  Loss: 1.750185
Iter: 144  Loss: 1.548308
Iter: 145  Loss: 1.425497
Iter: 146  Loss: 1.422190
Iter: 147  Loss: 1.422191
Blurify
Iter: 148  Loss: 2.197482
Iter: 149  Loss: 1.677197
Iter: 150  Loss: 1.483543
Iter: 151  Loss: 1.381848
Iter: 152  Loss: 1.370030
Iter: 153  Loss: 1.370035
Blurify
Iter: 154  Loss: 2.191892
Iter: 155  Loss: 1.642742
Iter: 156  Loss: 1.462429
Iter: 157  Loss: 1.357848
Iter: 158  Loss: 1.282951
Iter: 159  Loss: 1.262653
Iter: 160  Loss: 1.262663
Blurify
Iter: 161  Loss: 2.119348
Iter: 162  Loss: 1.571251
Iter: 163  Loss: 1.404496
Iter: 164  Loss: 1.307903
Iter: 165  Loss: 1.289759
Iter: 166  Loss: 1.289763
Blurify
Iter: 167  Loss: 2.142907
Iter: 168  Loss: 1.616430
Iter: 169  Loss: 1.441315
Iter: 170  Loss: 1.333306
Iter: 171  Loss: 1.257399
Iter: 172  Loss: 1.199955
Iter: 173  Loss: 1.170250
Iter: 174  Loss: 1.170300
Blurify
Iter: 175  Loss: 2.055926
Iter: 176  Loss: 1.497262
Iter: 177  Loss: 1.337545
Iter: 178  Loss: 1.248868
Iter: 179  Loss: 1.185790
Iter: 180  Loss: 1.138944
Iter: 181  Loss: 1.138542
Blurify
Iter: 182  Loss: 2.026123
Iter: 183  Loss: 1.497175
Iter: 184  Loss: 1.319403
Iter: 185  Loss: 1.228595
Iter: 186  Loss: 1.157804
Iter: 187  Loss: 1.109451
Iter: 188  Loss: 1.109047
Blurify
Iter: 189  Loss: 1.970667
Iter: 190  Loss: 1.461758
Iter: 191  Loss: 1.292134
Iter: 192  Loss: 1.200104
Iter: 193  Loss: 1.191285
Iter: 194  Loss: 1.191288
Blurify
Iter: 195  Loss: 2.000707
Iter: 196  Loss: 1.452933
Iter: 197  Loss: 1.288172
Iter: 198  Loss: 1.194241
Iter: 199  Loss: 1.193621
Blurify
Iter: 200  Loss: 2.036135
Iter: 201  Loss: 1.472216
Iter: 202  Loss: 1.311498
Iter: 203  Loss: 1.216014
Iter: 204  Loss: 1.180757
Iter: 205  Loss: 1.180771
Blurify
Iter: 206  Loss: 2.095854
Iter: 207  Loss: 1.488455
Iter: 208  Loss: 1.311389
Iter: 209  Loss: 1.210376
Iter: 210  Loss: 1.145466
Iter: 211  Loss: 1.144973
Blurify
Iter: 212  Loss: 1.943347
Iter: 213  Loss: 1.431346
Iter: 214  Loss: 1.277030
Iter: 215  Loss: 1.197426
Iter: 216  Loss: 1.197464
Blurify
Iter: 217  Loss: 2.001596
Iter: 218  Loss: 1.462071
Iter: 219  Loss: 1.297743
Iter: 220  Loss: 1.207827
Iter: 221  Loss: 1.144922
Iter: 222  Loss: 1.092487
Iter: 223  Loss: 1.068815
Iter: 224  Loss: 1.068832
Blurify
Iter: 225  Loss: 1.925748
Iter: 226  Loss: 1.390729
Iter: 227  Loss: 1.238013
Iter: 228  Loss: 1.155514
Iter: 229  Loss: 1.115181
Iter: 230  Loss: 1.115203
Blurify
Iter: 231  Loss: 2.032277
Iter: 232  Loss: 1.455521
Iter: 233  Loss: 1.273718
Iter: 234  Loss: 1.178594
Iter: 235  Loss: 1.111957
Iter: 236  Loss: 1.058639
Iter: 237  Loss: 1.051636
Iter: 238  Loss: 1.051638
Blurify
Iter: 239  Loss: 1.937854
Iter: 240  Loss: 1.377743
Iter: 241  Loss: 1.222426
Iter: 242  Loss: 1.129910
Iter: 243  Loss: 1.071089
Iter: 244  Loss: 1.046510
Iter: 245  Loss: 1.046524
Blurify
Iter: 246  Loss: 1.951347
Iter: 247  Loss: 1.445107
Iter: 248  Loss: 1.229120
Iter: 249  Loss: 1.131885
Iter: 250  Loss: 1.070002
Iter: 251  Loss: 1.049373
Iter: 252  Loss: 1.049378
Blurify
Iter: 253  Loss: 2.000892
Iter: 254  Loss: 1.442447
Iter: 255  Loss: 1.238814
Iter: 256  Loss: 1.126239
Iter: 257  Loss: 1.106446
Iter: 258  Loss: 1.106450
Blurify
Iter: 259  Loss: 1.970386
Iter: 260  Loss: 1.411084
Iter: 261  Loss: 1.211497
Iter: 262  Loss: 1.114671
Iter: 263  Loss: 1.055484
Iter: 264  Loss: 1.054969
Blurify
Iter: 265  Loss: 1.945976
Iter: 266  Loss: 1.404606
Iter: 267  Loss: 1.204569
Iter: 268  Loss: 1.112050
Iter: 269  Loss: 1.078671
Iter: 270  Loss: 1.078685
Blurify
Iter: 271  Loss: 1.902259
Iter: 272  Loss: 1.372724
Iter: 273  Loss: 1.207352
Iter: 274  Loss: 1.096894
Iter: 275  Loss: 1.071516
Iter: 276  Loss: 1.071519
Blurify
Iter: 277  Loss: 1.846668
Iter: 278  Loss: 1.327475
Iter: 279  Loss: 1.171771
Iter: 280  Loss: 1.121377
Iter: 281  Loss: 1.121382
Blurify
Iter: 282  Loss: 1.943167
Iter: 283  Loss: 1.366761
Iter: 284  Loss: 1.179744
Iter: 285  Loss: 1.079134
Iter: 286  Loss: 1.010223
Iter: 287  Loss: 0.979561
Iter: 288  Loss: 0.979587
Blurify
Iter: 289  Loss: 1.760459
Iter: 290  Loss: 1.305357
Iter: 291  Loss: 1.131122
Iter: 292  Loss: 1.036079
Iter: 293  Loss: 0.976252
Iter: 294  Loss: 0.956948
Iter: 295  Loss: 0.956960
Blurify
Iter: 296  Loss: 1.809328
Iter: 297  Loss: 1.269864
Iter: 298  Loss: 1.097080
Iter: 299  Loss: 1.053950
Iter: 300  Loss: 1.053956
Blurify
Iter: 301  Loss: 1.829281
Iter: 302  Loss: 1.293646
Iter: 303  Loss: 1.122176
Iter: 304  Loss: 1.029310
Iter: 305  Loss: 1.010332
Iter: 306  Loss: 1.010334
Blurify
Iter: 307  Loss: 1.787094
Iter: 308  Loss: 1.287532
Iter: 309  Loss: 1.126492
Iter: 310  Loss: 1.034710
Iter: 311  Loss: 0.999540
Iter: 312  Loss: 0.999548
Blurify
Iter: 313  Loss: 1.849936
Iter: 314  Loss: 1.279453
Iter: 315  Loss: 1.108685
Iter: 316  Loss: 1.014677
Iter: 317  Loss: 1.013999
Blurify
Iter: 318  Loss: 1.859227
Iter: 319  Loss: 1.323013
Iter: 320  Loss: 1.137218
Iter: 321  Loss: 1.039899
Iter: 322  Loss: 0.960291
Iter: 323  Loss: 0.920928
Iter: 324  Loss: 0.920947
Blurify
Iter: 325  Loss: 1.762552
Iter: 326  Loss: 1.231276
Iter: 327  Loss: 1.072776
Iter: 328  Loss: 0.985171
Iter: 329  Loss: 0.928061
Iter: 330  Loss: 0.927485
Blurify
Iter: 331  Loss: 1.772559
Iter: 332  Loss: 1.221562
Iter: 333  Loss: 1.061660
Iter: 334  Loss: 0.973697
Iter: 335  Loss: 0.939623
Iter: 336  Loss: 0.939630
Blurify
Iter: 337  Loss: 1.757710
Iter: 338  Loss: 1.246606
Iter: 339  Loss: 1.061812
Iter: 340  Loss: 0.969069
Iter: 341  Loss: 0.912672
Iter: 342  Loss: 0.885523
Iter: 343  Loss: 0.885543
Blurify
Iter: 344  Loss: 1.700181
Iter: 345  Loss: 1.188488
Iter: 346  Loss: 1.031597
Iter: 347  Loss: 0.947435
Iter: 348  Loss: 0.924868
Iter: 349  Loss: 0.924874
Blurify
Iter: 350  Loss: 1.739188
Iter: 351  Loss: 1.231966
Iter: 352  Loss: 1.065842
Iter: 353  Loss: 0.960356
Iter: 354  Loss: 0.932957
Iter: 355  Loss: 0.932958
Blurify
Iter: 356  Loss: 1.673444
Iter: 357  Loss: 1.216911
Iter: 358  Loss: 1.053849
Iter: 359  Loss: 0.963652
Iter: 360  Loss: 0.903687
Iter: 361  Loss: 0.903160
Blurify
Iter: 362  Loss: 1.778198
Iter: 363  Loss: 1.258198
Iter: 364  Loss: 1.064842
Iter: 365  Loss: 0.968148
Iter: 366  Loss: 0.900130
Iter: 367  Loss: 0.880332
Iter: 368  Loss: 0.880339
Blurify
Iter: 369  Loss: 1.652490
Iter: 370  Loss: 1.191633
Iter: 371  Loss: 1.010065
Iter: 372  Loss: 0.931983
Iter: 373  Loss: 0.872074
Iter: 374  Loss: 0.848909
Iter: 375  Loss: 0.848920
Blurify
Iter: 376  Loss: 1.637472
Iter: 377  Loss: 1.171467
Iter: 378  Loss: 1.008442
Iter: 379  Loss: 0.918789
Iter: 380  Loss: 0.857720
Iter: 381  Loss: 0.816111
Iter: 382  Loss: 0.815630
Blurify
Iter: 383  Loss: 1.559670
Iter: 384  Loss: 1.143955
Iter: 385  Loss: 0.982518
Iter: 386  Loss: 0.883551
Iter: 387  Loss: 0.831147
Iter: 388  Loss: 0.794361
Iter: 389  Loss: 0.793890
Blurify
Iter: 390  Loss: 1.591282
Iter: 391  Loss: 1.101236
Iter: 392  Loss: 0.941607
Iter: 393  Loss: 0.860540
Iter: 394  Loss: 0.806040
Iter: 395  Loss: 0.782340
Iter: 396  Loss: 0.782357
Blurify
Iter: 397  Loss: 1.572863
Iter: 398  Loss: 1.097611
Iter: 399  Loss: 0.942797
Iter: 400  Loss: 0.916586
Iter: 401  Loss: 0.916587
Blurify
Iter: 402  Loss: 1.704975
Iter: 403  Loss: 1.155810
Iter: 404  Loss: 0.989672
Iter: 405  Loss: 0.884353
Iter: 406  Loss: 0.823370
Iter: 407  Loss: 0.822778
Blurify
Iter: 408  Loss: 1.560934
Iter: 409  Loss: 1.083441
Iter: 410  Loss: 0.930265
Iter: 411  Loss: 0.888969
Iter: 412  Loss: 0.888973
Blurify
Iter: 413  Loss: 1.668294
Iter: 414  Loss: 1.126786
Iter: 415  Loss: 0.959848
Iter: 416  Loss: 0.870241
Iter: 417  Loss: 0.811356
Iter: 418  Loss: 0.810739
Blurify
Iter: 419  Loss: 1.556719
Iter: 420  Loss: 1.087693
Iter: 421  Loss: 0.928446
Iter: 422  Loss: 0.845352
Iter: 423  Loss: 0.844636
Blurify
Iter: 424  Loss: 1.636164
Iter: 425  Loss: 1.108777
Iter: 426  Loss: 0.949618
Iter: 427  Loss: 0.858305
Iter: 428  Loss: 0.823756
Iter: 429  Loss: 0.823761
Blurify
Iter: 430  Loss: 1.577144
Iter: 431  Loss: 1.083413
Iter: 432  Loss: 0.922992
Iter: 433  Loss: 0.841505
Iter: 434  Loss: 0.840849
Blurify
Iter: 435  Loss: 1.572929
Iter: 436  Loss: 1.088615
Iter: 437  Loss: 0.937286
Iter: 438  Loss: 0.847529
Iter: 439  Loss: 0.791021
Iter: 440  Loss: 0.791058
Blurify
Iter: 441  Loss: 1.589831
Iter: 442  Loss: 1.089297
Iter: 443  Loss: 0.935899
Iter: 444  Loss: 0.834762
Iter: 445  Loss: 0.778591
Iter: 446  Loss: 0.778017
Blurify
Iter: 447  Loss: 1.570980
Iter: 448  Loss: 1.060264
Iter: 449  Loss: 0.908834
Iter: 450  Loss: 0.827388
Iter: 451  Loss: 0.770876
Iter: 452  Loss: 0.733375
Iter: 453  Loss: 0.733403
Blurify
Iter: 454  Loss: 1.481859
Iter: 455  Loss: 1.012001
Iter: 456  Loss: 0.876981
Iter: 457  Loss: 0.795810
Iter: 458  Loss: 0.766046
Iter: 459  Loss: 0.766048
Blurify
Iter: 460  Loss: 1.611749
Iter: 461  Loss: 1.063050
Iter: 462  Loss: 0.885479
Iter: 463  Loss: 0.805289
Iter: 464  Loss: 0.773311
Iter: 465  Loss: 0.773319
Blurify
Iter: 466  Loss: 1.599124
Iter: 467  Loss: 1.071766
Iter: 468  Loss: 0.900724
Iter: 469  Loss: 0.810972
Iter: 470  Loss: 0.774977
Iter: 471  Loss: 0.774982
Blurify
Iter: 472  Loss: 1.527201
Iter: 473  Loss: 1.049990
Iter: 474  Loss: 0.897775
Iter: 475  Loss: 0.816652
Iter: 476  Loss: 0.756219
Iter: 477  Loss: 0.718972
Iter: 478  Loss: 0.718984
Blurify
Iter: 479  Loss: 1.526689
Iter: 480  Loss: 0.999512
Iter: 481  Loss: 0.851616
Iter: 482  Loss: 0.779564
Iter: 483  Loss: 0.763833
Iter: 484  Loss: 0.763834
Blurify
Iter: 485  Loss: 1.443695
Iter: 486  Loss: 1.029675
Iter: 487  Loss: 0.881382
Iter: 488  Loss: 0.796284
Iter: 489  Loss: 0.743688
Iter: 490  Loss: 0.706002
Iter: 491  Loss: 0.705372
Blurify
Iter: 492  Loss: 1.494984
Iter: 493  Loss: 1.031159
Iter: 494  Loss: 0.868246
Iter: 495  Loss: 0.788398
Iter: 496  Loss: 0.732744
Iter: 497  Loss: 0.693835
Iter: 498  Loss: 0.683280
Iter: 499  Loss: 0.683282
Blurify
Iter: 500  Loss: 1.410359
Iter: 501  Loss: 0.974614
Iter: 502  Loss: 0.836255
Iter: 503  Loss: 0.755400
Iter: 504  Loss: 0.708049
Iter: 505  Loss: 0.707528
Blurify
Iter: 506  Loss: 1.440402
Iter: 507  Loss: 0.976092
Iter: 508  Loss: 0.830280
Iter: 509  Loss: 0.756310
Iter: 510  Loss: 0.707636
Iter: 511  Loss: 0.706975
Blurify
Iter: 512  Loss: 1.437204
Iter: 513  Loss: 1.030268
Iter: 514  Loss: 0.842084
Iter: 515  Loss: 0.759858
Iter: 516  Loss: 0.710317
Iter: 517  Loss: 0.694512
Iter: 518  Loss: 0.694514
Blurify
Iter: 519  Loss: 1.462489
Iter: 520  Loss: 0.982899
Iter: 521  Loss: 0.834143
Iter: 522  Loss: 0.759565
Iter: 523  Loss: 0.707675
Iter: 524  Loss: 0.671632
Iter: 525  Loss: 0.671126
Blurify
Iter: 526  Loss: 1.438211
Iter: 527  Loss: 1.002239
Iter: 528  Loss: 0.841731
Iter: 529  Loss: 0.759153
Iter: 530  Loss: 0.707697
Iter: 531  Loss: 0.696573
Iter: 532  Loss: 0.696574
Blurify
Iter: 533  Loss: 1.455075
Iter: 534  Loss: 0.972083
Iter: 535  Loss: 0.822862
Iter: 536  Loss: 0.745846
Iter: 537  Loss: 0.734977
Iter: 538  Loss: 0.734977
Blurify
Iter: 539  Loss: 1.492445
Iter: 540  Loss: 1.001542
Iter: 541  Loss: 0.837991
Iter: 542  Loss: 0.757749
Iter: 543  Loss: 0.703516
Iter: 544  Loss: 0.668864
Iter: 545  Loss: 0.668878
Blurify
Iter: 546  Loss: 1.475490
Iter: 547  Loss: 0.962056
Iter: 548  Loss: 0.813602
Iter: 549  Loss: 0.738938
Iter: 550  Loss: 0.729097
Iter: 551  Loss: 0.729098
Blurify
Iter: 552  Loss: 1.500962
Iter: 553  Loss: 0.995407
Iter: 554  Loss: 0.834013
Iter: 555  Loss: 0.754452
Iter: 556  Loss: 0.700929
Iter: 557  Loss: 0.664672
Iter: 558  Loss: 0.649790
Iter: 559  Loss: 0.649792
Blurify
Iter: 560  Loss: 1.412122
Iter: 561  Loss: 0.938912
Iter: 562  Loss: 0.802258
Iter: 563  Loss: 0.728709
Iter: 564  Loss: 0.681848
Iter: 565  Loss: 0.681246
Blurify
Iter: 566  Loss: 1.471962
Iter: 567  Loss: 0.970502
Iter: 568  Loss: 0.818586
Iter: 569  Loss: 0.739780
Iter: 570  Loss: 0.691805
Iter: 571  Loss: 0.680846
Iter: 572  Loss: 0.680849
Blurify
Iter: 573  Loss: 1.448032
Iter: 574  Loss: 0.969330
Iter: 575  Loss: 0.817864
Iter: 576  Loss: 0.734741
Iter: 577  Loss: 0.685624
Iter: 578  Loss: 0.666480
Iter: 579  Loss: 0.666484
Blurify
Iter: 580  Loss: 1.482483
Iter: 581  Loss: 0.950285
Iter: 582  Loss: 0.805447
Iter: 583  Loss: 0.724411
Iter: 584  Loss: 0.674971
Iter: 585  Loss: 0.674396
Blurify
Iter: 586  Loss: 1.478230
Iter: 587  Loss: 1.003268
Iter: 588  Loss: 0.831991
Iter: 589  Loss: 0.740734
Iter: 590  Loss: 0.699794
Iter: 591  Loss: 0.699800
Blurify
Iter: 592  Loss: 1.493797
Iter: 593  Loss: 1.002069
Iter: 594  Loss: 0.822957
Iter: 595  Loss: 0.735454
Iter: 596  Loss: 0.682964
Iter: 597  Loss: 0.682369
Blurify
Iter: 598  Loss: 1.444898
Iter: 599  Loss: 0.946164
Iter: 600  Loss: 0.793544
Iter: 601  Loss: 0.721501
Iter: 602  Loss: 0.720938
Blurify
Iter: 603  Loss: 1.483903
Iter: 604  Loss: 1.006200
Iter: 605  Loss: 0.826320
Iter: 606  Loss: 0.733916
Iter: 607  Loss: 0.679402
Iter: 608  Loss: 0.638108
Iter: 609  Loss: 0.613787
Iter: 610  Loss: 0.613791
Blurify
Iter: 611  Loss: 1.400314
Iter: 612  Loss: 0.923948
Iter: 613  Loss: 0.789037
Iter: 614  Loss: 0.704250
Iter: 615  Loss: 0.686687
Iter: 616  Loss: 0.686689
Blurify
Iter: 617  Loss: 1.444268
Iter: 618  Loss: 0.950646
Iter: 619  Loss: 0.800328
Iter: 620  Loss: 0.715556
Iter: 621  Loss: 0.663274
Iter: 622  Loss: 0.632915
Iter: 623  Loss: 0.632925
Blurify
Iter: 624  Loss: 1.343709
Iter: 625  Loss: 0.915177
Iter: 626  Loss: 0.778131
Iter: 627  Loss: 0.701385
Iter: 628  Loss: 0.651819
Iter: 629  Loss: 0.651065
Blurify
Iter: 630  Loss: 1.426860
Iter: 631  Loss: 0.935720
Iter: 632  Loss: 0.785331
Iter: 633  Loss: 0.707588
Iter: 634  Loss: 0.656999
Iter: 635  Loss: 0.619579
Iter: 636  Loss: 0.619055
Blurify
Iter: 637  Loss: 1.354297
Iter: 638  Loss: 0.926412
Iter: 639  Loss: 0.776874
Iter: 640  Loss: 0.750729
Iter: 641  Loss: 0.750729
Blurify
Iter: 642  Loss: 1.570702
Iter: 643  Loss: 1.019190
Iter: 644  Loss: 0.822733
Iter: 645  Loss: 0.735962
Iter: 646  Loss: 0.675927
Iter: 647  Loss: 0.659721
Iter: 648  Loss: 0.659724
Blurify
Iter: 649  Loss: 1.390557
Iter: 650  Loss: 0.937672
Iter: 651  Loss: 0.788466
Iter: 652  Loss: 0.714315
Iter: 653  Loss: 0.713716
Blurify
Iter: 654  Loss: 1.510025
Iter: 655  Loss: 0.984095
Iter: 656  Loss: 0.802979
Iter: 657  Loss: 0.717856
Iter: 658  Loss: 0.698400
Iter: 659  Loss: 0.698399
Blurify
Iter: 660  Loss: 1.451519
Iter: 661  Loss: 0.952145
Iter: 662  Loss: 0.804511
Iter: 663  Loss: 0.723896
Iter: 664  Loss: 0.671170
Iter: 665  Loss: 0.631325
Iter: 666  Loss: 0.619053
Iter: 667  Loss: 0.619054
Blurify
Iter: 668  Loss: 1.322697
Iter: 669  Loss: 0.909290
Iter: 670  Loss: 0.764783
Iter: 671  Loss: 0.692253
Iter: 672  Loss: 0.672955
Iter: 673  Loss: 0.672958
Blurify
Iter: 674  Loss: 1.435284
Iter: 675  Loss: 0.949033
Iter: 676  Loss: 0.798236
Iter: 677  Loss: 0.714178
Iter: 678  Loss: 0.661407
Iter: 679  Loss: 0.622092
Iter: 680  Loss: 0.621521
Blurify
Iter: 681  Loss: 1.314919
Iter: 682  Loss: 0.896356
Iter: 683  Loss: 0.758935
Iter: 684  Loss: 0.681273
Iter: 685  Loss: 0.644335
Iter: 686  Loss: 0.644342
Blurify
Iter: 687  Loss: 1.393354
Iter: 688  Loss: 0.927930
Iter: 689  Loss: 0.771988
Iter: 690  Loss: 0.695213
Iter: 691  Loss: 0.645475
Iter: 692  Loss: 0.609691
Iter: 693  Loss: 0.609142
Blurify
Iter: 694  Loss: 1.322874
Iter: 695  Loss: 0.899253
Iter: 696  Loss: 0.763395
Iter: 697  Loss: 0.685118
Iter: 698  Loss: 0.670940
Iter: 699  Loss: 0.670941
Blurify
Iter: 700  Loss: 1.412105
Iter: 701  Loss: 0.935407
Iter: 702  Loss: 0.780348
Iter: 703  Loss: 0.698621
Iter: 704  Loss: 0.686184
Iter: 705  Loss: 0.686186
Blurify
Iter: 706  Loss: 1.413243
Iter: 707  Loss: 0.962220
Iter: 708  Loss: 0.803778
Iter: 709  Loss: 0.718136
Iter: 710  Loss: 0.661469
Iter: 711  Loss: 0.624687
Iter: 712  Loss: 0.624707
Blurify
Iter: 713  Loss: 1.317298
Iter: 714  Loss: 0.905004
Iter: 715  Loss: 0.772311
Iter: 716  Loss: 0.684468
Iter: 717  Loss: 0.633271
Iter: 718  Loss: 0.596623
Iter: 719  Loss: 0.583212
Iter: 720  Loss: 0.583214
Blurify
Iter: 721  Loss: 1.270234
Iter: 722  Loss: 0.874606
Iter: 723  Loss: 0.734107
Iter: 724  Loss: 0.666439
Iter: 725  Loss: 0.617352
Iter: 726  Loss: 0.607012
Iter: 727  Loss: 0.607015
Blurify
Iter: 728  Loss: 1.361198
Iter: 729  Loss: 0.868904
Iter: 730  Loss: 0.728986
Iter: 731  Loss: 0.707686
Iter: 732  Loss: 0.707688
Blurify
Iter: 733  Loss: 1.466394
Iter: 734  Loss: 0.954970
Iter: 735  Loss: 0.789176
Iter: 736  Loss: 0.699792
Iter: 737  Loss: 0.644412
Iter: 738  Loss: 0.605118
Iter: 739  Loss: 0.604601
Blurify
Iter: 740  Loss: 1.322494
Iter: 741  Loss: 0.876010
Iter: 742  Loss: 0.743885
Iter: 743  Loss: 0.672234
Iter: 744  Loss: 0.626391
Iter: 745  Loss: 0.602576
Iter: 746  Loss: 0.602582
Blurify
Iter: 747  Loss: 1.357116
Iter: 748  Loss: 0.896633
Iter: 749  Loss: 0.740789
Iter: 750  Loss: 0.667143
Iter: 751  Loss: 0.620064
Iter: 752  Loss: 0.586258
Iter: 753  Loss: 0.585718
Blurify
Iter: 754  Loss: 1.311562
Iter: 755  Loss: 0.885459
Iter: 756  Loss: 0.733701
Iter: 757  Loss: 0.663475
Iter: 758  Loss: 0.631406
Iter: 759  Loss: 0.631417
Blurify
Iter: 760  Loss: 1.299721
Iter: 761  Loss: 0.892282
Iter: 762  Loss: 0.742484
Iter: 763  Loss: 0.666356
Iter: 764  Loss: 0.617527
Iter: 765  Loss: 0.583228
Iter: 766  Loss: 0.581389
Iter: 767  Loss: 0.581389
Blurify
Iter: 768  Loss: 1.339141
Iter: 769  Loss: 0.890005
Iter: 770  Loss: 0.739947
Iter: 771  Loss: 0.666516
Iter: 772  Loss: 0.618714
Iter: 773  Loss: 0.583107
Iter: 774  Loss: 0.582618
Blurify
Iter: 775  Loss: 1.297240
Iter: 776  Loss: 0.868372
Iter: 777  Loss: 0.730999
Iter: 778  Loss: 0.662781
Iter: 779  Loss: 0.615521
Iter: 780  Loss: 0.608122
Iter: 781  Loss: 0.608123
Blurify
Iter: 782  Loss: 1.324782
Iter: 783  Loss: 0.885691
Iter: 784  Loss: 0.755685
Iter: 785  Loss: 0.681464
Iter: 786  Loss: 0.629521
Iter: 787  Loss: 0.590525
Iter: 788  Loss: 0.586801
Iter: 789  Loss: 0.586801
Blurify
Iter: 790  Loss: 1.400053
Iter: 791  Loss: 0.912270
Iter: 792  Loss: 0.752839
Iter: 793  Loss: 0.675338
Iter: 794  Loss: 0.624025
Iter: 795  Loss: 0.603171
Iter: 796  Loss: 0.603175
Blurify
Iter: 797  Loss: 1.360415
Iter: 798  Loss: 0.914216
Iter: 799  Loss: 0.773592
Iter: 800  Loss: 0.682914
Iter: 801  Loss: 0.628781
Iter: 802  Loss: 0.619597
Iter: 803  Loss: 0.619599
Blurify
Iter: 804  Loss: 1.344754
Iter: 805  Loss: 0.877717
Iter: 806  Loss: 0.738761
Iter: 807  Loss: 0.668881
Iter: 808  Loss: 0.629468
Iter: 809  Loss: 0.629487
Blurify
Iter: 810  Loss: 1.391941
Iter: 811  Loss: 0.902576
Iter: 812  Loss: 0.752333
Iter: 813  Loss: 0.674536
Iter: 814  Loss: 0.673766
Blurify
Iter: 815  Loss: 1.333019
Iter: 816  Loss: 0.903175
Iter: 817  Loss: 0.765250
Iter: 818  Loss: 0.691223
Iter: 819  Loss: 0.630375
Iter: 820  Loss: 0.591675
Iter: 821  Loss: 0.583066
Iter: 822  Loss: 0.583067
Blurify
Iter: 823  Loss: 1.366395
Iter: 824  Loss: 0.884498
Iter: 825  Loss: 0.734912
Iter: 826  Loss: 0.663974
Iter: 827  Loss: 0.615942
Iter: 828  Loss: 0.606170
Iter: 829  Loss: 0.606172
Blurify
Iter: 830  Loss: 1.296562
Iter: 831  Loss: 0.864799
Iter: 832  Loss: 0.739163
Iter: 833  Loss: 0.665583
Iter: 834  Loss: 0.616152
Iter: 835  Loss: 0.598536
Iter: 836  Loss: 0.598542
Blurify
Iter: 837  Loss: 1.358876
Iter: 838  Loss: 0.915143
Iter: 839  Loss: 0.760596
Iter: 840  Loss: 0.679063
Iter: 841  Loss: 0.622877
Iter: 842  Loss: 0.603115
Iter: 843  Loss: 0.603119
Blurify
Iter: 844  Loss: 1.368879
Iter: 845  Loss: 0.894500
Iter: 846  Loss: 0.742970
Iter: 847  Loss: 0.666316
Iter: 848  Loss: 0.646866
Iter: 849  Loss: 0.646870
Blurify
Iter: 850  Loss: 1.348483
Iter: 851  Loss: 0.922918
Iter: 852  Loss: 0.762991
Iter: 853  Loss: 0.679894
Iter: 854  Loss: 0.629478
Iter: 855  Loss: 0.608967
Iter: 856  Loss: 0.608971
Blurify
Iter: 857  Loss: 1.338495
Iter: 858  Loss: 0.903690
Iter: 859  Loss: 0.754525
Iter: 860  Loss: 0.674765
Iter: 861  Loss: 0.622137
Iter: 862  Loss: 0.586825
Iter: 863  Loss: 0.584948
Iter: 864  Loss: 0.584948
Blurify
Iter: 865  Loss: 1.331987
Iter: 866  Loss: 0.877920
Iter: 867  Loss: 0.744623
Iter: 868  Loss: 0.664072
Iter: 869  Loss: 0.615631
Iter: 870  Loss: 0.615031
Blurify
Iter: 871  Loss: 1.334820
Iter: 872  Loss: 0.916680
Iter: 873  Loss: 0.779404
Iter: 874  Loss: 0.684045
Iter: 875  Loss: 0.624200
Iter: 876  Loss: 0.584569
Iter: 877  Loss: 0.583939
Blurify
Iter: 878  Loss: 1.299375
Iter: 879  Loss: 0.877454
Iter: 880  Loss: 0.747997
Iter: 881  Loss: 0.665398
Iter: 882  Loss: 0.634166
Iter: 883  Loss: 0.634169
Blurify
Iter: 884  Loss: 1.390724
Iter: 885  Loss: 0.887778
Iter: 886  Loss: 0.742429
Iter: 887  Loss: 0.665075
Iter: 888  Loss: 0.613890
Iter: 889  Loss: 0.602208
Iter: 890  Loss: 0.602209
Blurify
Iter: 891  Loss: 1.327836
Iter: 892  Loss: 0.931290
Iter: 893  Loss: 0.748469
Iter: 894  Loss: 0.661728
Iter: 895  Loss: 0.614207
Iter: 896  Loss: 0.572118
Iter: 897  Loss: 0.546246
Iter: 898  Loss: 0.546252
Blurify
Iter: 899  Loss: 1.388196
Iter: 900  Loss: 0.888692
Iter: 901  Loss: 0.723575
Iter: 902  Loss: 0.652392
Iter: 903  Loss: 0.594746
Iter: 904  Loss: 0.557119
Iter: 905  Loss: 0.556605
Blurify
Iter: 906  Loss: 1.338814
Iter: 907  Loss: 0.897129
Iter: 908  Loss: 0.748425
Iter: 909  Loss: 0.658480
Iter: 910  Loss: 0.607440
Iter: 911  Loss: 0.579476
Iter: 912  Loss: 0.579483
Blurify
Iter: 913  Loss: 1.310315
Iter: 914  Loss: 0.866654
Iter: 915  Loss: 0.723339
Iter: 916  Loss: 0.645515
Iter: 917  Loss: 0.605835
Iter: 918  Loss: 0.605848
Blurify
Iter: 919  Loss: 1.338557
Iter: 920  Loss: 0.884216
Iter: 921  Loss: 0.729727
Iter: 922  Loss: 0.658603
Iter: 923  Loss: 0.607693
Iter: 924  Loss: 0.573069
Iter: 925  Loss: 0.573089
Blurify
Iter: 926  Loss: 1.240992
Iter: 927  Loss: 0.836057
Iter: 928  Loss: 0.706836
Iter: 929  Loss: 0.637109
Iter: 930  Loss: 0.588644
Iter: 931  Loss: 0.554079
Iter: 932  Loss: 0.553561
Blurify
Iter: 933  Loss: 1.300593
Iter: 934  Loss: 0.860603
Iter: 935  Loss: 0.724729
Iter: 936  Loss: 0.647422
Iter: 937  Loss: 0.601770
Iter: 938  Loss: 0.567954
Iter: 939  Loss: 0.556640
Iter: 940  Loss: 0.556641
Blurify
Iter: 941  Loss: 1.242631
Iter: 942  Loss: 0.874079
Iter: 943  Loss: 0.739234
Iter: 944  Loss: 0.650430
Iter: 945  Loss: 0.595259
Iter: 946  Loss: 0.594527
Blurify
Iter: 947  Loss: 1.366264
Iter: 948  Loss: 0.895386
Iter: 949  Loss: 0.760103
Iter: 950  Loss: 0.665730
Iter: 951  Loss: 0.612236
Iter: 952  Loss: 0.579394
Iter: 953  Loss: 0.579410
Blurify
Iter: 954  Loss: 1.284133
Iter: 955  Loss: 0.853311
Iter: 956  Loss: 0.717772
Iter: 957  Loss: 0.646120
Iter: 958  Loss: 0.623659
Iter: 959  Loss: 0.623661
Blurify
Iter: 960  Loss: 1.399782
Iter: 961  Loss: 0.895885
Iter: 962  Loss: 0.729888
Iter: 963  Loss: 0.648027
Iter: 964  Loss: 0.632871
Iter: 965  Loss: 0.632872
Blurify
Iter: 966  Loss: 1.339839
Iter: 967  Loss: 0.892210
Iter: 968  Loss: 0.745026
Iter: 969  Loss: 0.666165
Iter: 970  Loss: 0.603911
Iter: 971  Loss: 0.563752
Iter: 972  Loss: 0.563282
Blurify
Iter: 973  Loss: 1.283730
Iter: 974  Loss: 0.858825
Iter: 975  Loss: 0.730703
Iter: 976  Loss: 0.656177
Iter: 977  Loss: 0.604913
Iter: 978  Loss: 0.561716
Iter: 979  Loss: 0.535116
Iter: 980  Loss: 0.535120
Blurify
Iter: 981  Loss: 1.226458
Iter: 982  Loss: 0.827190
Iter: 983  Loss: 0.685638
Iter: 984  Loss: 0.616972
Iter: 985  Loss: 0.570298
Iter: 986  Loss: 0.569601
Blurify
Iter: 987  Loss: 1.289554
Iter: 988  Loss: 0.876034
Iter: 989  Loss: 0.734361
Iter: 990  Loss: 0.645794
Iter: 991  Loss: 0.584484
Iter: 992  Loss: 0.541989
Iter: 993  Loss: 0.529610
Iter: 994  Loss: 0.529612
Blurify
Iter: 995  Loss: 1.199055
Iter: 996  Loss: 0.801824
Iter: 997  Loss: 0.675154
Iter: 998  Loss: 0.604573
Iter: 999  Loss: 0.557178

As it turns out, the fmin_l_bfgs_b optimizer used in the original gets stuck in a local minimum in approximately 13-15 iterations, with a loss of 3.19. It does not seem to move from there regardless of how many more iterations it is run for.

This is what the image looks like at that point:

In [17]:
Image.open('/home/ubuntu/results/res_at_iteration_14.png')
Out[17]:

If the image is blurred whenever the loss stops improving, after 994 iterations the loss reaches 0.53. The image looks like this:

In [15]:
Image.open('/home/ubuntu/results/res_at_iteration_994.png')
Out[15]:

The original image, for comparison:

In [16]:
img
Out[16]: