This notebook is based on neural-style.ipynb plus one key observation: the L-BFGS-B optimization algorithm used always gets stuck in local minima, and the local minima seems to always have a characteristic pattern of artifacts with a high spatial frequency in the generated image. Can we suppress those artifacts by simple blurring?
%matplotlib inline
import importlib
import utils2; importlib.reload(utils2)
from utils2 import *
from scipy.optimize import fmin_l_bfgs_b
from scipy.misc import imsave
from keras import metrics
from vgg16_avg import VGG16_Avg
Using TensorFlow backend. /home/ubuntu/anaconda3/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20. "This module will be removed in 0.20.", DeprecationWarning)
limit_mem()
img = Image.open('/home/ubuntu/www.platform.ai/data/imagenet-sample-train/train/n01558993/n01558993_9684.JPEG'); img
rn_mean = np.array([123.68, 116.779, 103.939], dtype=np.float32)
preproc = lambda x: (x - rn_mean)[:, :, :, ::-1]
img_arr = preproc(np.expand_dims(np.array(img), 0))
shp = img_arr.shape
deproc = lambda x,s: np.clip(x.reshape(s)[:, :, :, ::-1] + rn_mean, 0, 255)
model = VGG16_Avg(include_top=False)
layer = model.get_layer('block5_conv1').output
layer_model = Model(model.input, layer)
targ = K.variable(layer_model.predict(img_arr))
class Evaluator(object):
def __init__(self, f, shp): self.f, self.shp = f, shp
def loss(self, x):
loss_, self.grad_values = self.f([x.reshape(self.shp)])
return loss_.astype(np.float64)
def grads(self, x): return self.grad_values.flatten().astype(np.float64)
loss = metrics.mse(layer, targ)
grads = K.gradients(loss, model.input)
fn = K.function([model.input], [loss]+grads)
evaluator = Evaluator(fn, shp)
Let's see if it's possible to recreate the input image from the block5_conv1 activations, but with a twist: blur the image whenever the optimizer gets stuck.
First, a function that blurs an image in-place (channel by channel) using simple gaussian blur with sigma = 1
import scipy.ndimage.filters
def blurify(x):
x = x.reshape(shp)
for k in range(3):
x[:,:,:,k] = scipy.ndimage.filters.gaussian_filter(x[:,:,:,k], 1)
return x
Then, modify solve_image to detect when a local minimum has been reached (very small change per step < 0.001), and blur the image then continue. This may need to be tuned, it is possible that it detects local minima somewhat too early (could try change per step < 0.0001 instead).
def solve_image(eval_obj, niter, x):
last_min_val = 1000 # Start from some arbitrary very high number # TODO init with None
loss_history = {}
for i in range(niter):
x, min_val, info = fmin_l_bfgs_b(eval_obj.loss, x.flatten(),
fprime=eval_obj.grads, maxfun=20)
x = np.clip(x, -127,127)
print('Iter: %d Loss: %f' %(i, min_val))
if abs(last_min_val - min_val) < 0.001:
x = blurify(x)
print('Blurify')
last_min_val = min_val
loss_history[i] = min_val
imsave('/home/ubuntu/results/res_at_iteration_%d.png' %(i), (deproc(x.copy(), shp)[0]))
return x, loss_history
Start from all zeros. Noise vs all zeros doesn't seem to make any difference in practice...
rand_img = lambda shape: np.random.uniform(-2.5, 2.5, shape)/100
# x = rand_img(shp)
x = np.zeros(shp)
plt.imshow(x[0]);
Run for a ridiculous number of iterations
iterations=1000
x, loss_history = solve_image(evaluator, iterations, x)
Iter: 0 Loss: 56.332458 Iter: 1 Loss: 21.307999 Iter: 2 Loss: 11.739264 Iter: 3 Loss: 8.211292 Iter: 4 Loss: 6.507470 Iter: 5 Loss: 5.483242 Iter: 6 Loss: 4.860817 Iter: 7 Loss: 4.388783 Iter: 8 Loss: 3.996233 Iter: 9 Loss: 3.699281 Iter: 10 Loss: 3.469527 Iter: 11 Loss: 3.264268 Iter: 12 Loss: 3.202618 Iter: 13 Loss: 3.190215 Iter: 14 Loss: 3.190377 Blurify Iter: 15 Loss: 5.273498 Iter: 16 Loss: 3.830165 Iter: 17 Loss: 3.301814 Iter: 18 Loss: 2.958852 Iter: 19 Loss: 2.727372 Iter: 20 Loss: 2.563292 Iter: 21 Loss: 2.436050 Iter: 22 Loss: 2.337002 Iter: 23 Loss: 2.315290 Iter: 24 Loss: 2.316147 Blurify Iter: 25 Loss: 3.951824 Iter: 26 Loss: 2.967965 Iter: 27 Loss: 2.604438 Iter: 28 Loss: 2.388319 Iter: 29 Loss: 2.237290 Iter: 30 Loss: 2.131394 Iter: 31 Loss: 2.109755 Iter: 32 Loss: 2.108187 Iter: 33 Loss: 2.108195 Blurify Iter: 34 Loss: 3.451382 Iter: 35 Loss: 2.599714 Iter: 36 Loss: 2.289659 Iter: 37 Loss: 2.129872 Iter: 38 Loss: 2.020915 Iter: 39 Loss: 2.001437 Iter: 40 Loss: 2.001369 Blurify Iter: 41 Loss: 3.246094 Iter: 42 Loss: 2.457810 Iter: 43 Loss: 2.199605 Iter: 44 Loss: 2.031650 Iter: 45 Loss: 1.950836 Iter: 46 Loss: 1.951602 Blurify Iter: 47 Loss: 3.095697 Iter: 48 Loss: 2.364367 Iter: 49 Loss: 2.109860 Iter: 50 Loss: 1.964509 Iter: 51 Loss: 1.935248 Iter: 52 Loss: 1.935365 Blurify Iter: 53 Loss: 2.983280 Iter: 54 Loss: 2.290123 Iter: 55 Loss: 2.053647 Iter: 56 Loss: 1.899210 Iter: 57 Loss: 1.874540 Iter: 58 Loss: 1.874610 Blurify Iter: 59 Loss: 2.853730 Iter: 60 Loss: 2.220968 Iter: 61 Loss: 2.010096 Iter: 62 Loss: 1.853112 Iter: 63 Loss: 1.801871 Iter: 64 Loss: 1.802014 Blurify Iter: 65 Loss: 2.734373 Iter: 66 Loss: 2.126369 Iter: 67 Loss: 1.905844 Iter: 68 Loss: 1.902967 Iter: 69 Loss: 1.902974 Blurify Iter: 70 Loss: 2.715861 Iter: 71 Loss: 2.146672 Iter: 72 Loss: 1.923933 Iter: 73 Loss: 1.792043 Iter: 74 Loss: 1.789842 Iter: 75 Loss: 1.789849 Blurify Iter: 76 Loss: 2.611001 Iter: 77 Loss: 2.051578 Iter: 78 Loss: 1.843392 Iter: 79 Loss: 1.729595 Iter: 80 Loss: 1.706310 Iter: 81 Loss: 1.706330 Blurify Iter: 82 Loss: 2.605799 Iter: 83 Loss: 2.003242 Iter: 84 Loss: 1.791754 Iter: 85 Loss: 1.788433 Iter: 86 Loss: 1.788436 Blurify Iter: 87 Loss: 2.556896 Iter: 88 Loss: 1.965723 Iter: 89 Loss: 1.770293 Iter: 90 Loss: 1.712831 Iter: 91 Loss: 1.712929 Blurify Iter: 92 Loss: 2.514152 Iter: 93 Loss: 1.947668 Iter: 94 Loss: 1.755485 Iter: 95 Loss: 1.752536 Iter: 96 Loss: 1.752539 Blurify Iter: 97 Loss: 2.599280 Iter: 98 Loss: 1.963046 Iter: 99 Loss: 1.756613 Iter: 100 Loss: 1.631467 Iter: 101 Loss: 1.628476 Iter: 102 Loss: 1.628479 Blurify Iter: 103 Loss: 2.485286 Iter: 104 Loss: 1.907258 Iter: 105 Loss: 1.698251 Iter: 106 Loss: 1.694783 Iter: 107 Loss: 1.694786 Blurify Iter: 108 Loss: 2.429407 Iter: 109 Loss: 1.886185 Iter: 110 Loss: 1.691889 Iter: 111 Loss: 1.656286 Iter: 112 Loss: 1.656301 Blurify Iter: 113 Loss: 2.426243 Iter: 114 Loss: 1.861527 Iter: 115 Loss: 1.649160 Iter: 116 Loss: 1.530549 Iter: 117 Loss: 1.447638 Iter: 118 Loss: 1.377305 Iter: 119 Loss: 1.374920 Iter: 120 Loss: 1.374924 Blurify Iter: 121 Loss: 2.228023 Iter: 122 Loss: 1.703411 Iter: 123 Loss: 1.537968 Iter: 124 Loss: 1.457610 Iter: 125 Loss: 1.457708 Blurify Iter: 126 Loss: 2.265303 Iter: 127 Loss: 1.748284 Iter: 128 Loss: 1.544124 Iter: 129 Loss: 1.493667 Iter: 130 Loss: 1.493702 Blurify Iter: 131 Loss: 2.348393 Iter: 132 Loss: 1.742352 Iter: 133 Loss: 1.549081 Iter: 134 Loss: 1.431310 Iter: 135 Loss: 1.428494 Iter: 136 Loss: 1.428496 Blurify Iter: 137 Loss: 2.333052 Iter: 138 Loss: 1.710642 Iter: 139 Loss: 1.509532 Iter: 140 Loss: 1.505881 Iter: 141 Loss: 1.505883 Blurify Iter: 142 Loss: 2.297161 Iter: 143 Loss: 1.750185 Iter: 144 Loss: 1.548308 Iter: 145 Loss: 1.425497 Iter: 146 Loss: 1.422190 Iter: 147 Loss: 1.422191 Blurify Iter: 148 Loss: 2.197482 Iter: 149 Loss: 1.677197 Iter: 150 Loss: 1.483543 Iter: 151 Loss: 1.381848 Iter: 152 Loss: 1.370030 Iter: 153 Loss: 1.370035 Blurify Iter: 154 Loss: 2.191892 Iter: 155 Loss: 1.642742 Iter: 156 Loss: 1.462429 Iter: 157 Loss: 1.357848 Iter: 158 Loss: 1.282951 Iter: 159 Loss: 1.262653 Iter: 160 Loss: 1.262663 Blurify Iter: 161 Loss: 2.119348 Iter: 162 Loss: 1.571251 Iter: 163 Loss: 1.404496 Iter: 164 Loss: 1.307903 Iter: 165 Loss: 1.289759 Iter: 166 Loss: 1.289763 Blurify Iter: 167 Loss: 2.142907 Iter: 168 Loss: 1.616430 Iter: 169 Loss: 1.441315 Iter: 170 Loss: 1.333306 Iter: 171 Loss: 1.257399 Iter: 172 Loss: 1.199955 Iter: 173 Loss: 1.170250 Iter: 174 Loss: 1.170300 Blurify Iter: 175 Loss: 2.055926 Iter: 176 Loss: 1.497262 Iter: 177 Loss: 1.337545 Iter: 178 Loss: 1.248868 Iter: 179 Loss: 1.185790 Iter: 180 Loss: 1.138944 Iter: 181 Loss: 1.138542 Blurify Iter: 182 Loss: 2.026123 Iter: 183 Loss: 1.497175 Iter: 184 Loss: 1.319403 Iter: 185 Loss: 1.228595 Iter: 186 Loss: 1.157804 Iter: 187 Loss: 1.109451 Iter: 188 Loss: 1.109047 Blurify Iter: 189 Loss: 1.970667 Iter: 190 Loss: 1.461758 Iter: 191 Loss: 1.292134 Iter: 192 Loss: 1.200104 Iter: 193 Loss: 1.191285 Iter: 194 Loss: 1.191288 Blurify Iter: 195 Loss: 2.000707 Iter: 196 Loss: 1.452933 Iter: 197 Loss: 1.288172 Iter: 198 Loss: 1.194241 Iter: 199 Loss: 1.193621 Blurify Iter: 200 Loss: 2.036135 Iter: 201 Loss: 1.472216 Iter: 202 Loss: 1.311498 Iter: 203 Loss: 1.216014 Iter: 204 Loss: 1.180757 Iter: 205 Loss: 1.180771 Blurify Iter: 206 Loss: 2.095854 Iter: 207 Loss: 1.488455 Iter: 208 Loss: 1.311389 Iter: 209 Loss: 1.210376 Iter: 210 Loss: 1.145466 Iter: 211 Loss: 1.144973 Blurify Iter: 212 Loss: 1.943347 Iter: 213 Loss: 1.431346 Iter: 214 Loss: 1.277030 Iter: 215 Loss: 1.197426 Iter: 216 Loss: 1.197464 Blurify Iter: 217 Loss: 2.001596 Iter: 218 Loss: 1.462071 Iter: 219 Loss: 1.297743 Iter: 220 Loss: 1.207827 Iter: 221 Loss: 1.144922 Iter: 222 Loss: 1.092487 Iter: 223 Loss: 1.068815 Iter: 224 Loss: 1.068832 Blurify Iter: 225 Loss: 1.925748 Iter: 226 Loss: 1.390729 Iter: 227 Loss: 1.238013 Iter: 228 Loss: 1.155514 Iter: 229 Loss: 1.115181 Iter: 230 Loss: 1.115203 Blurify Iter: 231 Loss: 2.032277 Iter: 232 Loss: 1.455521 Iter: 233 Loss: 1.273718 Iter: 234 Loss: 1.178594 Iter: 235 Loss: 1.111957 Iter: 236 Loss: 1.058639 Iter: 237 Loss: 1.051636 Iter: 238 Loss: 1.051638 Blurify Iter: 239 Loss: 1.937854 Iter: 240 Loss: 1.377743 Iter: 241 Loss: 1.222426 Iter: 242 Loss: 1.129910 Iter: 243 Loss: 1.071089 Iter: 244 Loss: 1.046510 Iter: 245 Loss: 1.046524 Blurify Iter: 246 Loss: 1.951347 Iter: 247 Loss: 1.445107 Iter: 248 Loss: 1.229120 Iter: 249 Loss: 1.131885 Iter: 250 Loss: 1.070002 Iter: 251 Loss: 1.049373 Iter: 252 Loss: 1.049378 Blurify Iter: 253 Loss: 2.000892 Iter: 254 Loss: 1.442447 Iter: 255 Loss: 1.238814 Iter: 256 Loss: 1.126239 Iter: 257 Loss: 1.106446 Iter: 258 Loss: 1.106450 Blurify Iter: 259 Loss: 1.970386 Iter: 260 Loss: 1.411084 Iter: 261 Loss: 1.211497 Iter: 262 Loss: 1.114671 Iter: 263 Loss: 1.055484 Iter: 264 Loss: 1.054969 Blurify Iter: 265 Loss: 1.945976 Iter: 266 Loss: 1.404606 Iter: 267 Loss: 1.204569 Iter: 268 Loss: 1.112050 Iter: 269 Loss: 1.078671 Iter: 270 Loss: 1.078685 Blurify Iter: 271 Loss: 1.902259 Iter: 272 Loss: 1.372724 Iter: 273 Loss: 1.207352 Iter: 274 Loss: 1.096894 Iter: 275 Loss: 1.071516 Iter: 276 Loss: 1.071519 Blurify Iter: 277 Loss: 1.846668 Iter: 278 Loss: 1.327475 Iter: 279 Loss: 1.171771 Iter: 280 Loss: 1.121377 Iter: 281 Loss: 1.121382 Blurify Iter: 282 Loss: 1.943167 Iter: 283 Loss: 1.366761 Iter: 284 Loss: 1.179744 Iter: 285 Loss: 1.079134 Iter: 286 Loss: 1.010223 Iter: 287 Loss: 0.979561 Iter: 288 Loss: 0.979587 Blurify Iter: 289 Loss: 1.760459 Iter: 290 Loss: 1.305357 Iter: 291 Loss: 1.131122 Iter: 292 Loss: 1.036079 Iter: 293 Loss: 0.976252 Iter: 294 Loss: 0.956948 Iter: 295 Loss: 0.956960 Blurify Iter: 296 Loss: 1.809328 Iter: 297 Loss: 1.269864 Iter: 298 Loss: 1.097080 Iter: 299 Loss: 1.053950 Iter: 300 Loss: 1.053956 Blurify Iter: 301 Loss: 1.829281 Iter: 302 Loss: 1.293646 Iter: 303 Loss: 1.122176 Iter: 304 Loss: 1.029310 Iter: 305 Loss: 1.010332 Iter: 306 Loss: 1.010334 Blurify Iter: 307 Loss: 1.787094 Iter: 308 Loss: 1.287532 Iter: 309 Loss: 1.126492 Iter: 310 Loss: 1.034710 Iter: 311 Loss: 0.999540 Iter: 312 Loss: 0.999548 Blurify Iter: 313 Loss: 1.849936 Iter: 314 Loss: 1.279453 Iter: 315 Loss: 1.108685 Iter: 316 Loss: 1.014677 Iter: 317 Loss: 1.013999 Blurify Iter: 318 Loss: 1.859227 Iter: 319 Loss: 1.323013 Iter: 320 Loss: 1.137218 Iter: 321 Loss: 1.039899 Iter: 322 Loss: 0.960291 Iter: 323 Loss: 0.920928 Iter: 324 Loss: 0.920947 Blurify Iter: 325 Loss: 1.762552 Iter: 326 Loss: 1.231276 Iter: 327 Loss: 1.072776 Iter: 328 Loss: 0.985171 Iter: 329 Loss: 0.928061 Iter: 330 Loss: 0.927485 Blurify Iter: 331 Loss: 1.772559 Iter: 332 Loss: 1.221562 Iter: 333 Loss: 1.061660 Iter: 334 Loss: 0.973697 Iter: 335 Loss: 0.939623 Iter: 336 Loss: 0.939630 Blurify Iter: 337 Loss: 1.757710 Iter: 338 Loss: 1.246606 Iter: 339 Loss: 1.061812 Iter: 340 Loss: 0.969069 Iter: 341 Loss: 0.912672 Iter: 342 Loss: 0.885523 Iter: 343 Loss: 0.885543 Blurify Iter: 344 Loss: 1.700181 Iter: 345 Loss: 1.188488 Iter: 346 Loss: 1.031597 Iter: 347 Loss: 0.947435 Iter: 348 Loss: 0.924868 Iter: 349 Loss: 0.924874 Blurify Iter: 350 Loss: 1.739188 Iter: 351 Loss: 1.231966 Iter: 352 Loss: 1.065842 Iter: 353 Loss: 0.960356 Iter: 354 Loss: 0.932957 Iter: 355 Loss: 0.932958 Blurify Iter: 356 Loss: 1.673444 Iter: 357 Loss: 1.216911 Iter: 358 Loss: 1.053849 Iter: 359 Loss: 0.963652 Iter: 360 Loss: 0.903687 Iter: 361 Loss: 0.903160 Blurify Iter: 362 Loss: 1.778198 Iter: 363 Loss: 1.258198 Iter: 364 Loss: 1.064842 Iter: 365 Loss: 0.968148 Iter: 366 Loss: 0.900130 Iter: 367 Loss: 0.880332 Iter: 368 Loss: 0.880339 Blurify Iter: 369 Loss: 1.652490 Iter: 370 Loss: 1.191633 Iter: 371 Loss: 1.010065 Iter: 372 Loss: 0.931983 Iter: 373 Loss: 0.872074 Iter: 374 Loss: 0.848909 Iter: 375 Loss: 0.848920 Blurify Iter: 376 Loss: 1.637472 Iter: 377 Loss: 1.171467 Iter: 378 Loss: 1.008442 Iter: 379 Loss: 0.918789 Iter: 380 Loss: 0.857720 Iter: 381 Loss: 0.816111 Iter: 382 Loss: 0.815630 Blurify Iter: 383 Loss: 1.559670 Iter: 384 Loss: 1.143955 Iter: 385 Loss: 0.982518 Iter: 386 Loss: 0.883551 Iter: 387 Loss: 0.831147 Iter: 388 Loss: 0.794361 Iter: 389 Loss: 0.793890 Blurify Iter: 390 Loss: 1.591282 Iter: 391 Loss: 1.101236 Iter: 392 Loss: 0.941607 Iter: 393 Loss: 0.860540 Iter: 394 Loss: 0.806040 Iter: 395 Loss: 0.782340 Iter: 396 Loss: 0.782357 Blurify Iter: 397 Loss: 1.572863 Iter: 398 Loss: 1.097611 Iter: 399 Loss: 0.942797 Iter: 400 Loss: 0.916586 Iter: 401 Loss: 0.916587 Blurify Iter: 402 Loss: 1.704975 Iter: 403 Loss: 1.155810 Iter: 404 Loss: 0.989672 Iter: 405 Loss: 0.884353 Iter: 406 Loss: 0.823370 Iter: 407 Loss: 0.822778 Blurify Iter: 408 Loss: 1.560934 Iter: 409 Loss: 1.083441 Iter: 410 Loss: 0.930265 Iter: 411 Loss: 0.888969 Iter: 412 Loss: 0.888973 Blurify Iter: 413 Loss: 1.668294 Iter: 414 Loss: 1.126786 Iter: 415 Loss: 0.959848 Iter: 416 Loss: 0.870241 Iter: 417 Loss: 0.811356 Iter: 418 Loss: 0.810739 Blurify Iter: 419 Loss: 1.556719 Iter: 420 Loss: 1.087693 Iter: 421 Loss: 0.928446 Iter: 422 Loss: 0.845352 Iter: 423 Loss: 0.844636 Blurify Iter: 424 Loss: 1.636164 Iter: 425 Loss: 1.108777 Iter: 426 Loss: 0.949618 Iter: 427 Loss: 0.858305 Iter: 428 Loss: 0.823756 Iter: 429 Loss: 0.823761 Blurify Iter: 430 Loss: 1.577144 Iter: 431 Loss: 1.083413 Iter: 432 Loss: 0.922992 Iter: 433 Loss: 0.841505 Iter: 434 Loss: 0.840849 Blurify Iter: 435 Loss: 1.572929 Iter: 436 Loss: 1.088615 Iter: 437 Loss: 0.937286 Iter: 438 Loss: 0.847529 Iter: 439 Loss: 0.791021 Iter: 440 Loss: 0.791058 Blurify Iter: 441 Loss: 1.589831 Iter: 442 Loss: 1.089297 Iter: 443 Loss: 0.935899 Iter: 444 Loss: 0.834762 Iter: 445 Loss: 0.778591 Iter: 446 Loss: 0.778017 Blurify Iter: 447 Loss: 1.570980 Iter: 448 Loss: 1.060264 Iter: 449 Loss: 0.908834 Iter: 450 Loss: 0.827388 Iter: 451 Loss: 0.770876 Iter: 452 Loss: 0.733375 Iter: 453 Loss: 0.733403 Blurify Iter: 454 Loss: 1.481859 Iter: 455 Loss: 1.012001 Iter: 456 Loss: 0.876981 Iter: 457 Loss: 0.795810 Iter: 458 Loss: 0.766046 Iter: 459 Loss: 0.766048 Blurify Iter: 460 Loss: 1.611749 Iter: 461 Loss: 1.063050 Iter: 462 Loss: 0.885479 Iter: 463 Loss: 0.805289 Iter: 464 Loss: 0.773311 Iter: 465 Loss: 0.773319 Blurify Iter: 466 Loss: 1.599124 Iter: 467 Loss: 1.071766 Iter: 468 Loss: 0.900724 Iter: 469 Loss: 0.810972 Iter: 470 Loss: 0.774977 Iter: 471 Loss: 0.774982 Blurify Iter: 472 Loss: 1.527201 Iter: 473 Loss: 1.049990 Iter: 474 Loss: 0.897775 Iter: 475 Loss: 0.816652 Iter: 476 Loss: 0.756219 Iter: 477 Loss: 0.718972 Iter: 478 Loss: 0.718984 Blurify Iter: 479 Loss: 1.526689 Iter: 480 Loss: 0.999512 Iter: 481 Loss: 0.851616 Iter: 482 Loss: 0.779564 Iter: 483 Loss: 0.763833 Iter: 484 Loss: 0.763834 Blurify Iter: 485 Loss: 1.443695 Iter: 486 Loss: 1.029675 Iter: 487 Loss: 0.881382 Iter: 488 Loss: 0.796284 Iter: 489 Loss: 0.743688 Iter: 490 Loss: 0.706002 Iter: 491 Loss: 0.705372 Blurify Iter: 492 Loss: 1.494984 Iter: 493 Loss: 1.031159 Iter: 494 Loss: 0.868246 Iter: 495 Loss: 0.788398 Iter: 496 Loss: 0.732744 Iter: 497 Loss: 0.693835 Iter: 498 Loss: 0.683280 Iter: 499 Loss: 0.683282 Blurify Iter: 500 Loss: 1.410359 Iter: 501 Loss: 0.974614 Iter: 502 Loss: 0.836255 Iter: 503 Loss: 0.755400 Iter: 504 Loss: 0.708049 Iter: 505 Loss: 0.707528 Blurify Iter: 506 Loss: 1.440402 Iter: 507 Loss: 0.976092 Iter: 508 Loss: 0.830280 Iter: 509 Loss: 0.756310 Iter: 510 Loss: 0.707636 Iter: 511 Loss: 0.706975 Blurify Iter: 512 Loss: 1.437204 Iter: 513 Loss: 1.030268 Iter: 514 Loss: 0.842084 Iter: 515 Loss: 0.759858 Iter: 516 Loss: 0.710317 Iter: 517 Loss: 0.694512 Iter: 518 Loss: 0.694514 Blurify Iter: 519 Loss: 1.462489 Iter: 520 Loss: 0.982899 Iter: 521 Loss: 0.834143 Iter: 522 Loss: 0.759565 Iter: 523 Loss: 0.707675 Iter: 524 Loss: 0.671632 Iter: 525 Loss: 0.671126 Blurify Iter: 526 Loss: 1.438211 Iter: 527 Loss: 1.002239 Iter: 528 Loss: 0.841731 Iter: 529 Loss: 0.759153 Iter: 530 Loss: 0.707697 Iter: 531 Loss: 0.696573 Iter: 532 Loss: 0.696574 Blurify Iter: 533 Loss: 1.455075 Iter: 534 Loss: 0.972083 Iter: 535 Loss: 0.822862 Iter: 536 Loss: 0.745846 Iter: 537 Loss: 0.734977 Iter: 538 Loss: 0.734977 Blurify Iter: 539 Loss: 1.492445 Iter: 540 Loss: 1.001542 Iter: 541 Loss: 0.837991 Iter: 542 Loss: 0.757749 Iter: 543 Loss: 0.703516 Iter: 544 Loss: 0.668864 Iter: 545 Loss: 0.668878 Blurify Iter: 546 Loss: 1.475490 Iter: 547 Loss: 0.962056 Iter: 548 Loss: 0.813602 Iter: 549 Loss: 0.738938 Iter: 550 Loss: 0.729097 Iter: 551 Loss: 0.729098 Blurify Iter: 552 Loss: 1.500962 Iter: 553 Loss: 0.995407 Iter: 554 Loss: 0.834013 Iter: 555 Loss: 0.754452 Iter: 556 Loss: 0.700929 Iter: 557 Loss: 0.664672 Iter: 558 Loss: 0.649790 Iter: 559 Loss: 0.649792 Blurify Iter: 560 Loss: 1.412122 Iter: 561 Loss: 0.938912 Iter: 562 Loss: 0.802258 Iter: 563 Loss: 0.728709 Iter: 564 Loss: 0.681848 Iter: 565 Loss: 0.681246 Blurify Iter: 566 Loss: 1.471962 Iter: 567 Loss: 0.970502 Iter: 568 Loss: 0.818586 Iter: 569 Loss: 0.739780 Iter: 570 Loss: 0.691805 Iter: 571 Loss: 0.680846 Iter: 572 Loss: 0.680849 Blurify Iter: 573 Loss: 1.448032 Iter: 574 Loss: 0.969330 Iter: 575 Loss: 0.817864 Iter: 576 Loss: 0.734741 Iter: 577 Loss: 0.685624 Iter: 578 Loss: 0.666480 Iter: 579 Loss: 0.666484 Blurify Iter: 580 Loss: 1.482483 Iter: 581 Loss: 0.950285 Iter: 582 Loss: 0.805447 Iter: 583 Loss: 0.724411 Iter: 584 Loss: 0.674971 Iter: 585 Loss: 0.674396 Blurify Iter: 586 Loss: 1.478230 Iter: 587 Loss: 1.003268 Iter: 588 Loss: 0.831991 Iter: 589 Loss: 0.740734 Iter: 590 Loss: 0.699794 Iter: 591 Loss: 0.699800 Blurify Iter: 592 Loss: 1.493797 Iter: 593 Loss: 1.002069 Iter: 594 Loss: 0.822957 Iter: 595 Loss: 0.735454 Iter: 596 Loss: 0.682964 Iter: 597 Loss: 0.682369 Blurify Iter: 598 Loss: 1.444898 Iter: 599 Loss: 0.946164 Iter: 600 Loss: 0.793544 Iter: 601 Loss: 0.721501 Iter: 602 Loss: 0.720938 Blurify Iter: 603 Loss: 1.483903 Iter: 604 Loss: 1.006200 Iter: 605 Loss: 0.826320 Iter: 606 Loss: 0.733916 Iter: 607 Loss: 0.679402 Iter: 608 Loss: 0.638108 Iter: 609 Loss: 0.613787 Iter: 610 Loss: 0.613791 Blurify Iter: 611 Loss: 1.400314 Iter: 612 Loss: 0.923948 Iter: 613 Loss: 0.789037 Iter: 614 Loss: 0.704250 Iter: 615 Loss: 0.686687 Iter: 616 Loss: 0.686689 Blurify Iter: 617 Loss: 1.444268 Iter: 618 Loss: 0.950646 Iter: 619 Loss: 0.800328 Iter: 620 Loss: 0.715556 Iter: 621 Loss: 0.663274 Iter: 622 Loss: 0.632915 Iter: 623 Loss: 0.632925 Blurify Iter: 624 Loss: 1.343709 Iter: 625 Loss: 0.915177 Iter: 626 Loss: 0.778131 Iter: 627 Loss: 0.701385 Iter: 628 Loss: 0.651819 Iter: 629 Loss: 0.651065 Blurify Iter: 630 Loss: 1.426860 Iter: 631 Loss: 0.935720 Iter: 632 Loss: 0.785331 Iter: 633 Loss: 0.707588 Iter: 634 Loss: 0.656999 Iter: 635 Loss: 0.619579 Iter: 636 Loss: 0.619055 Blurify Iter: 637 Loss: 1.354297 Iter: 638 Loss: 0.926412 Iter: 639 Loss: 0.776874 Iter: 640 Loss: 0.750729 Iter: 641 Loss: 0.750729 Blurify Iter: 642 Loss: 1.570702 Iter: 643 Loss: 1.019190 Iter: 644 Loss: 0.822733 Iter: 645 Loss: 0.735962 Iter: 646 Loss: 0.675927 Iter: 647 Loss: 0.659721 Iter: 648 Loss: 0.659724 Blurify Iter: 649 Loss: 1.390557 Iter: 650 Loss: 0.937672 Iter: 651 Loss: 0.788466 Iter: 652 Loss: 0.714315 Iter: 653 Loss: 0.713716 Blurify Iter: 654 Loss: 1.510025 Iter: 655 Loss: 0.984095 Iter: 656 Loss: 0.802979 Iter: 657 Loss: 0.717856 Iter: 658 Loss: 0.698400 Iter: 659 Loss: 0.698399 Blurify Iter: 660 Loss: 1.451519 Iter: 661 Loss: 0.952145 Iter: 662 Loss: 0.804511 Iter: 663 Loss: 0.723896 Iter: 664 Loss: 0.671170 Iter: 665 Loss: 0.631325 Iter: 666 Loss: 0.619053 Iter: 667 Loss: 0.619054 Blurify Iter: 668 Loss: 1.322697 Iter: 669 Loss: 0.909290 Iter: 670 Loss: 0.764783 Iter: 671 Loss: 0.692253 Iter: 672 Loss: 0.672955 Iter: 673 Loss: 0.672958 Blurify Iter: 674 Loss: 1.435284 Iter: 675 Loss: 0.949033 Iter: 676 Loss: 0.798236 Iter: 677 Loss: 0.714178 Iter: 678 Loss: 0.661407 Iter: 679 Loss: 0.622092 Iter: 680 Loss: 0.621521 Blurify Iter: 681 Loss: 1.314919 Iter: 682 Loss: 0.896356 Iter: 683 Loss: 0.758935 Iter: 684 Loss: 0.681273 Iter: 685 Loss: 0.644335 Iter: 686 Loss: 0.644342 Blurify Iter: 687 Loss: 1.393354 Iter: 688 Loss: 0.927930 Iter: 689 Loss: 0.771988 Iter: 690 Loss: 0.695213 Iter: 691 Loss: 0.645475 Iter: 692 Loss: 0.609691 Iter: 693 Loss: 0.609142 Blurify Iter: 694 Loss: 1.322874 Iter: 695 Loss: 0.899253 Iter: 696 Loss: 0.763395 Iter: 697 Loss: 0.685118 Iter: 698 Loss: 0.670940 Iter: 699 Loss: 0.670941 Blurify Iter: 700 Loss: 1.412105 Iter: 701 Loss: 0.935407 Iter: 702 Loss: 0.780348 Iter: 703 Loss: 0.698621 Iter: 704 Loss: 0.686184 Iter: 705 Loss: 0.686186 Blurify Iter: 706 Loss: 1.413243 Iter: 707 Loss: 0.962220 Iter: 708 Loss: 0.803778 Iter: 709 Loss: 0.718136 Iter: 710 Loss: 0.661469 Iter: 711 Loss: 0.624687 Iter: 712 Loss: 0.624707 Blurify Iter: 713 Loss: 1.317298 Iter: 714 Loss: 0.905004 Iter: 715 Loss: 0.772311 Iter: 716 Loss: 0.684468 Iter: 717 Loss: 0.633271 Iter: 718 Loss: 0.596623 Iter: 719 Loss: 0.583212 Iter: 720 Loss: 0.583214 Blurify Iter: 721 Loss: 1.270234 Iter: 722 Loss: 0.874606 Iter: 723 Loss: 0.734107 Iter: 724 Loss: 0.666439 Iter: 725 Loss: 0.617352 Iter: 726 Loss: 0.607012 Iter: 727 Loss: 0.607015 Blurify Iter: 728 Loss: 1.361198 Iter: 729 Loss: 0.868904 Iter: 730 Loss: 0.728986 Iter: 731 Loss: 0.707686 Iter: 732 Loss: 0.707688 Blurify Iter: 733 Loss: 1.466394 Iter: 734 Loss: 0.954970 Iter: 735 Loss: 0.789176 Iter: 736 Loss: 0.699792 Iter: 737 Loss: 0.644412 Iter: 738 Loss: 0.605118 Iter: 739 Loss: 0.604601 Blurify Iter: 740 Loss: 1.322494 Iter: 741 Loss: 0.876010 Iter: 742 Loss: 0.743885 Iter: 743 Loss: 0.672234 Iter: 744 Loss: 0.626391 Iter: 745 Loss: 0.602576 Iter: 746 Loss: 0.602582 Blurify Iter: 747 Loss: 1.357116 Iter: 748 Loss: 0.896633 Iter: 749 Loss: 0.740789 Iter: 750 Loss: 0.667143 Iter: 751 Loss: 0.620064 Iter: 752 Loss: 0.586258 Iter: 753 Loss: 0.585718 Blurify Iter: 754 Loss: 1.311562 Iter: 755 Loss: 0.885459 Iter: 756 Loss: 0.733701 Iter: 757 Loss: 0.663475 Iter: 758 Loss: 0.631406 Iter: 759 Loss: 0.631417 Blurify Iter: 760 Loss: 1.299721 Iter: 761 Loss: 0.892282 Iter: 762 Loss: 0.742484 Iter: 763 Loss: 0.666356 Iter: 764 Loss: 0.617527 Iter: 765 Loss: 0.583228 Iter: 766 Loss: 0.581389 Iter: 767 Loss: 0.581389 Blurify Iter: 768 Loss: 1.339141 Iter: 769 Loss: 0.890005 Iter: 770 Loss: 0.739947 Iter: 771 Loss: 0.666516 Iter: 772 Loss: 0.618714 Iter: 773 Loss: 0.583107 Iter: 774 Loss: 0.582618 Blurify Iter: 775 Loss: 1.297240 Iter: 776 Loss: 0.868372 Iter: 777 Loss: 0.730999 Iter: 778 Loss: 0.662781 Iter: 779 Loss: 0.615521 Iter: 780 Loss: 0.608122 Iter: 781 Loss: 0.608123 Blurify Iter: 782 Loss: 1.324782 Iter: 783 Loss: 0.885691 Iter: 784 Loss: 0.755685 Iter: 785 Loss: 0.681464 Iter: 786 Loss: 0.629521 Iter: 787 Loss: 0.590525 Iter: 788 Loss: 0.586801 Iter: 789 Loss: 0.586801 Blurify Iter: 790 Loss: 1.400053 Iter: 791 Loss: 0.912270 Iter: 792 Loss: 0.752839 Iter: 793 Loss: 0.675338 Iter: 794 Loss: 0.624025 Iter: 795 Loss: 0.603171 Iter: 796 Loss: 0.603175 Blurify Iter: 797 Loss: 1.360415 Iter: 798 Loss: 0.914216 Iter: 799 Loss: 0.773592 Iter: 800 Loss: 0.682914 Iter: 801 Loss: 0.628781 Iter: 802 Loss: 0.619597 Iter: 803 Loss: 0.619599 Blurify Iter: 804 Loss: 1.344754 Iter: 805 Loss: 0.877717 Iter: 806 Loss: 0.738761 Iter: 807 Loss: 0.668881 Iter: 808 Loss: 0.629468 Iter: 809 Loss: 0.629487 Blurify Iter: 810 Loss: 1.391941 Iter: 811 Loss: 0.902576 Iter: 812 Loss: 0.752333 Iter: 813 Loss: 0.674536 Iter: 814 Loss: 0.673766 Blurify Iter: 815 Loss: 1.333019 Iter: 816 Loss: 0.903175 Iter: 817 Loss: 0.765250 Iter: 818 Loss: 0.691223 Iter: 819 Loss: 0.630375 Iter: 820 Loss: 0.591675 Iter: 821 Loss: 0.583066 Iter: 822 Loss: 0.583067 Blurify Iter: 823 Loss: 1.366395 Iter: 824 Loss: 0.884498 Iter: 825 Loss: 0.734912 Iter: 826 Loss: 0.663974 Iter: 827 Loss: 0.615942 Iter: 828 Loss: 0.606170 Iter: 829 Loss: 0.606172 Blurify Iter: 830 Loss: 1.296562 Iter: 831 Loss: 0.864799 Iter: 832 Loss: 0.739163 Iter: 833 Loss: 0.665583 Iter: 834 Loss: 0.616152 Iter: 835 Loss: 0.598536 Iter: 836 Loss: 0.598542 Blurify Iter: 837 Loss: 1.358876 Iter: 838 Loss: 0.915143 Iter: 839 Loss: 0.760596 Iter: 840 Loss: 0.679063 Iter: 841 Loss: 0.622877 Iter: 842 Loss: 0.603115 Iter: 843 Loss: 0.603119 Blurify Iter: 844 Loss: 1.368879 Iter: 845 Loss: 0.894500 Iter: 846 Loss: 0.742970 Iter: 847 Loss: 0.666316 Iter: 848 Loss: 0.646866 Iter: 849 Loss: 0.646870 Blurify Iter: 850 Loss: 1.348483 Iter: 851 Loss: 0.922918 Iter: 852 Loss: 0.762991 Iter: 853 Loss: 0.679894 Iter: 854 Loss: 0.629478 Iter: 855 Loss: 0.608967 Iter: 856 Loss: 0.608971 Blurify Iter: 857 Loss: 1.338495 Iter: 858 Loss: 0.903690 Iter: 859 Loss: 0.754525 Iter: 860 Loss: 0.674765 Iter: 861 Loss: 0.622137 Iter: 862 Loss: 0.586825 Iter: 863 Loss: 0.584948 Iter: 864 Loss: 0.584948 Blurify Iter: 865 Loss: 1.331987 Iter: 866 Loss: 0.877920 Iter: 867 Loss: 0.744623 Iter: 868 Loss: 0.664072 Iter: 869 Loss: 0.615631 Iter: 870 Loss: 0.615031 Blurify Iter: 871 Loss: 1.334820 Iter: 872 Loss: 0.916680 Iter: 873 Loss: 0.779404 Iter: 874 Loss: 0.684045 Iter: 875 Loss: 0.624200 Iter: 876 Loss: 0.584569 Iter: 877 Loss: 0.583939 Blurify Iter: 878 Loss: 1.299375 Iter: 879 Loss: 0.877454 Iter: 880 Loss: 0.747997 Iter: 881 Loss: 0.665398 Iter: 882 Loss: 0.634166 Iter: 883 Loss: 0.634169 Blurify Iter: 884 Loss: 1.390724 Iter: 885 Loss: 0.887778 Iter: 886 Loss: 0.742429 Iter: 887 Loss: 0.665075 Iter: 888 Loss: 0.613890 Iter: 889 Loss: 0.602208 Iter: 890 Loss: 0.602209 Blurify Iter: 891 Loss: 1.327836 Iter: 892 Loss: 0.931290 Iter: 893 Loss: 0.748469 Iter: 894 Loss: 0.661728 Iter: 895 Loss: 0.614207 Iter: 896 Loss: 0.572118 Iter: 897 Loss: 0.546246 Iter: 898 Loss: 0.546252 Blurify Iter: 899 Loss: 1.388196 Iter: 900 Loss: 0.888692 Iter: 901 Loss: 0.723575 Iter: 902 Loss: 0.652392 Iter: 903 Loss: 0.594746 Iter: 904 Loss: 0.557119 Iter: 905 Loss: 0.556605 Blurify Iter: 906 Loss: 1.338814 Iter: 907 Loss: 0.897129 Iter: 908 Loss: 0.748425 Iter: 909 Loss: 0.658480 Iter: 910 Loss: 0.607440 Iter: 911 Loss: 0.579476 Iter: 912 Loss: 0.579483 Blurify Iter: 913 Loss: 1.310315 Iter: 914 Loss: 0.866654 Iter: 915 Loss: 0.723339 Iter: 916 Loss: 0.645515 Iter: 917 Loss: 0.605835 Iter: 918 Loss: 0.605848 Blurify Iter: 919 Loss: 1.338557 Iter: 920 Loss: 0.884216 Iter: 921 Loss: 0.729727 Iter: 922 Loss: 0.658603 Iter: 923 Loss: 0.607693 Iter: 924 Loss: 0.573069 Iter: 925 Loss: 0.573089 Blurify Iter: 926 Loss: 1.240992 Iter: 927 Loss: 0.836057 Iter: 928 Loss: 0.706836 Iter: 929 Loss: 0.637109 Iter: 930 Loss: 0.588644 Iter: 931 Loss: 0.554079 Iter: 932 Loss: 0.553561 Blurify Iter: 933 Loss: 1.300593 Iter: 934 Loss: 0.860603 Iter: 935 Loss: 0.724729 Iter: 936 Loss: 0.647422 Iter: 937 Loss: 0.601770 Iter: 938 Loss: 0.567954 Iter: 939 Loss: 0.556640 Iter: 940 Loss: 0.556641 Blurify Iter: 941 Loss: 1.242631 Iter: 942 Loss: 0.874079 Iter: 943 Loss: 0.739234 Iter: 944 Loss: 0.650430 Iter: 945 Loss: 0.595259 Iter: 946 Loss: 0.594527 Blurify Iter: 947 Loss: 1.366264 Iter: 948 Loss: 0.895386 Iter: 949 Loss: 0.760103 Iter: 950 Loss: 0.665730 Iter: 951 Loss: 0.612236 Iter: 952 Loss: 0.579394 Iter: 953 Loss: 0.579410 Blurify Iter: 954 Loss: 1.284133 Iter: 955 Loss: 0.853311 Iter: 956 Loss: 0.717772 Iter: 957 Loss: 0.646120 Iter: 958 Loss: 0.623659 Iter: 959 Loss: 0.623661 Blurify Iter: 960 Loss: 1.399782 Iter: 961 Loss: 0.895885 Iter: 962 Loss: 0.729888 Iter: 963 Loss: 0.648027 Iter: 964 Loss: 0.632871 Iter: 965 Loss: 0.632872 Blurify Iter: 966 Loss: 1.339839 Iter: 967 Loss: 0.892210 Iter: 968 Loss: 0.745026 Iter: 969 Loss: 0.666165 Iter: 970 Loss: 0.603911 Iter: 971 Loss: 0.563752 Iter: 972 Loss: 0.563282 Blurify Iter: 973 Loss: 1.283730 Iter: 974 Loss: 0.858825 Iter: 975 Loss: 0.730703 Iter: 976 Loss: 0.656177 Iter: 977 Loss: 0.604913 Iter: 978 Loss: 0.561716 Iter: 979 Loss: 0.535116 Iter: 980 Loss: 0.535120 Blurify Iter: 981 Loss: 1.226458 Iter: 982 Loss: 0.827190 Iter: 983 Loss: 0.685638 Iter: 984 Loss: 0.616972 Iter: 985 Loss: 0.570298 Iter: 986 Loss: 0.569601 Blurify Iter: 987 Loss: 1.289554 Iter: 988 Loss: 0.876034 Iter: 989 Loss: 0.734361 Iter: 990 Loss: 0.645794 Iter: 991 Loss: 0.584484 Iter: 992 Loss: 0.541989 Iter: 993 Loss: 0.529610 Iter: 994 Loss: 0.529612 Blurify Iter: 995 Loss: 1.199055 Iter: 996 Loss: 0.801824 Iter: 997 Loss: 0.675154 Iter: 998 Loss: 0.604573 Iter: 999 Loss: 0.557178
As it turns out, the fmin_l_bfgs_b optimizer used in the original gets stuck in a local minimum in approximately 13-15 iterations, with a loss of 3.19. It does not seem to move from there regardless of how many more iterations it is run for.
This is what the image looks like at that point:
Image.open('/home/ubuntu/results/res_at_iteration_14.png')
If the image is blurred whenever the loss stops improving, after 994 iterations the loss reaches 0.53. The image looks like this:
Image.open('/home/ubuntu/results/res_at_iteration_994.png')
The original image, for comparison:
img
The loss history shows a pattern of going up by roughly 0.5 after each blur, then quickly dropping into another local minimum. Not every local minimum is better than the previous one, but over time there is a definite downward trend...
plt.figure(figsize=(16,8))
plt.plot([loss_history[i] for i in range(iterations)])
plt.ylim((0, 4))
plt.xlim((0, 1000))
(0, 1000)
It is quite likely that even after 1000 iterations this is not fully converged.
Here are some possible reasons why this works:
We can suppose that realistic images form a manifold in the space of all images. One of the key properties of that manifold is that nearby pixels are highly correlated: edges are rare. The optimizer (BFGS in this case, but that is likely not specific to the choice of optimizer) is unaware of this, and moves in the direction of the local gradient which in general is away from the manifold. Once the image starts being sufficiently different in type (or pixel-pixel correlation statistics) from a natural image, the gradient of the image wrt the feaure map that the network produces stops being useful, and it falls into a (likely shallow) local minimum.
Instead, we want to move along the manifold. In this case, we keep moving the image back to the manifold whenever it has diverged, by essentially forcing a correlation between nearby pixels by brute force ie blurring. However, it should also be possible to add a loss term that corresponds to the correlation of pairs of pixels vs their distance, or (to keep things simple) just the correlation of nearest-neighbor pixels.
Use this for style transfer, not just to recreate input image
Modify the loss by adding a pixel-pixel correlation term, or nearest-neighbor correlation term, and see if we can prevent local minima entirely
Figure out how long it takes for this to fully converge...
Try different sigma values. Sigma=1 is likely not optimal, it just kind of works
Try blurring more often, perhaps a slight blur at every step without waiting for BFGS to converge