Energy auto-encoder: comparison with Xavier Primal-Dual matlab implementation


In [ ]:
import numpy as np
import numpy.linalg as la
import matplotlib.pyplot as plt
%matplotlib inline
from pyunlocbox import functions, solvers
import time
import scipy, matplotlib, pyunlocbox  # For versions only.

# Import auto-encoder definition.
%run auto_encoder.ipynb

print('Software versions:')
for pkg in [np, matplotlib, scipy, pyunlocbox]:
    print('  %s: %s' % (pkg.__name__, pkg.__version__))


  • The $\lambda$ are the relative importance of each term in the composite objective function.
In [ ]:
ld = 10  # Xavier sets the weight of the L1 regularization to 1e-1.


  • The set of data vectors $X \in R^{n \times N}$ is given by patches extracted from a grayscale image.
  • There is as many patches as pixels in the image.
  • The saved patches already have zero mean.
In [ ]:
mat ='data/xavier_X.mat')
X = mat['X'].T

N, n = X.shape
Np = np.sqrt(n)
print('N = %d samples with dimensionality n = %d (patches of %dx%d).' % (N, n, Np, Np))

patches = [24, 1000, 2004, 10782]
for k in range(len(patches)):
    patch = patches[k]
    img = np.reshape(X[patch, :], (Np, Np))
    plt.subplot(1, 4, k+1)
    plt.imshow(img, cmap='gray')
    plt.title('Patch %d' % patch)

Initial conditions

  • $Z$ is drawn from a uniform distribution in ]0,1[.
  • Same for $D$. Its columns were then normalized to unit L2 norm.
  • The sparse code dimensionality $m$ should be greater than $n$ for an overcomplete representation but much smaller than $N$ to avoid over-fitting.
  • The dictionary and sparse codes are initialized this way in the auto_encoder class.
In [ ]:
if False:
    mat ='data/xavier_initZD.mat')
    Zinit = mat['Zinit']
    Dinit = mat['Dinit']

    m, N = Zinit.shape
    n, m = Dinit.shape
    print('Sparse code dimensionality m = %d --> %s dictionary' % (m, 'overcomplete' if m > n else 'undercomplete'))

    print('mean(Z) = %f' % np.mean(Zinit))

    d = np.sqrt(np.sum(Dinit*Dinit, axis=0))
    print('Constraints on D: %s' % np.alltrue(d <= 1+1e-15))


Given $X \in R^{n \times N}$, solve $\min\limits_{Z \in R^{m \times N}, D \in R^{n \times m}} \frac{\lambda_d}{2} \|X - DZ\|_F^2 + \|Z\|_1$ s.t. $\|d_i\|_2 \leq 1$, $i = 1, \ldots, m$

In [ ]:
ae = auto_encoder(m=64, ld=ld, rtol=1e-5, xtol=None, N_inner=100, N_outer=30)
#ae = auto_encoder(m=64, ld=ld, rtol=None, xtol=1e-6, N_inner=100, N_outer=15)
tstart = time.time()
Z = ae.fit_transform(X)
print('Elapsed time: {:.0f} seconds'.format(time.time() - tstart))
In [ ]:
if False:
    tstart = time.time()
    Z = ae.transform(X)
    print('Elapsed time: {:.0f} seconds'.format(time.time() - tstart))


In [ ]:

Solution analysis

Solution from Xavier

In [ ]:
mat ='data/xavier_ZD.mat')
Zxavier = mat['Z'].T
Dxavier = mat['D'].T
print('Elapsed time: %d seconds' % mat['exectime'])


In [ ]:
objective(X, Zxavier, Dxavier, ld)
objective(X, Z, ae.D, ld)

Sparse codes

In [ ]:


In [ ]:
In [ ]:
atoms(Dxavier, Np)
atoms(ae.D, Np)