It is worth noting that the SVD api in numpy, tensorflow and pytorch are different in several aspects, which I will give a short demo here.

library import and sys info¶

In [1]:

import numpy as np
import tensorflow as tf
import torch

In [3]:

import platform, sys
platform.platform(), sys.version # OS, python system info

Out[3]:

('Linux-4.15.0-43-generic-x86_64-with-Ubuntu-16.04-xenial',
 '3.5.2 (default, Nov 12 2018, 13:43:14) \n[GCC 5.4.0 20160609]')

In [5]:

np.__version__, tf.__version__, torch.__version__ # library version info

Out[5]:

('1.16.2', '1.13.1', '1.0.1.post2')

In [6]:

tf.enable_eager_execution()

SVD in numpy¶

General SVD theory: $A=USV^{H}$ , where $H$ is for transpose conjugate, $U, V$ is unitary matrix. The corresponding shape of the decomposion is $n\times n, n\times m, m\times m$ , where the shape of A is $n\times m$ . $S$ only has nonzero element in the main diagonal, which are called single values.

Furthermore, WLOG, assume $n>m$ , we can recast the decomposition as $A_{n\times m}=U'_{n\times m}S'_{m\times m}V^{H}_{m\times m}$ , where $U'$ and $S'$ is just simple cut of original matrix. Some library gives this reduced decomposition by default, such a feature is usually controlled by full_matrices flag.

In [52]:

a = np.random.rand(4,3) # the matrix to be SVD decomposed in this work
a

Out[52]:

array([[0.05845521, 0.45514259, 0.84723484],
       [0.20908514, 0.26327405, 0.40786231],
       [0.48486814, 0.04937339, 0.69909792],
       [0.00927412, 0.52368466, 0.05372105]])

In [53]:

u_np, s_np, vh_np = np.linalg.svd(a, full_matrices=True, compute_uv=True) # default paramters for svd in np

In [54]:

u_np.shape, s_np.shape, vh_np.shape

Out[54]:

((4, 4), (3,), (3, 3))

In [55]:

u_np, s_np, vh_np = np.linalg.svd(a, full_matrices=False, compute_uv=True) 

In [58]:

u_np.shape, s_np.shape, vh_np.shape

Out[58]:

((4, 3), (3,), (3, 3))

In [57]:

np.isclose((u_np*s_np)@vh_np, a), np.isclose((u_np*s_np)@vh_np.T, a) ## the given v matrix is already vh

Out[57]:

(array([[ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True]]), array([[False, False, False],
        [False, False, False],
        [False, False, False],
        [False, False, False]]))

In [12]:

np.linalg.svd(a, compute_uv=False) # gives only single value, no zero placehold for u and v matrix

Out[12]:

array([1.51996147, 0.55862926, 0.07539824])

In [66]:

np.svd ## no alias

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-66-e5798aee4f5f> in <module>()
----> 1 np.svd

AttributeError: module 'numpy' has no attribute 'svd'

SVD in tensorflow¶

In [60]:

s_tf, u_tf, v_tf = tf.linalg.svd(a, full_matrices=False, compute_uv=True) 
# default parameters for svd in np, the default value is different from numpy API
# the sorted order of s u v is also different

In [62]:

s_tf, u_tf, v_tf = s_tf.numpy(), u_tf.numpy(), v_tf.numpy()

In [64]:

s_tf.shape, u_tf.shape, v_tf.shape

Out[64]:

((3,), (4, 3), (3, 3))

In [63]:

np.isclose((u_tf*s_tf)@v_tf, a), np.isclose((u_tf*s_tf)@v_tf.T, a) ## the given v matrix is not vh

Out[63]:

(array([[False, False, False],
        [False, False, False],
        [False, False, False],
        [False, False, False]]), array([[ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True]]))

In [65]:

tf.linalg.svd(a, compute_uv=False)  ## return only singular value when compute_uv is false

Out[65]:

<tf.Tensor: id=16, shape=(3,), dtype=float64, numpy=array([1.34306156, 0.57709753, 0.2727537 ])>

In [67]:

tf.svd ## alias exists

Out[67]:

<function tensorflow.python.ops.linalg_ops.svd(tensor, full_matrices=False, compute_uv=True, name=None)>

SVD in pytorch¶

In [76]:

u_pt, s_pt, v_pt = torch.svd(torch.tensor(a), some=True, compute_uv=True) 
# default paramter in pytorch
# some is the exact opposite thing as full_matrices above, i.e. if some is true by default, returned u is reduced
# besides, one must give a input with torch.tensor format, a numpy.array will raise error, which is not the case in tensorflow

In [78]:

u_pt, s_pt, v_pt = u_pt.numpy(), s_pt.numpy(), v_pt.numpy()

In [79]:

u_pt.shape, s_pt.shape, v_pt.shape ## reduced form due to some = True

Out[79]:

((4, 3), (3,), (3, 3))

In [80]:

np.isclose((u_pt*s_pt)@v_pt, a), np.isclose((u_pt*s_pt)@v_pt.T, a) ## the given v matrix is not vh

Out[80]:

(array([[False, False, False],
        [False, False, False],
        [False, False, False],
        [False, False, False]]), array([[ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True]]))

In [81]:

torch.svd(torch.tensor(a), compute_uv=False)  ## u, v matrices is still returned as full 0 matrix, 
# some is ignored, since u is always n times n

Out[81]:

(tensor([[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]], dtype=torch.float64),
 tensor([1.3431, 0.5771, 0.2728], dtype=torch.float64),
 tensor([[0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.]], dtype=torch.float64))

library import and sys info¶

SVD in numpy¶

SVD in tensorflow¶

SVD in pytorch¶

References¶