It is worth noting that the SVD api in numpy, tensorflow and pytorch are different in several aspects, which I will give a short demo here.

library import and sys info

In [1]:
import numpy as np
import tensorflow as tf
import torch
In [3]:
import platform, sys
platform.platform(), sys.version # OS, python system info
Out[3]:
('Linux-4.15.0-43-generic-x86_64-with-Ubuntu-16.04-xenial',
 '3.5.2 (default, Nov 12 2018, 13:43:14) \n[GCC 5.4.0 20160609]')
In [5]:
np.__version__, tf.__version__, torch.__version__ # library version info
Out[5]:
('1.16.2', '1.13.1', '1.0.1.post2')
In [6]:
tf.enable_eager_execution()

SVD in numpy

General SVD theory: $A=USV^{H}$, where $H$ is for transpose conjugate, $U, V$ is unitary matrix. The corresponding shape of the decomposion is $n\times n, n\times m, m\times m$, where the shape of A is $n\times m$. $S$ only has nonzero element in the main diagonal, which are called single values.

Furthermore, WLOG, assume $n>m$, we can recast the decomposition as $A_{n\times m}=U'_{n\times m}S'_{m\times m}V^{H}_{m\times m}$, where $U'$ and $S'$ is just simple cut of original matrix. Some library gives this reduced decomposition by default, such a feature is usually controlled by full_matrices flag.

In [52]:
a = np.random.rand(4,3) # the matrix to be SVD decomposed in this work
a
Out[52]:
array([[0.05845521, 0.45514259, 0.84723484],
       [0.20908514, 0.26327405, 0.40786231],
       [0.48486814, 0.04937339, 0.69909792],
       [0.00927412, 0.52368466, 0.05372105]])
In [53]:
u_np, s_np, vh_np = np.linalg.svd(a, full_matrices=True, compute_uv=True) # default paramters for svd in np
In [54]:
u_np.shape, s_np.shape, vh_np.shape
Out[54]:
((4, 4), (3,), (3, 3))
In [55]:
u_np, s_np, vh_np = np.linalg.svd(a, full_matrices=False, compute_uv=True) 
In [58]:
u_np.shape, s_np.shape, vh_np.shape
Out[58]:
((4, 3), (3,), (3, 3))
In [57]:
np.isclose((u_np*s_np)@vh_np, a), np.isclose((u_np*s_np)@vh_np.T, a) ## the given v matrix is already vh
Out[57]:
(array([[ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True]]), array([[False, False, False],
        [False, False, False],
        [False, False, False],
        [False, False, False]]))
In [12]:
np.linalg.svd(a, compute_uv=False) # gives only single value, no zero placehold for u and v matrix
Out[12]:
array([1.51996147, 0.55862926, 0.07539824])
In [66]:
np.svd ## no alias
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-66-e5798aee4f5f> in <module>()
----> 1 np.svd

AttributeError: module 'numpy' has no attribute 'svd'

SVD in tensorflow

In [60]:
s_tf, u_tf, v_tf = tf.linalg.svd(a, full_matrices=False, compute_uv=True) 
# default parameters for svd in np, the default value is different from numpy API
# the sorted order of s u v is also different
In [62]:
s_tf, u_tf, v_tf = s_tf.numpy(), u_tf.numpy(), v_tf.numpy()
In [64]:
s_tf.shape, u_tf.shape, v_tf.shape
Out[64]:
((3,), (4, 3), (3, 3))
In [63]:
np.isclose((u_tf*s_tf)@v_tf, a), np.isclose((u_tf*s_tf)@v_tf.T, a) ## the given v matrix is not vh
Out[63]:
(array([[False, False, False],
        [False, False, False],
        [False, False, False],
        [False, False, False]]), array([[ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True]]))
In [65]:
tf.linalg.svd(a, compute_uv=False)  ## return only singular value when compute_uv is false
Out[65]:
<tf.Tensor: id=16, shape=(3,), dtype=float64, numpy=array([1.34306156, 0.57709753, 0.2727537 ])>
In [67]:
tf.svd ## alias exists
Out[67]:
<function tensorflow.python.ops.linalg_ops.svd(tensor, full_matrices=False, compute_uv=True, name=None)>

SVD in pytorch

In [76]:
u_pt, s_pt, v_pt = torch.svd(torch.tensor(a), some=True, compute_uv=True) 
# default paramter in pytorch
# some is the exact opposite thing as full_matrices above, i.e. if some is true by default, returned u is reduced
# besides, one must give a input with torch.tensor format, a numpy.array will raise error, which is not the case in tensorflow
In [78]:
u_pt, s_pt, v_pt = u_pt.numpy(), s_pt.numpy(), v_pt.numpy()
In [79]:
u_pt.shape, s_pt.shape, v_pt.shape ## reduced form due to some = True
Out[79]:
((4, 3), (3,), (3, 3))
In [80]:
np.isclose((u_pt*s_pt)@v_pt, a), np.isclose((u_pt*s_pt)@v_pt.T, a) ## the given v matrix is not vh
Out[80]:
(array([[False, False, False],
        [False, False, False],
        [False, False, False],
        [False, False, False]]), array([[ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True],
        [ True,  True,  True]]))
In [81]:
torch.svd(torch.tensor(a), compute_uv=False)  ## u, v matrices is still returned as full 0 matrix, 
# some is ignored, since u is always n times n
Out[81]:
(tensor([[0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.],
         [0., 0., 0., 0.]], dtype=torch.float64),
 tensor([1.3431, 0.5771, 0.2728], dtype=torch.float64),
 tensor([[0., 0., 0.],
         [0., 0., 0.],
         [0., 0., 0.]], dtype=torch.float64))