Notebook

Topic 25. Fundamental Subspaces of a Matrix¶

Author: Autumn Twitchell atwitch23@gmail.com

Introduction¶

The fundamental subspaces of a matrix are four vector spaces of a matrix. If we are given a matrix mxn matrix $A$ , the fundamental spaces are as follows:

The range space (or column space) of $A$ : $R(A)$
The nullspace (or kernel) of $A$ : $N(A)$
The range space of $A^*$ (or row space of $A$ ): $R(A^*)$ and
The nullspace of $A^*$ (or left nullspace of $A$ ): $N(A^*)$ .

Note: If the matrix $A$ is real, $A^* = A^T$ . If the matrix $A$ is complex, $A^* = A^H$ .

The four fundamental subspaces are important to understand the properties of matrices such as the rank of a matrix. They are also important in the Fundamental Theorem of Linear Algebra. The subspaces are also connected to the Singular Value Decomposition (SVD) of a matrix. When we know what the four subspaces of a matrix are, we have a good understanding of whether the matrix has no solution, one solution, or many solutions. Knowing this helps us to know how to find the best solution for the matrix.

Explanation of the theory¶

$R(A)$ ¶

Let $A$ be an mxn matrix. An easy way to think about $R(A)$ is to remember that it is called the column space of $A$ , thus it is characterized as the span of the columns of $A$ . So if we show $A$ as $A = [c_1,c_2,...,c_n]$ , then a point $x \in \mathbb{R}^n$ is transformed as $Ax = x_1c_1 + x_2c_2 + ... + x_nc_n$ , which is a linear combination of the columns of $A$ . So the range can be expressed as $R(A) = span(\{c_1,c_2,...,c_n\})$ .

$N(A)$ ¶

To find the nullspace, we are looking to solve the equation $Ax = 0$ . It is often easiest to solve by reducing the matrix to row echelon form.

$R(A^)$ and $N(A^)$ ¶

The method of solving $R(A^*)$ and $N(A^*)$ is extremely similar to solving $R(A)$ and $N(A)$ . The difference is that we are looking at the transpose of $A$ (or hermitian if $A \in \mathbb{C}^{mxn}$ ). So we transpose the matrix and find the range space and nullspace of the transformed matrix.

Fundamental Theorem of Linear Algebra¶

Some important properties for the fundamental subspaces are found in the fundamental theorem of linear algebra. It states that the column and row spaces ( $R(A)$ and $R(A^*)$ ) have the same dimension r, which is the rank of the matrix. The nullspace has dimension n−r, and the left nullspace has dimension $m−r$ . This means that if we are looking at an mxn matrix where $m \geq n$ , and it is full rank, the nullspace will only contain the zero vector.

Another part of the fundamental theorem of algebra is that the nullspace and row space are orthogonal, as well as the range space and left nullspace. These properties are depicted in Dr. Beard's telephone pole diagram depicted below.

alt text

As mentioned in the previous section, the fundamental subspaces are connected to the SVD of a matrix. If A is an mxn matrix, where $A = U\Sigma V^H$ , we find that

$R(A) = span(U_1)$

$N(A) = span(V_2)$

$R(A^*) = span(V_1)$

$N(A^*) = span(U_2)$

where $U$ is an mxm matrix, $\Sigma$ is an mxn diagonal matrix, and $V$ is an nxn matrix. $U_1$ is the first $r$ columns of $U$ and $U_2$ is the last $m-r$ columns of $U$ . $V_1$ is the first $r$ columns of $V$ and $V_2$ is the last $n-r$ columns of $V$ .
So then our SVD looks something like this

$A = U\Sigma V^H = \begin{bmatrix} U_1 & U_2\\ \end{bmatrix} \begin{bmatrix} \Sigma_1 & 0\\ 0 & \Sigma_2\\ \end{bmatrix} \begin{bmatrix} V_1^H \\ V_2^H\\ \end{bmatrix}$

More information on SVDs can be found Topic 24. Singular Value Decomposition.

Simple Numerical Examples¶

We're going to look at two matrices and solve for their four fundamental subspaces.

Example 1: Square Matrix¶

Suppose $A = \begin{bmatrix} 1 & 3 & 3\\ 4 & 5 & 6\\ 7 & 8 & 9\\ \end{bmatrix}$

Solving for $R(A)$ ¶

We can easily spot that the columns of $A$ are $\begin{bmatrix} 1 \\ 4 \\ 7 \\ \end{bmatrix}, \begin{bmatrix} 3\\ 5 \\ 8 \\ \end{bmatrix}, \begin{bmatrix} 3\\ 6 \\ 9 \\ \end{bmatrix}$

Thus $R(A) = span\Bigg(\Bigg\{ \begin{bmatrix} 1 \\ 4 \\ 7 \\ \end{bmatrix}, \begin{bmatrix} 3\\ 5\\ 8\\ \end{bmatrix}, \begin{bmatrix} 3\\6\\ 9\\ \end{bmatrix}\Bigg\}\Bigg)$

It isn't necessary to include all of the columns of $A$ if some are linearly dependent. If we were unsure as to whether they are linearly independent we could verify this with some linear algebra (see Topic 4. Linear Independence for more information). Here we're just going to let Python do it for us:

In [23]:

# Import modules for linear algebra
import numpy as np
import sympy
sympy.init_printing()

A = sympy.Matrix([[1,3,3], [4,5,6], [7,8,9]])
print("A = ")
display(A)

# row reduce to see if it's linearly independent
rr,a = A.rref()

print("rref of A = ")
display(rr)

A =

$\displaystyle \left[\begin{matrix}1 & 3 & 3\\4 & 5 & 6\\7 & 8 & 9\end{matrix}\right]$

rref of A =

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

When we row reduce the matrix $A$ , we can see that the rank of $A$ is 3, which is the number of columns and rows of $A$ , thus $A$ is full rank and linearly independent. Therefore, we include all the columns of $A$ in the range of $A$ .

Solving for $N(A)$ ¶

Now to find the nullspace $N(A)$ , we solve for the equation $Ax = 0$

Thus we have $\begin{bmatrix} 1 & 3 & 3\\ 4 & 5 & 6\\ 7 & 8 & 9\\ \end{bmatrix} \begin{bmatrix} x_1\\ x_2\\ x_3\\ \end{bmatrix} = \begin{bmatrix} 0\\ 0\\ 0\\ \end{bmatrix}$

Interesting property of a matrix with linearly independent columns is that there is no vector $x$ that will that will make $Ax = 0$ unless $x$ is the zero vector. Therefore $N(A) = 0$

Solving for $R(A^*)$ ¶

Because $A$ is a real matrix we know we will be solving for $R(A^T)$ . As was mentioned in the introduction, the range space of $A^T$ is equivalent to the row space of $A$ because we are looking at the columns of $A^T$ .

$A^T = \begin{bmatrix} 1 & 4 & 7\\ 3 & 5 & 8\\ 3 & 6 & 9\\ \end{bmatrix}$

We can see here that the column space of $A^T$ is equal to the $span\Bigg(\Bigg\{ \begin{bmatrix} 1 \\ 3 \\ 3 \\ \end{bmatrix}, \begin{bmatrix} 4\\ 5\\ 6\\ \end{bmatrix}, \begin{bmatrix} 7\\ 8\\ 9\\ \end{bmatrix}\Bigg\}\Bigg)$

Checking for linear independence, we get:

In [25]:

AT = A.T
# row reduce to see if it's linearly independent
rrat,a = AT.rref()

print("rref of A^T = ")
display(rrat)

rref of A^T =

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

Yay! The columns of $A^T$ are linearly independent which means the span of all the columns of $A^T$ are in $R(A^T)$ . So $R(A^T) = span\Bigg(\Bigg\{ \begin{bmatrix} 1 \\ 3 \\ 3 \\ \end{bmatrix}, \begin{bmatrix} 4\\ 5\\ 6\\ \end{bmatrix}, \begin{bmatrix} 7\\ 8\\ 9\\ \end{bmatrix}\Bigg\}\Bigg)$

Solving for $N(A^T)$ ¶

Similar to finding $N(A)$ , since the columns of $A^T$ are linearly independent, $N(A^T) = 0$ .

Example 2: Stout Matrix¶

(which means our transposed matrix will be tall)

Suppose $B = \begin{bmatrix} 2 & 3 & 4\\ 5 & 6 & 7\\ \end{bmatrix}$

Solving for $R(B)$ ¶

$R(B) = span\Bigg(\Bigg\{ \begin{bmatrix} 2 \\ 5 \\ \end{bmatrix}, \begin{bmatrix} 3\\ 6\\ \end{bmatrix}, \begin{bmatrix} 4\\ 7\\ \end{bmatrix}\Bigg\}\Bigg)$

Solving for $N(B)$ ¶

Solve $Bx = 0$

Thus we have $\begin{bmatrix} 2 & 3 & 4\\ 5 & 6 & 7\\ \end{bmatrix} \begin{bmatrix} x_1\\ x_2\\ x_3\\ \end{bmatrix} = \begin{bmatrix} 0\\ 0\\ \end{bmatrix}$

If we row reduce the matrix, we can find our nullspace:

In [45]:

B = sympy.Matrix([[2,3,4], [5,6,7]])

# row reduce to see if it's linearly independent
rrb,b = B.rref()

print("rref of B = ")
display(rrb)

rref of B =

$\displaystyle \left[\begin{matrix}1 & 0 & -1\\0 & 1 & 2\end{matrix}\right]$

We can see from the row reduced form that this matrix is full rank, but since there are more m columns than n rows, the nullspace will have a dimension of m-n. So in this case we should have one column in our nullspace. From the row reduced form we get the following equations:

$\begin{equation} \begin{cases} x_1 - x_3 = 0\\ x_2 + 2x_3 = 0 \end{cases} \end{equation}$

Thus we have $\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} = \begin{bmatrix} x_3\\ -2x_3\\ x_3\\ \end{bmatrix}$

Setting $x_3$ to 1, we have a basis for our nullspace: $N(B) = span\Bigg(\begin{bmatrix} 1\\ -2\\ 1\\ \end{bmatrix}\Bigg)$

Solving for $R(B^T)$ ¶

$B^T = \begin{bmatrix} 2 & 5 \\ 3 & 6\\ 4 & 7\\ \end{bmatrix}$

Thus $R(B^T) = span\Bigg(\begin{bmatrix}2 \\ 3\\ 4 \\ \end{bmatrix}, \begin{bmatrix} 5 \\ 6\\ 7 \\ \end{bmatrix}\Bigg)$

Solving for $N(B^T)$ ¶

Solve for

$\begin{bmatrix} 2 & 5 \\ 3 & 6\\ 4 & 7\\ \end{bmatrix} \begin{bmatrix} x_1 \\ x_2\\ \end{bmatrix} = \begin{bmatrix} 0 \\ 0\\ 0\\ \end{bmatrix}$

Let's look at the reduced row echelon form:

In [46]:

# row reduce to see if it's linearly independent
rrb,b = B.T.rref()

print("rref of B = ")
display(rrb)

rref of B =

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\\0 & 0\end{matrix}\right]$

Because this is a tall matrix, our solution is overdetermined. Since our columns are linearly independent, the matrix is considered full rank and therefore $N(B^T)=0$ .

An Engineering Application¶

Linear algebra is heavily used when it comes to solving electrical circuits. In most simple circuits that are solved, there is one solution for the system of equations. This means that the matrix used to solve the linear equations of the circuit is linearly independent and full rank, therefore the dimensions of $R(A)$ and $R(A^T)$ are the minimum value between the columns and rows of the matrix. This also means that $N(A)$ and $N(A^T)$ only contain the zero vector.

In the case below, we are going to look at the current of a circuit that doesn't have any elements to it besides wire.

alt text

In the picture above, we are going to solve for the difference in current (current is represented by the $b$ 's) at the nodes $x_1,x_2,x_3,x_4$ .

Solving for the currents, we have:

$\begin{equation} \begin{cases} -x_1 + x_3 = b_1\\ x_3 - x_4 = b_2\\ -x_1 + x_4 = b_3\\ x_1 - x_2 = b_4\\ -x_2 + x_4 = b_5 \end{cases} \end{equation}$

Which forms $Ax = b$ : $\begin{bmatrix} -1 & 0 & 1 & 0 \\ 0 & 0 & 1 & -1\\ -1 & 0 & 0 & 1\\ 1 & -1 & 0 & 0\\ 0 & -1 & 0 & 1\\ \end{bmatrix} \begin{bmatrix} x_1\\ x_2\\ x_3\\ x_4\\ \end{bmatrix} = \begin{bmatrix} b_1\\ b_2\\ b_3\\ b_4\\ \end{bmatrix}$

To see if our matrix $A$ is linearly independent, we'll row reduce the matrix using Python.

In [85]:

A = sympy.Matrix([[-1,0,1,0], [0,0,1,-1], [-1,0,0,1],[1,-1,0,0], [0,-1,0,1]])
rref,a  = A.rref()

print("rref of A = ")
display(rref)

rref of A =

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & -1\\0 & 1 & 0 & -1\\0 & 0 & 1 & -1\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right]$

This shows us that the rank of the matrix is 3 since there are 3 rows with values in the matrix after the row reduction. Since the fourth column is a linear combination of the first three, we do not need to include it in our range space.

Therefore we get $R(A) = span\Bigg\{\begin{bmatrix} -1\\ 0\\ -1\\ 1\\ 0\\ \end{bmatrix}, \begin{bmatrix} 0\\ 0\\ 0\\ -1\\ -1\\ \end{bmatrix}, \begin{bmatrix} 1\\ 1\\ 0\\ 0\\ 0\\ \end{bmatrix}\Bigg\}$

Now we can keep going like this, or we can use some Python :)

In [142]:

# Use sympy.columnspace() and sympy.nullspace() method  
A_columnspace = A.columnspace()
A_nullspace = A.nullspace()
A_T_columnspace = A.T.columnspace()   
A_T_nullspace = A.T.nullspace() 
      
print("R(A) = span") 
display(A_columnspace)

print("N(A) = span") 
display(A_nullspace)

print("R(A^T) = span") 
display(A_T_columnspace)

print("N(A^T) = span") 
display(A_T_nullspace)

R(A) = span

$\displaystyle \left[ \left[\begin{matrix}-1\\0\\-1\\1\\0\end{matrix}\right], \ \left[\begin{matrix}0\\0\\0\\-1\\-1\end{matrix}\right], \ \left[\begin{matrix}1\\1\\0\\0\\0\end{matrix}\right]\right]$

N(A) = span

$\displaystyle \left[ \left[\begin{matrix}1\\1\\1\\1\end{matrix}\right]\right]$

R(A^T) = span

$\displaystyle \left[ \left[\begin{matrix}-1\\0\\1\\0\end{matrix}\right], \ \left[\begin{matrix}0\\0\\1\\-1\end{matrix}\right], \ \left[\begin{matrix}1\\-1\\0\\0\end{matrix}\right]\right]$

N(A^T) = span

$\displaystyle \left[ \left[\begin{matrix}-1\\1\\1\\0\\0\end{matrix}\right], \ \left[\begin{matrix}-1\\1\\0\\-1\\1\end{matrix}\right]\right]$

Knowing these spaces helps us to know what kind of problem we are dealing with. In this case the problem has many solutions.

Challenge Problem¶

For an mxn matrix $A$ , show that the $dim(N(A)) = n- rank(A)$ .

In [ ]:

Topic 25. Fundamental Subspaces of a Matrix¶

Introduction¶

Explanation of the theory¶

R(A)R(A)¶

N(A)N(A)¶

R(A∗)R(A^*) and N(A∗)N(A^*)¶

Fundamental Theorem of Linear Algebra¶

Simple Numerical Examples¶

Example 1: Square Matrix¶

Solving for R(A)R(A)¶

Solving for N(A)N(A)¶

Solving for R(A∗)R(A^*)¶

Solving for N(AT)N(A^T)¶