Vector and Matrix norms

$\newcommand{\magn}[1]{\left\lVert#1\right\rVert}$ $\newcommand{\Rb}{\mathbb{R}}$ $\newcommand{\abs}[1]{\left\lvert#1\right\rvert}$ $\newcommand{\bkt}[1]{\left(#1\right)}$ $\newcommand{\dsum}{\displaystyle\sum}$

Recall that when we defined condition number, we relied on the notion of norm. In today's lecture, we will formalize this notion.

A norm (usually denoted as $\magn{\cdot}$) is a mapping from the underlying vector space (say $V$) to set of non-negative real numbers ($[0,\infty)$) satisfying the following conditions:

  1. $\magn{\cdot}: V \mapsto [0,\infty)$
  2. $\magn{v} = 0$ iff $v = 0$
  3. For all $v \in V$ and $\alpha \in \Rb$, $\magn{\alpha v} = \abs{\alpha} \magn{v}$
  4. Triangle inequality: For all $v,w \in V$, we have $$\magn{v+w} \leq \magn{v} + \magn{w}$$

On a given vector space, there can be many possible norms.

One of the most common norm on $\Rb^n$ is the $p$-norm defined as below. Consider $x = \begin{bmatrix}x_1 & x_2 & \cdots & x_n\end{bmatrix}^T \in \Rb^n$. We then define the $p$-norm of the vector $x$ as $$\magn{x}_p = \bkt{\abs{x_1}^p + \abs{x_2}^p + \cdots + \abs{x_n}^p}^{1/p}$$ It is immediate to note that the $p$-norm as defined above is a norm only when $p \geq 1$. (Exercise: Check)

Note that $$\lim_{p \to \infty} \magn{x}_p = \max_i \abs{x_i}$$ is also a valid norm on $\Rb^n$ (Exercise: Check). This norm is denoted as $\magn{x}_{\infty}$.

Matrix norms:

Let $A \in \Rb^{m \times n}$. We could take inspiration from the way we have defined norms on $\Rb^n$ to define matrix norms, i.e., we could define the $p$-norm (for $p \geq 1$) of the matrix as $$\bkt{\dsum_{i=1}^m \dsum_{j=1}^n \abs{A_{ij}}^p}^{1/p}$$ The above is indeed a valid norm. However, in the cases of matrices, we would prefer the norm to satisfy an additional criteria, namely the notion of sub-multiplicativity. We say that a matrix-norm ($\magn{\cdot}$) is sub-multiplicative if $$\magn{AB} \leq \magn{A} \magn{B}$$ The way we have defined the $p$-norm above, it is easy to show that the norm is not sub-multiplicative always (except for $p=2$).

A more natural definition of matrix-norm is the notion of vector induced matrix norm, which is defined as follows.

$$\magn{A}_{p,q} = \max \left\{\dfrac{\magn{Ax}_{p}}{\magn{x}_q}: x \in \Rb^n - \{0\}\right\}$$

It is easy to show that (Exercise: Check) the above definition can also be rewritten as

$$\magn{A}_{p,q} = \max \left\{\magn{Ax}_{p} : \magn{x}_q = 1 \right\}$$

When $p=q$, it is convinient to denote that above as just $\magn{A}_p$ instead of $\magn{A}_{p,p}$.

Lemma: $\magn{A}_{\infty} = \text{Maximum absolute row sum of the matrix} = \max_{i \in \{1,2,\ldots,m\}} \dsum_{j=1}^n \abs{A_{ij}}$

Proof: We have $$\magn{A}_{\infty} = \max \left\{\magn{Ax}_{\infty} : \magn{x}_{\infty} = 1 \right\}$$ i.e., $\magn{A}_{\infty} = \max_{i \in \{1,2,\ldots,m\}} \abs{\dsum_{j=1}^n A_{ij}x_j}$ subject to the condition that $\max_j \abs{x_j} = 1$.

We have $$\abs{\dsum_{j=1}^n A_{ij}x_j} \leq \dsum_{j=1}^n \abs{A_{ij}} \abs{x_j} \leq \dsum_{j=1}^n \abs{A_{ij}} $$ where the last inequality is due to the fact that $\abs{x_j} \leq \max_k \abs{x_k} = 1$. Let $i^*$ be the row number corresponding to the largest row sum. Hence, we see that $$\magn{A}_{\infty} \leq \max_{i \in \{1,2,\ldots,m\}} \dsum_{j=1}^n \abs{A_{ij}} = \dsum_{j=1}^n \abs{A_{i^* j}}$$

Now note that equality can be attained by choosing the vector $x_j = \text{sign}\bkt{A_{i^* j}}$. Hence, we see that $$\magn{A}_{\infty} = \max_{i \in \{1,2,\ldots,m\}} \dsum_{j=1}^n \abs{A_{ij}}$$

Hence, $\magn{A}_{\infty}$ is the maximum absolute row sum of the matrix.