Multivariante Gaussians and Mixtures of Gaussians (MoG)

In [11]:
import matplotlib.pyplot as plt
import numpy as np
from numpy import *
from mpl_toolkits.mplot3d import Axes3D
%matplotlib inline  

First, let's generate a "2D cloud" of points by independently generating $x_1$'s and $x_2$'s.

In [2]:
mu1 = 5
mu2 = 10
sigma1 = 17
sigma2 = 3

x1 = np.random.normal(loc=mu1, scale=sigma1, size = 10000)
x2 = np.random.normal(loc=mu2, scale=sigma2, size = 10000)
x = vstack((x1, x2))
In [3]:
plt.scatter(x1, x2)
Out[3]:
<matplotlib.collections.PathCollection at 0x7f6af444bf50>

What if we wanted for the $x_1$ and $x_2$ coordinates to be correlated? I.e., we want the $x_2$ to be larger if $x_1$ is larger, on average. That would make the cloud be tilted.

In [4]:
mu = [5, 10]

Sigma = array([[ 289,   30.    ],
               [  30.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
In [5]:
plt.scatter(x1, x2)
Out[5]:
<matplotlib.collections.PathCollection at 0x7f6af234bd10>

A higher covariance between $x_1$ and $x_2$ means more tilt and a "thinner" cloud.

In [6]:
mu = [5, 10]

Sigma = array([[ 289,   45.    ],
               [  45.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
plt.scatter(x1, x2)
Out[6]:
<matplotlib.collections.PathCollection at 0x7f6af22f8b90>

If the covariance is high enough, the cloud is very thin, since a higher $x_1$ always implies a higher $x_2$:

In [7]:
mu = [5, 10]

Sigma = array([[ 289,   51.    ],
               [  51.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
plt.scatter(x1, x2)
Out[7]:
<matplotlib.collections.PathCollection at 0x7f6af2227c10>

(Optional aside: we got 51 by using the fact that $-1 \leq \rho(x_1, x_2) = \frac{Cov(x_1, x_2)}{\sqrt{Var(x_1)Var(x_2)}} \leq 1$.

Diagonal Covariances

Of course, we can set $Cov(x_1, x_2)$ to 0, in which case we'll get an "axis-aligned" cloud:

In [8]:
mu = [5, 10]

Sigma = array([[ 289,   0.    ],
               [  0.,   9.    ]])
x1, x2 = np.random.multivariate_normal(mu, Sigma, 10000).T
plt.scatter(x1, x2)
Out[8]:
<matplotlib.collections.PathCollection at 0x7f6af2153c90>

Mixtures of Gaussians

Let's now generate some Mixtures of Gaussians in 2d.

In [9]:
pi = array([.2, .3, .5])
mu = [[10, 5], [-10, -5], [-10, 5]]
sigma = [ [[9, 5], 
           [5, 9]],
         
         
          [[5, 0], 
           [0, 5]],
         
         
          [[9, -5], 
           [-5, 9]]
        ]

N = 10000
Ns = (N*pi).astype(int)
colors = ["r", "g", "b"]
for i in range(len(pi)):
    x1, x2 = np.random.multivariate_normal(mu[i], sigma[i], Ns[i]).T
    plt.scatter(x1, x2, color=colors[i])

Mixtures of Gaussians in 3d

Finally, let's plot a Mixture of Gaussians in 3d. For simplicity, let's make the clouds axis-aligned.

In [10]:
pi = array([.2, .3, .5])
mu = [[10, 5, 0], [-10, -5, 0], [-10, 5, 3]]
sigma = [ [[9, 0, 0], 
           [0, 9, 0],
           [0, 0, 9]],
         
          [[9, 0, 0], 
           [0, 9, 0],
           [0, 0, 9]],         
         
          [[9, 0, 0], 
           [0, 9, 0],
           [0, 0, 9]],        ]

N = 1000
Ns = (N*pi).astype(int)
colors = ["r", "g", "b"]

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

for i in range(len(pi)):
    x1, x2, x3 = np.random.multivariate_normal(mu[i], sigma[i], Ns[i]).T
    ax.scatter(x1, x2, x3, c=colors[i])