Notebook

Lecture 18: MGFs to get moments of Expo and Normal, sums of Poissons, joint distributions¶

Stat 110, Prof. Joe Blitzstein, Harvard University¶

MGF for $\operatorname{Expo}(1)$ ¶

Let $X \sim \operatorname{Expo}(1)$ .

We begin by finding the MGF of $X$ :

$\begin{align} M(t) &= \mathbb{E}(e^{tX}) &\quad \text{ definition of MGF} \\ &= \int_{0}^{\infty} e^{-x} \, e^{tx} dx \\ &= \int_{0}^{\infty} e^{-x(1-t)} dx \\ &= \boxed{\frac{1}{1-t}} &\quad \text{ for } t < 1 \end{align}$

In finding the moments, by definition we have:

$M'(0) = \mathbb{E}(X)$
$M''(0) = \mathbb{E}(X^2)$
$M'''(0) = \mathbb{E}(X^3)$
... and so on ...

Even though finding derivatives of $\frac{1}{1-t}$ is not all that bad, it is nevertheless annoying busywork. But since we know that the $n^{th}$ moment is the coefficient of the $n^{th}$ term of the Taylor Expansion of $X$ , we can leverage that fact instead.

$\begin{align} \frac{1}{1-t} &= \sum_{n=0}^{\infty} t^n &\quad \text{ for } |t| < 1 \\ &= \sum_{n=0}^{\infty} \frac{n! \, t^n}{n!} &\quad \text{ since we need the form } \sum_{n=0}^{\infty} \left( \frac{\mathbb{E}(X^{n}) \, t^{n}}{n!}\right) \\ \\ \Rightarrow \mathbb{E}(X^n) &= \boxed{n!} \end{align}$

And now we can simply generate arbitary moments for r.v. $X$ !

$\mathbb{E}(X) = 1! = 1$
$\mathbb{E}(X^2) = 2! = 2$
$\Rightarrow \operatorname{Var}(X) = \mathbb{E}(X^2) - \mathbb{E}X^2 = 2 - 1 = 1$

MGF for $\operatorname{Expo}(\lambda)$ ¶

Let $Y \sim \operatorname{Expo}(\lambda)$ .

We begin with

$\begin{align} \text{let } X &= \lambda Y \\ \text{and so } X &= \lambda Y \sim \operatorname{Expo}(1) \\ \\ \text{then } Y &= \frac{X}{\lambda} \\ Y^n &= \frac{X^n}{\lambda^n} \\ \\ \Rightarrow \mathbb{E}(Y^n) &= \frac{\mathbb{E}(X^n)}{\lambda^n} \\ &= \boxed{\frac{n!}{\lambda^n}} \end{align}$

And as before, we now can simply generate arbitary moments for r.v. $Y$ !

$\mathbb{E}(Y) = \frac{1!}{\lambda^1} = \frac{1}{\lambda}$
$\mathbb{E}(Y^2) = \frac{2!}{\lambda^2} = \frac{2}{\lambda^2}$
$\Rightarrow \operatorname{Var}(Y) = \mathbb{E}(Y^2) - \mathbb{E}Y^2 = \frac{2}{\lambda^2} - \left(\frac{1}{\lambda}\right)^2 = \frac{1}{\lambda^2}$

MGF for standard Normal $\mathcal{N}(0, 1)$ ¶

Let $Z \sim \mathcal{N}(0,1)$ ; find all its moments.

We have seen before that, by symmetry, $\mathbb{E}(Z^{2n+1}) = 0$ for all odd values of $n$ .

So we will focus in on the even values of $n$ .

Now the MGF $M(t) = e^{t^2/2}$ . Without taking any derivatives, we can immediately Taylor expand that, since it is continuous everywhere.

$\begin{align} M(t) &= e^{t^2/2} \\ &= \sum_{n=0}^{\infty} \frac{\left(t^2/2\right)^n}{n!} \\ &= \sum_{n=0}^{\infty} \frac{t^{2n}}{2^n \, n!} \\ &= \sum_{n=0}^{\infty} \frac{(2n)! \, t^{2n}}{2^n \, n! \, (2n)!} &\quad \text{ since we need the form } \sum_{n=0}^{\infty} \left( \frac{\mathbb{E}(X^{n}) \, t^{n}}{n!}\right) \\ \\ \Rightarrow \mathbb{E}(Z^{2n}) &= \boxed{\frac{(2n)!}{2^n \, n!}} \end{align}$

Let's double-check this with what know about $\mathbb{Var}(Z)$

by symmetry, we know that $\mathbb{E}(Z) = 0$
at $n = 1$ , $\mathbb{E}(Z^2) = \frac{2!}{2 \times 1!} = 1$
$\Rightarrow \operatorname{Var}(Z) = \mathbb{E}(Z^2) - \mathbb{E}Z^2 = 1 - 0 = 1$
at $n = 2$ , $\mathbb{E}(Z^4) = \frac{4!}{4 \times 2!} = 3$
at $n = 3$ , $\mathbb{E}(Z^6) = \frac{6!}{8 \times 3!} = 15$

And so you might have noticed a pattern here. Let us rewrite those even moments once more:

at $n = 1$ , $\mathbb{E}(Z^2) = 1$
at $n = 2$ , $\mathbb{E}(Z^4) = 1 \times 3 = 3$
at $n = 3$ , $\mathbb{E}(Z^6) = 1 \times 3 \times 5 = 15$

MGF for $\operatorname{Pois}(\lambda)$ ¶

Let $X \sim \operatorname{Pois}(\lambda)$ ; now let's consider MGFs and how to use them to find sums of random variables (convolutions).

$\begin{align} M(t) &= \mathbb{E}(e^{tX}) \\ &= \sum_{k=0}^{\infty} e^{tk} \, \frac{\lambda^k e^{-\lambda}}{k!} \\ &= e^{-\lambda} \sum_{k=0}^{\infty} \frac{\lambda^k e^{tk}}{k!} \\ &= e^{-\lambda} e^{\lambda e^t} &\quad \text{but the right is just another Taylor expansion} \\ &= \boxed{e^{\lambda (e^t - 1)}} \end{align}$

Now let's let $Y \sim \operatorname{Pois}(\mu)$ , and it is independent of $X$ . Find the distribution of $(X + Y)$ .

You may recall that with MGFs, all we need to do is multiply the MGFs.

$\begin{align} e^{\lambda (e^t - 1)} \, e^{\mu (e^t - 1)} &= e^{(\lambda + \mu)(e^t - 1)} \\ \\ \Rightarrow X + Y &\sim \mathcal{Pois}(\lambda + \mu) \end{align}$

When adding a Poisson distribution $X$ with another independent Poisson distribution $Y$ , the resulting convolution $X + Y$ will also be Poisson. This interesting relation only happens with Poisson distributions.

Now think about what happens with $X$ and $Y$ are not independent.

Let $Y = X$ , so that $X + Y = 2X$ , which is clearly not Poisson, as

$X + Y = 2X$ is an even function which cannot be Poisson, since Poisson can take on all values both even and odd
Mean $\mathbb{E}(2x) = 2\lambda$ , but $\operatorname{Var}(2X) = 4\lambda$ , and since the mean and variance are not equal, this cannot be Poisson

Some definitions¶

In the most basic case of two r.v. in a joint distribution, consider both r.v.'s together:

Joint CDF

In the general case, the joint CDF fof two r.v.'s is $F(x,y) = P(X \le x, Y \le y)$

Joint PDF $f(x, y)$ such that, in the continuous case $P((X,Y) \in B) = \iint_B f(x,y)\,dx\,dy$

Joint PMF¶

$f(x, y)$ such that, in the discrete case

$\begin{align} P(X=x, Y=y) \end{align}$

We also can consider a single r.v. of a joint distribution:

Independence and Joint Distributions¶

$X, Y$ are independent iff $F(x,y) = F_X(x) \, F_Y(y)$ .

$\begin{align} P(X=x, Y=y) &= P(X=x) \, P(Y=y) &\quad \text{discrete case} \\ \\\\ f(x, y) &= f_X(x) \, f_Y(y) &\quad \text{continuous case} \end{align}$

... with the caveat that this must be so for all $\text{x, y} \in \mathbb{R}$

Marginals (and how to get them)¶

$P(X \le x)$ is the marginal distribution of $X$ , where we consider one r.v. at a time.

In the case of a two-r.v. joint distribution, we can get the marginals by using the joint distribution itself:

$\begin{align} P(X=x) &= \sum_y P(X=x, Y=y) &\quad \text{marginal PMF, discrete case, for } x \\ \\\\ f_Y(y) &= \int_{-\infty}^{\infty} f_{(X,Y)}(x,y) \, dx &\quad \text{marginal PDF, continuous case, for } y \end{align}$

Example: Discrete Joint Distribution¶

Let $X, Y$ be both Bernoulli. $X$ and $Y$ may be independent; or they might be dependent. They may or may not have the same $p$ . But they are both related in the form of a joint distribution.

We can lay out this joint distribution in a $2 \times 2$ contigency table like below:

	$Y=0$	$Y=1$
$X=0$	2/6	1/6
$X=1$	2/6	1/6

In order to be a joint distribution, all of the values in our contigency table must be positive; and they must all sum up to 1. The example above shows such a PMF.

Let's add the marginals for $X$ and $Y$ to our $2 \times 2$ contigency table:

	$Y=0$	$Y=1$	...
$X=0$	2/6	1/6	3/6
$X=1$	2/6	1/6	3/6
...	4/6	2/6

Observe how in our example, we have:

$\begin{align} P(X=0,Y=0) &= P(X=0) \, P(Y=0) \\ &= 3/6 \times 4/6 = 12/36 &= \boxed{2/6} \\ \\ P(X=0,Y=1) &= P(X=0) \, P(Y=1) \\ &= 3/6 \times 2/6 = 6/36 &= \boxed{1/6} \\ P(X=1,Y=0) &= P(X=1) \, P(Y=0) \\ &= 3/6 \times 4/6 = 12/36 &= \boxed{2/6} \\ \\ P(X=1,Y=1) &= P(X=1) \, P(Y=1) \\ &= 3/6 \times 2/6 = 6/36 &= \boxed{1/6} \\ \end{align}$

and so you can see that $X$ and $Y$ are independent.

Now here's an example of a two r.v. joint distribution where $X$ and $Y$ are dependent; check it out for yourself.

	$Y=0$	$Y=1$
$X=0$	1/3	0
$X=1$	1/3	1/3

Example: Continuous Joint Distribution¶

Now say we had Uniform distributions on a square such that $x,y \in [0,1]$ .

The joint PDF would be constant on/within the square; and 0 outside.

$\begin{align} \text{joint PDF} &= \begin{cases} c &\quad \text{if } 0 \le x \le 1 \text{, } 0 \le y \le 1 \\ \\ 0 &\quad \text{otherwise} \end{cases} \end{align}$

title

In 1-dimension space, if you integrate $1$ over some interval you get the length of that interval.

In 2-dimension space, if you integrate $1$ over some region, you get the area of that region.

Normalizing $c$ , we know that $c = \frac{1}{area} = 1$ .

Marginally, $X$ and $Y$ are independent $\mathcal{Unif}(1)$ .

Example: Dependent, Continuous Joint Distribution¶

Now say we had Uniform distributions on a disc such that $x^2 + y^2 \le 1$ .

Joint PDF¶

In this case, the joint PDF is $\operatorname{Unif}$ over the area of a disc centered at the origin with radius 1.

$\begin{align} \text{joint PDF} &= \begin{cases} \frac{1}{\pi} &\quad \text{if } x^2 + y^2 \le 1 \\ \\ 0 &\quad \text{otherwise} \end{cases} \end{align}$

Marginal PDF¶