Moment generating function

The moment generating function (MGF) of a r.v. $X$ is a function of $t$. It’s defined as $M:\mathbb{R}\to \mathbb{R}$.

$$M_X(t)=\mathbb{E}(e^{tX})$$

Note that whatever we choose for $t$ might cause the expectation to blow up (approach $\infty$). This is fine, as long as $\mathbb{E}(e^{tX})$ is finite for all $t$ in some open interval $(-a,a)$ around zero. Otherwise, the MGF of $X$ does not exist.

Here, $e^{tX}$ is a random variable that depends on $t$ and the original random variable $X$.

If the MGF exists, then when $t=0$ then $M_X(0)=\mathbb{E}(1)=1$. There’s no intuitive explanation for what $t$ really is. LOL.

Usefulness

There are three theorems involving MGFs that we’ll use.

1. If $M_X$ exists, then $\mathbb{E}(X^n)$ equals the $n$th derivative of the MGF, evaluated at zero. That is, we have $\mathbb{E}(X^n)=M^{(n)}(0)$ for $n=1,2,3,\dots$

For a not very rigorous proof, we can consider the Taylor series, replace the $x$ in $e^x$ with $e^{tX}$, and see that every term corresponds to a derivative of the moment.

Example

Suppose $X\sim \text{Expo}(1)$. We can then derive $M_X$, which turns out to be $M_X(t)=\frac{1}{1-t}=(1-t{)}^{-1}$.

Then, to find the Variance of $X$, we simply have

$$\text{Var}(X)=\mathbb{E}(X^2)-\mathbb{E}(X{)}^2=M_X^{\prime \prime }(0)-M_X^{\prime }(0)=2-1=1$$

Finding the derivative of the moment is much easier than using LOTUS.

2. If $X$ and $Y$ have the same MGF, then they must also have the same distribution.
3. If $X$ and $Y$ are Independent and both of their MGFs exist, then the MGF of $X+Y$ is the product of their MGFs. We have:

$$M_{X+Y}(t)=M_X(t)\cdot M_Y(t)$$

Location-scale transformation

If $X$ has a MGF of $M_X(t)$, then the MGF of $a+bX$ is given by

$$M_{a+bX}(t)=\mathbb{E}(e^{t(a+bX)})=e^{at}\cdot \mathbb{E}(e^{btX})=e^{at}\cdot M_X(bt)$$

Since $b$ is a constant, we can basically pretend that we’re transforming $t$. If we let $u=bt$, then the above is the same as $M_X(u)=\mathbb{E}(e^{uX})$, which is the definition of the MGF.

This can be useful for finding the MGF of a general continuous distribution after we derive the MGF of the standard version of the distribution (e.g. uniform, exponential, Normal).

Solving the MGF for a distribution

When we finish deriving a MGF for a distribution, we can case on what values $t$ can take on, and see where we define our neighborhood $(-a,a)$.

• We can pick the cutoff $a$ so that $M_X$ is well defined.
• Remember, as long as such an interval exists, then the MGF of $X$ can exist.

Discrete variable

To solve the MGF, begin with the definition of expectation. We can use LOTUS because $e^{tX}$ is really a transformation of $X$, with $g(X)=e^{tX}$.

$$M_X(t)=\mathbb{E}(e^{tX})=\sum _{x\in \text{Support}(X)}{e}^{tx}P(X=x)$$

where $\text{Support(X)}$ denotes us iterating over all the possible values of the random variable $X$. We substitute $P(X=x)$ with whatever relevant PMF.

Continuous variable

Instead of a summation, we use an integral instead over $(-\infty ,\infty )$. Remember that the PDF is often nonzero outside of a specific range, so the integral simplifies. We’re also using LOTUS here as well.

$$M_X(t)=\mathbb{E}(e^{tX})={\int }_{-\infty }^{\infty }{e}^{tx}f(x)dx$$