Law of large numbers

Given a Sample mean ${\overline{X}}_n$, how close is ${\overline{X}}_n$ to the real mean $\mu$?

It turns out that as $n$ increases, ${\overline{X}}_n$ will increasingly get closer to $\mu$, and the distribution of ${\overline{X}}_n$ gets closer to Normal.

Weak law of large numbers (WLLN)

Given a sample mean ${\overline{X}}_n$ with finite Expectation $\mu$, for some $ϵ>0$, we have

$$\underset{n\to \infty }{\lim }P(|{\overline{X}}_n-\mu |<ϵ)=1$$

The probability that ${\overline{X}}_n$ gets close to $\mu$ goes to 100% when $n$ approaches infinity, no matter what definition of $ϵ$ we choose, whether 0.00001 or 1000.

• We say that ${\overline{X}}_n$ converges in probability to $\mu$.
• WLLN holds for both finite and infinite variance (but is much harder to prove for the latter).

Proof of correctness

From Finding the variance, we know that $\text{Var}({\overline{X}}_n)=\frac{{\sigma }^2}{n}$, where ${\sigma }^2=\text{Var}(X_i)$ for any $i\in [\mathrm{1..}n]$. So $\text{SD}({\overline{X}}_n)=\sqrt{\text{Var}({\overline{X}}_n)}=\frac{\sigma }{\sqrt{n}}$.

Given the derivations above and a constant $ϵ>0$, Chebyshev’s inequality tells us

$$0\le P(|{\overline{X}}_n-\mu |\ge ϵ)\le \frac{{\sigma }^2/n}{{ϵ}^2}=\frac{{\sigma }^2}{n{ϵ}^2}$$

The $0\le$ is not part of the original inequality, but it must be true because probabilities are not negative.

We can now analyze this:

• As $n\to \infty$, the denominator goes to infinity and the overall probability goes to zero. This is the probability that $|{\overline{X}}_n-\mu |$ is greater than $ϵ$.
• This means that $P(|{\overline{X}}_n-\mu |<ϵ)$ approaches a probability of 100%.

Strong law of numbers (SLLN)

An even stronger claim can be made about WLLN. In particular, the following claim is true:

$$P(\underset{n\to \infty }{\lim }{\overline{X}}_n=\mu )=1$$

which completely eliminates the need for an $ϵ$ and directly states that as the sample size approaches infinity, the sample mean will equal the population mean with 100% certainty.

• We say that ${\overline{X}}_n$ converges almost surely (with probability 1) to $\mu$.