Probability inequalities

For distributions like the Normal, it is easy to see what the probability that a r.v. will be towards the “tail end” of the distribution via ${\sigma }^2$ and the 68-95-99.7 rule. However, how do we get an estimate on the probability for any arbitrary distribution?

There are two important inequalities that provide us upper bounds on the tail probabilities.

Markov’s inequality

Let $X$ be a r.v. with a finite expectation. For any constant $a>0$, we have

$$P(|X|\ge a)\le \frac{\mathbb{E}(|X|)}{a}$$

Chebyshev’s inequality

Suppose $X$ is a r.v. with $\mathbb{E}(X)=\mu$ and $\text{Var}(X)={\sigma }^2$. For any constant $a>0$, we have

$$P(|X-\mathbb{E}(X)|\ge a)\le \frac{\text{Var}(X)}{a^2}⟺P(|X-\mu |\ge a)\le \frac{{\sigma }^2}{a^2}$$

To prove, we first square both sides of $P(|X-\mu |\ge a)$, giving us $P([X-\mu {]}^2\ge a^2)$. These two probabilities are equivalent. Plugging this into Markov’s inequality, we get

$$P(|X-\mu |\ge a)=P([X-\mu {]}^2\ge a^2)\le \frac{\mathbb{E}([X-\mu {]}^2)}{a^2}=\frac{{\sigma }^2}{a^2}$$

Alternate form

Assume that ${\sigma }^2=\text{Var}(X)>0$. Then for any constant $c>0$, we have

$$P(|X-\mu |\ge c\sigma )\le \frac{1}{c^2}$$

This is useful because it sets up an inequality similar to the 68-95-99.7 rule. For example, we have

• $P(|X-\mu |\ge 2\sigma )\le \frac{1}{4}=0.25$
• $P(|X-\mu |\le 3\sigma )\le \frac{1}{9}\approx 0.11$

If $X\sim \mathcal{N}$, then we can see that both of the above are true: when $X$ is within 2 standard deviations, this means this encompasses 95% of the probability, so we are outside $2\sigma$ with a probability of 0.05. This is indeed less than $\frac{1}{4}$. The same for $3\sigma$.