Joint distribution

Sometimes we want to find the probability of events that involve two or more Random variables. For example, finding $P(H>65,L<55)$, where $H$ and $L$ represent tomorrow’s high and low temp, respectively.

Also, while we’ve discussed ways of summarizing a distribution of a single r.v. (with Expectation and Variance), we would also like a way to explain the relationship between two r.v.s.

Discrete variables

There are three kinds of PMFs that we can observe between two discrete random variables. Given two discrete $X$ and $Y$, we have:

• The joint PMF is given by the function $p_{X,Y}$ and equals $p_{X,Y}(x,y)=P(X=x,Y=y)$. Three or more variables follow the same pattern.
• WLOG, the marginal PMF of $X$ is the function $p_X$ given by

$$p_X(x)=P(X=x)=\sum _{\text{Support}(Y)}P(X=x,Y=y)$$

Note that this is just the original PMF that we’ve learned before. Here, we are summing up across the whole support of $Y$. The only new part is that $P(x=x)$ can be defined in terms of the joint PMF.

• WLOG, for each value $x\in \text{Support}(X)$, the conditional PMF of $Y$, given $X=x$, is simply the equation for Conditional probability.

$$P(Y=y\mid X=x)=\frac{P(X=x,Y=y)}{P(X=x)}$$

Here, we’re holding the value of $x$ constant. Depending on $X=x$, we will get a different conditional probability for $Y$.

Continuous variables

Motivation

Remember that in PDFs, the probability that a variable is exactly one value is zero. But, over a range, the probability is positive.

Consider two continuous r.v.s, $X$ and $Y$. If we were to draw a plane, with $X$ on the horizontal axis and $Y$ on the vertical, something similar occurs.

• Any point, line, or curve in the plane has zero probability.
• However, regions of the plane (e.g. $P(63\le X\le 65,47\le Y\le 49)$) will have positive probability.

Then, the probability of an event involving both $X$ and $Y$ will be the volume, which we will calculate with a double integral over the relevant region.

Given two continuous r.v.s $X$ and $Y$, we can define the following.

Joint CDF

This is a function $F_{X,Y}(x,y)$ given by

$$F_{X,Y}(x,y)=P(X\le x,Y\le y)$$

It is very similar to a regular CDF for one variable. Three or more variables follow the same pattern. Like regular CDFs, this definition also holds for discrete r.v.s.

Joint PDF

The joint PDF is the derivative of their joint CDF with respect to $x$ and $y$.

$$f_{X,Y}(x,y)=\frac{{\partial }^2}{\partial x\partial y}{F}_{X,Y}(x,y)$$

Valid joint PDFs are nonnegative for all $x,y$ and integrates to $1$ over $x,y\in (-\infty ,\infty )$, which makes sense, given that it is the total volume under the PDF.

For any set $A\subseteq {\mathbb{R}}^2$, we have

$$P((X,Y)\in A)={\iint }_A{f}_{X,Y}(x,y)dxdy$$

which just means that given $a\le X\le b$ and $c\le Y\le d$, we integrate over ${\int }_a^b{\int }_c^d$.

Bound definitions

Be careful with how the bounds are defined for $x$ and $y$. $0<x<1,0<y<1$ is different from $0<x<y<1$ and also changes the bounds for the integration. The latter would have the double integral

$${\int }_0^1{\int }_x^{1}dydx$$

This is because $y$ always has to be greater than $x$, so its lower bound has to start where $x$ ends.

Marginal PDF

Just like with the marginal PMF, WLOG we can get the marginal PDF of $X$ by integrating over the entire support of $Y$, i.e. $(-\infty ,\infty )$. This is just the regular PDF for $X$.

$$f_X(x)={\int }_{-\infty }^{\infty }{f}_{X,Y}(x,y)dy$$

Conditional PDF

WLOG, for each value of $x\in \text{Support}(X)$, the conditional PDF of $Y$, given $X=x$, is

$$f_{Y\mid X}(y\mid x)=\frac{f_{X,Y}(x,y)}{f_X(x)}$$

Note that $P(X=x)=0$, so we’re actually calculating the region defined by $\left(x-\frac{\Delta }{2},x+\frac{\Delta }{2}\right)$ and taking the limit as $\Delta \to 0$.

Factoring

If the joint PDF of $X$ and $Y$ can be factored as $f_{X,Y}(x,y)=g(x)\cdot h(y)$ for all $x,y\in \mathbb{R}$, where $g$ and $h$ are nonnegative functions, then $X$ and $Y$ are Independent.

• Useful when we don’t know their marginal PDFs, or if $g$ and $h$ are the marginal PDFs.
• Then, to get their marginal PDFs, we simply “rescale” $g$ and $h$ so they both integrate to 1.