Download Cheat Sheet for Probability Theory and more Cheat Sheet Probability and Statistics in PDF only on Docsity!
Cheat Sheet for Probability Theory
Ingmar Land
1 Scalar-valued Random Variables
Consider two real-valued random variables (RV) X and Y with the individual probabil-
ity distributions pX (x) and pY (y), and the joint distribution pX,Y (x, y). The probability
distributions are probability mass functions (pmf) if the random variables take discrete
values, and they are probability density functions (ptf) if the random variables are con-
tinuous. Some authors use f () instead of p(), especially for continuous RVs.
In the following, the RVs are assumed to be continuous. (For discrete RVs, the integrals
have simply to be replaced by sums.)
pX (x) =
pX,Y (x, y) d y pY (y) =
pX,Y (x, y) d x
- Conditional distributions:
pX|Y (x|y) =
pX,Y (x, y)
pY (y)
pY |X (y|x) =
pX,Y (x, y)
pX (x)
for pX (x) 6 = 0 and pY (y) 6 = 0
pX|Y (x|y) =
pX,Y (x, y) ∫ pX,Y (x′, y) d x′^
pY |X (y|x) =
pX,Y (x, y) ∫ pX,Y (x′, y) d y′
- Expected values (expectations):
E
[
g 1 (X)
]
g 1 (x) pX (x) d x
E
[
g 2 (Y )
]
g 2 (y) pY (y) d y
E
[
g 3 (X, Y )
]
g 3 (x, y) pX,Y (x, y) d x d y
for any functions g 1 (.), g 2 (.), g 3 (., .)
- Some special expected values:
μX := E
[
X
]
x pX (x) d x μY := E
[
Y
]
y pY (y) d y
σ 2 X ≡^ ΣXX^ := E
[
(X − μX ) 2
]
(x − μX ) 2 pX (x) d x
σ 2 Y ≡^ ΣY Y^ := E
[
(Y − μY ) 2
]
(y − μY ) 2 pY (y) d y
Remark: The variance measures the “width” of a distribution. A small variance means that most of the probability mass is concentrated around the mean value.
σXY ≡ ΣXY := E
[
(X − μX )(Y − μY )
]
(x − μX )(y − μY ) pX,Y (x, y) d x d y
Remark: The covariance measures how “related” two RVs are. Two indepen- dent RVs have covariance zero.
ρXY :=
σXY
σX σY
E
[
X
2
]
= ΣXX + μ 2 X
E
[
Y
2
]
= ΣY Y + μ 2 Y E
[
X · Y
]
= ΣXY + μX · μY
E
[
XY
]
= E
[
((X − μX ) + μX )((Y − μY ) + μY )
]
= E
[
(X − μX )(Y − μY )
]
− E
[
(X − μX )μY
]
− E
[
μX (Y − μY )
]
+ E
[
μX μY
]
= ΣXY − (E[X] − μX )μY − (E[Y ] − μY )μX + μX μY
= ΣXY + μX μY
This method of proof is typical.
- The RVs are called uncorrelated if
σXY ≡ ΣXY = E
[
(X − μX )(Y − μY )
]
Remark: If RVs are independent, they are also uncorrelated. The reverse holds only for Gaussian RVs (see below).
- Two RVs X and Y are called orthogonal if E[XY ] = 0.
Remark: The RVs with finite energy, E[X^2 ] < ∞, form a vector space with scalar product 〈X, Y 〉 = E[XY ] and norm ‖X‖ =
E[X^2 ]. (This is used in MMSE estimation.)
These relations for scalar-valued RVs are generalized to vector-valued RVs in the
following.
2 Vector-valued Random Variables
Consider two real-valued vector-valued random variables (RV)
X =
[
X 1
X 2
]
, Y =
[
Y 1
Y 2
]
with the individual probability distributions pX (x) and pY (y), and the joint distribution
pX,Y (x, y). (The following considerations can be generalized to longer vectors, of course.)
The probability distributions are probability mass functions (pmf) if the random vari-
ables take discrete values, and they are probability density functions (pmf) if the random
variables are continuous. Some authors use f () instead of p(), especially for continuous
RVs.
In the following, the RVs are assumed to be continuous. (For discrete RVs, the integrals
have simply to be replaced by sums.)
Remark: The following matrix notations may seem to be cumbersome at the first
glance, but they turn out to be quite handy and convenient (once you got used to).
- Marginal distributions, conditional distributions, Bayes’ rule, expected values work as in the scalar case.
- Some special expected values:
- Mean vector (vector of mean values):
μ X
:= E
[
X
]
= E
[
X 1
X 2
]
[
E[X 1 ]
E[X 2 ]
]
[
μX 1 μX 2
]
- Covariance matrix (auto-covariance matrix):
ΣXX := E
[
(X − μ X )(X − μ X
T
]
= E
[[
X 1 − μ 1 X 2 − μ 2
]
[
X 1 − μ 1 X 2 − μ 2
]
]
[
E
[
(X 1 − μ 1 )(X 1 − μ 1 )
]
E
[
(X 1 − μ 1 )(X 2 − μ 2 )
]
E
[
(X 2 − μ 2 )(X 1 − μ 1 )
]
E
[
(X 2 − μ 2 )(X 2 − μ 2 )
]
]
[
ΣX 1 X 1 ΣX 1 X 2
ΣX 2 X 1 ΣX 2 X 2
]
- Covariance matrix (cross-covariance matrix):
ΣXY := E
[
(X − μ X )(Y − μ Y
T
]
[
ΣX 1 Y 1 ΣX 1 Y 2
ΣX 2 Y 1 ΣX 2 Y 2
]
Remark: This matrix contains the covariance of each element of the first vector with each element of the second vector.
E
[
XX
T
]
= ΣXX + μ X μ T X
E
[
XY
T]^
= ΣXY + μ X
μT Y
Remark: This result is not too surprising when you know the result for the scalar case.
3 Gaussian Random Variables
- A Gaussian RV X with mean μX and variance σ^2 X is a continuous random variable with a Gaussian pdf, i.e., with
pX (x) =
2 πσ X^2
· exp
(x − μX )^2
2 σ X^2
The often used symbolic notation
X ∼ N (μX , σ 2 X )
may be read as: X is (distributed) Gaussian with mean μX and variance σ X^2.
- A Gaussian distribution with mean zero and variance one is called a normal distri- bution:
p(x) =
2 π
e − x
2 (^2).
