Download Complete Probability Cheatsheet and more Cheat Sheet Probability and Statistics in PDF only on Docsity!
Probability Cheatsheet v1.1.
Counting
Multiplication Rule - Let’s say we have a compound experiment (an experiment with multiple components). If the 1st component has n 1 possible outcomes, the 2nd component has n 2 possible outcomes, and the rth component has nr possible outcomes, then overall there are n 1 n 2... nr possibilities for the whole experiment.
Sampling Table - The sampling tables describes the different ways to take a sample of size k out of a population of size n. The column names denote whether order matters or not.
Matters Not Matter
With Replacement n k (n + k − 1
k
Without Replacement
n! (n − k)!
(n
k
Na¨ıve Definition of Probability - If the likelihood of each outcome is equal, the probability of any event happening is:
P (Event) =
number of favorable outcomes number of outcomes
Probability and Thinking Conditionally
Independence
Independent Events - A and B are independent if knowing one gives you no information about the other. A and B are independent if and only if one of the following equivalent statements hold:
P (A ∩ B) = P (A)P (B) P (A|B) = P (A)
Conditional Independence - A and B are conditionally independent given C if: P (A ∩ B|C) = P (A|C)P (B|C). Conditional independence does not imply independence, and independence does not imply conditional independence.
Unions, Intersections, and Complements
De Morgan’s Laws - Gives a useful relation that can make calculating probabilities of unions easier by relating them to intersections, and vice versa. De Morgan’s Law says that the complement is distributive as long as you flip the sign in the middle.
(A ∪ B)c^ ≡ Ac^ ∩ Bc
(A ∩ B) c ≡ A c ∪ B c
Joint, Marginal, and Conditional Probabilities
Joint Probability - P (A ∩ B) or P (A, B) - Probability of A and B.
Marginal (Unconditional) Probability - P (A) - Probability of A
Conditional Probability - P (A|B) - Probability of A given B occurred.
Conditional Probability is Probability - P (A|B) is a probability as well, restricting the sample space to B instead of Ω. Any theorem that holds for probability also holds for conditional probability.
Simpson’s Paradox
P (A | B, C) < P (A | B
c , C) and P (A | B, C c ) < P (A | B c , C c ) yet still, P (A | B) > P (A | B c )
Bayes’ Rule and Law of Total Probability
Law of Total Probability with partitioning set B 1 , B 2 , B 3 , ...Bn and with extra conditioning (just add C!)
P (A) = P (A|B 1 )P (B 1 ) + P (A|B 2 )P (B 2 ) + ...P (A|Bn)P (Bn) P (A) = P (A ∩ B 1 ) + P (A ∩ B 2 ) + ...P (A ∩ Bn) P (A|C) = P (A|B 1 , C)P (B 1 |C) + ...P (A|Bn, C)P (Bn|C) P (A|C) = P (A ∩ B 1 |C) + P (A ∩ B 2 |C) + ...P (A ∩ Bn|C)
Law of Total Probability with B and Bc^ (special case of a partitioning set), and with extra conditioning (just add C!)
P (A) = P (A|B)P (B) + P (A|Bc)P (Bc) P (A) = P (A ∩ B) + P (A ∩ Bc)
P (A|C) = P (A|B, C)P (B|C) + P (A|B c , C)P (B c |C) P (A|C) = P (A ∩ B|C) + P (A ∩ B c |C)
Bayes’ Rule, and with extra conditioning (just add C!)
P (A|B) =
P (A ∩ B)
P (B)
P (B|A)P (A)
P (B)
P (A|B, C) =
P (A ∩ B|C)
P (B|C)
P (B|A, C)P (A|C)
P (B|C)
Odds Form of Bayes’ Rule, and with extra conditioning (just add C!)
P (A|B) P (Ac|B)
P (B|A)
P (B|Ac)
P (A)
P (Ac)
P (A|B, C) P (Ac|B, C)
P (B|A, C)
P (B|Ac, C)
P (A|C)
P (Ac|C)
Random Variables and their Distributions
PMF, CDF, and Independence
Probability Mass Function (PMF) (Discrete Only) gives the probability that a random variable takes on the value X.
PX (x) = P (X = x)
Cumulative Distribution Function (CDF) gives the probability that a random variable takes on the value x or less
FX (x 0 ) = P (X ≤ x 0 )
Independence - Intuitively, two random variables are independent if knowing one gives you no information about the other. X and Y are independent if for ALL values of x and y:
P (X = x, Y = y) = P (X = x)P (Y = y)
Expected Value and Indicators
Distributions
Probability Mass Function (PMF) (Discrete Only) is a function that takes in the value x, and gives the probability that a random variable takes on the value x. The PMF is a positive-valued function, and
x P^ (X^ =^ x) = 1 PX (x) = P (X = x)
Cumulative Distribution Function (CDF) is a function that takes in the value x, and gives the probability that a random variable takes on the value at most x.
F (x) = P (X ≤ x)
Expected Value, Linearity, and Symmetry
Expected Value (aka mean, expectation, or average) can be thought of as the “weighted average” of the possible outcomes of our random variable. Mathematically, if x 1 , x 2 , x 3 ,... are all of the possible values that X can take, the expected value of X can be calculated as follows: E(X) =
i
xiP (X = xi)
Note that for any X and Y , a and b scaling coefficients and c is our constant, the following property of Linearity of Expectation holds:
E(aX + bY + c) = aE(X) + bE(Y ) + c If two Random Variables have the same distribution, even when they are dependent by the property of Symmetry their expected values are equal. Conditional Expected Value is calculated like expectation, only conditioned on any event A. E(X|A) =
x
xP (X = x|A)
Indicator Random Variables
Indicator Random Variables is random variable that takes on either 1 or 0. The indicator is always an indicator of some event. If the event occurs, the indicator is 1, otherwise it is 0. They are useful for many problems that involve counting and expected value. Distribution IA ∼ Bern(p) where p = P (A) Fundamental Bridge The expectation of an indicator for A is the probability of the event. E(IA) = P (A). Notation:
IA =
1 A occurs 0 A does not occur
Variance
Var(X) = E(X^2 ) − [E(X)]^2
Expectation and Independence
If X and Y are independent, then E(XY ) = E(X)E(Y )
Continuous RVs, LotUS, and UoU
Continuous Random Variables
What’s the prob that a CRV is in an interval? Use the CDF (or the PDF, see below). To find the probability that a CRV takes on a value in the interval [a, b], subtract the respective CDFs. P (a ≤ X ≤ b) = P (X ≤ b) − P (X ≤ a) = F (b) − F (a) Note that for an r.v. with a normal distribution, P (a ≤ X ≤ b) = P (X ≤ b) − P (X ≤ a)
b − μ σ^2
a − μ σ^2
What is the Cumulative Density Function (CDF)? It is the following function of x. F (x) = P (X ≤ x) What is the Probability Density Function (PDF)? The PDF, f (x), is the derivative of the CDF. F ′ (x) = f (x) Or alternatively, F (x) =
∫ (^) x
−∞
f (t)dt
Note that by the fundamental theorem of calculus,
F (b) − F (a) =
∫ (^) b
a
f (x)dx
Thus to find the probability that a CRV takes on a value in an interval, you can integrate the PDF, thus finding the area under the density curve.
How do I find the expected value of a CRV? Where in discrete cases you sum over the probabilities, in continuous cases you integrate over the densities.
E(X) =
−∞
xf (x)dx
Law of the Unconscious Statistician (LotUS)
Expected Value of Function of RV Normally, you would find the expected value of X this way:
E(X) = ΣxxP (X = x)
E(X) =
−∞
xf (x)dx
LotUS states that you can find the expected value of a function of a random variable g(X) this way:
E(g(X)) = Σxg(x)P (X = x)
E(g(X)) =
−∞
g(x)f (x)dx
What’s a function of a random variable? A function of a random variable is also a random variable. For example, if X is the number of bikes you see in an hour, then g(X) = 2X could be the number of bike wheels you see in an hour. Both are random variables.
What’s the point? You don’t need to know the PDF/PMF of g(X) to find its expected value. All you need is the PDF/PMF of X.
Universality of Uniform
When you plug any random variable into its own CDF, you get a Uniform[0,1] random variable. When you put a Uniform[0,1] into an inverse CDF, you get the corresponding random variable. For example, let’s say that a random variable X has a CDF
F (x) = 1 − e−x
By the Universality of the the Uniform, if we plug in X into this function then we get a uniformly distributed random variable.
F (X) = 1 − e −X ∼ U
Similarly, since F (X) ∼ U then X ∼ F −^1 (U ). The key point is that for any continuous random variable X, we can transform it into a uniform random variable and back by using its CDF.
Moment Generating Functions (MGFs)
Moments
Moments describe the shape of a distribution. The kth^ moment of a random variable X is μ ′ k =^ E(X
k )
The mean, variance, and skewness of a distribution can be expressed by its moments. Specifically:
Mean E(X) = μ′ 1
Variance Var(X) = E(X^2 ) − E(X)^2 = μ′ 2 − (μ′ 1 )^2
Moment Generating Functions
MGF For any random variable X, this expected value and function of dummy variable t;
MX (t) = E(e tX )
is the moment generating function (MGF) of X if it exists for a finitely-sized interval centered around 0. Note that the MGF is just a function of a dummy variable t.
Why is it called the Moment Generating Function? Because the kth^ derivative of the moment generating function evaluated 0 is the kth^ moment of X! μ′ k = E(Xk^ ) = M ( Xk )(0)
This is true by Taylor Expansion of etX
MX (t) = E(etX^ ) =
∑^ ∞
k=
E(Xk^ )tk k!
∑^ ∞
k=
μ′ k tk k!
Or by differentiation under the integral sign and then plugging in t = 0
M
(k) X (t) =^
dk dtk^
E(etX^ ) = E
dk dtk^
etX
= E(Xk^ etX^ )
M
(k) X (0) =^ E(X
k (^) e 0 X (^) ) = E(Xk (^) ) = μ′ k MGF of linear combinations If we have Y = aX + c, then
MY (t) = E(et(aX+c)) = ectE(e(at)X^ ) = ectMX (at)
Uniqueness of the MGF. If it exists, the MGF uniquely defines the distribution. This means that for any two random variables X and Y , they are distributed the same (their CDFs/PDFs are equal) if and only if their MGF’s are equal. You can’t have different PDFs when you have two random variables that have the same MGF. Summing Independent R.V.s by Multiplying MGFs. If X and Y are independent, then
M(X+Y )(t) = E(e t(X+Y ) ) = E(e tX )E(e tY ) = MX (t) · MY (t) M(X+Y )(t) = MX (t) · MY (t) The MGF of the sum of two random variables is the product of the MGFs of those two random variables.
Joint PDFs and CDFs
Joint Distributions
Review: Joint Probability of events A and B: P (A ∩ B) Both the Joint PMF and Joint PDF must be non-negative and sum/integrate to 1. (
x
y P^ (X^ =^ x, Y^ =^ y) = 1) (
x
y fX,Y^ (x, y) = 1). Like in the univariate cause, you sum/integrate the PMF/PDF to get the CDF.
Conditional Distributions
Review: By Baye’s Rule, P (A|B) = P (B|A)P (A) P (B) Similar conditions apply to conditional distributions of random variables. For discrete random variables:
P (Y = y|X = x) =
P (X = x, Y = y) P (X = x)
P (X = x|Y = y)P (Y = y) P (X = x) For continuous random variables:
fY |X (y|x) =
fX,Y (x, y) fX (x)
fX|Y (x|y)fY (y) fX (x) Hybrid Bayes’ Rule
f (x|A) =
P (A|X = x)f (x) P (A)
Marginal Distributions
Review: Law of Total Probability Says for an event A and partition B 1 , B 2 , ...Bn: P (A) =
i P^ (A^ ∩^ Bi) To find the distribution of one (or more) random variables from a joint distribution, sum or integrate over the irrelevant random variables. Getting the Marginal PMF from the Joint PMF
P (X = x) =
y
P (X = x, Y = y)
Getting the Marginal PDF from the Joint PDF
fX (x) =
y
fX,Y (x, y)dy
Independence of Random Variables
Review: A and B are independent if and only if either P (A ∩ B) = P (A)P (B) or P (A|B) = P (A). Similar conditions apply to determine whether random variables are independent - two random variables are independent if their joint distribution function is simply the product of their marginal distributions, or that the a conditional distribution of is the same as its marginal distribution. In words, random variables X and Y are independent for all x, y, if and only if one of the following hold:
- Joint PMF/PDF/CDFs are the product of the Marginal PMF
- Conditional distribution of X given Y is the same as the marginal distribution of X
Multivariate LotUS
Review: E(g(X)) =
x g(x)P^ (X^ =^ x), or E(g(X)) =
−∞ g(x)fX^ (x)dx For discrete random variables: E(g(X, Y )) =
x
y
g(x, y)P (X = x, Y = y)
For continuous random variables:
E(g(X, Y )) =
−∞
−∞
g(x, y)fX,Y (x, y)dxdy
Covariance and Transformations
Covariance and Correlation
Covariance is the two-random-variable equivalent of Variance, defined by the following: Cov(X, Y ) = E[(X − E(X))(Y − E(Y ))] = E(XY ) − E(X)E(Y ) Note that Cov(X, X) = E(XX) − E(X)E(X) = Var(X)
Correlation is a rescaled variant of Covariance that is always between -1 and 1.
Corr(X, Y ) =
Cov(X, Y ) √ Var(X)Var(Y )
Cov(X, Y ) σX σY
Covariance and Indepedence - If two random variables are independent, then they are uncorrelated. The inverse is not necessarily true. X ⊥⊥ Y −→ Cov(X, Y ) = 0 X ⊥⊥ Y −→ E(XY ) = E(X)E(Y )
Covariance and Variance - Note that Var(X + Y ) = Var(X) + Var(Y ) + 2Cov(X, Y )
Var(X 1 + X 2 + · · · + Xn) =
∑^ n
i=
Var(Xi) + 2
i<j
Cov(Xi, Xj )
In particular, if X and Y are independent then they have covariance 0 thus X ⊥⊥ Y =⇒ Var(X + Y ) = Var(X) + Var(Y ) In particular, If X 1 , X 2 ,... , Xn are identically distributed have the same covariance relationships, then
Var(X 1 + X 2 + · · · + Xn) = nVar(X 1 ) + 2
(n
2
Cov(X 1 , X 2 )
Covariance and Linearity - For random variables W, X, Y, Z and constants a, b:
Cov(X, Y ) = Cov(Y, X) Cov(X + a, Y + b) = Cov(X, Y ) Cov(aX, bY ) = abCov(X, Y ) Cov(W + X, Y + Z) = Cov(W, Y ) + Cov(W, Z) + Cov(X, Y )
Transition Matrix
Element qij in square transition matrix Q is the probability that the chain goes from state i to state j, or more formally:
qij = P (Xn+1 = j|Xn = i)
To find the probability that the chain goes from state i to state j in m steps, take the (i, j) th element of Q m .
q (m) ij =^ P^ (Xn+m^ =^ j|Xn^ =^ i)
If X 0 is distributed according to row-vector PMF ~p (e.g. pj = P (X 0 = ij )), then the PMF of Xn is pQ~ n.
Chain Properties
A chain is irreducible if you can get from anywhere to anywhere. An irreducible chain must have all of its states recurrent. A chain is periodic if any of its states are periodic, and is aperiodic if none of its states are periodic. In an irreducible chain, all states have the same period. A chain is reversible with respect to ~s if siqij = sj qji for all i, j. A reversible chain running on ~s is indistinguishable whether it is running forwards in time or backwards in time. Examples of reversible chains include random walks on undirected networks, or any chain with qij = qji, where the Markov chain would be stationary with respect to ~s = ( (^) M^1 , (^) M^1 ,... , (^) M^1 ). Reversibility Condition Implies Stationarity - If you have a PMF ~s on a Markov chain with transition matrix Q, then siqij = sj qji for all i, j implies that s is stationary.
Stationary Distribution
Let us say that the vector ~p = (p 1 , p 2 ,... , pM ) is a possible and valid PMF of where the Markov Chain is at at a certain time. We will call this vector the stationary distribution, ~s, if it satisfies ~sQ = ~s. As a consequence, if Xt has the stationary distribution, then all future Xt+1, Xt+2,... also has the stationary distribution. For irreducible, aperiodic chains, the stationary distribution exists, is unique, and si is the long-run probability of a chain being at state i. The expected number of steps to return back to i starting from i is 1 /si To solve for the stationary distribution, you can solve for (Q′^ − I)(~s)′^ = 0. The stationary distribution is uniform if the columns of Q sum to 1.
Random Walk on Undirected Network
If you have a certain number of nodes with edges between them, and a chain can pick any edge randomly and move to another node, then this is a random walk on an undirected network. The stationary distribution of this chain is proportional to the degree sequence. The degree sequence is the vector of the degrees of each node, defined as how many edges it has.
Continuous Distributions
Uniform
Let us say that U is distributed Unif(a, b). We know the following:
Properties of the Uniform For a uniform distribution, the probability of an draw from any interval on the uniform is proportion to the length of the uniform. The PDF of a Uniform is just a constant, so when you integrate over the PDF, you will get an area proportional to the length of the interval.
Example William throws darts really badly, so his darts are uniform over the whole room because they’re equally likely to appear anywhere. William’s darts have a uniform distribution on the surface of the room. The uniform is the only distribution where the probably of hitting in any specific region is proportion to the area/length/volume of that region, and where the density of occurrence in any one specific spot is constant throughout the whole support.
Normal
Let us say that X is distributed N (μ, σ^2 ). We know the following: Central Limit Theorem The Normal distribution is ubiquitous because of the central limit theorem, which states that averages of independent identically-distributed variables will approach a normal distribution regardless of the initial distribution. Transformable Every time we stretch or scale the normal distribution, we change it to another normal distribution. If we add c to a normally distributed random variable, then its mean increases additively by c. If we multiply a normally distributed random variable by c, then its variance increases multiplicatively by c^2. Note that for every normally distributed random variable X ∼ N (μ, σ^2 ), we can transform it to the standard N (0, 1) by the following transformation:
X − μ σ
∼ N (0, 1)
Example Heights are normal. Measurement error is normal. By the central limit theorem, the sampling average from a population is also normal. Standard Normal - The Standard Normal, denoted Z, is Z ∼ N (0, 1) CDF - It’s too difficult to write this one out, so we express it as the function Φ(x)
Exponential Distribution
Let us say that X is distributed Expo(λ). We know the following: Story You’re sitting on an open meadow right before the break of dawn, wishing that airplanes in the night sky were shooting stars, because you could really use a wish right now. You know that shooting stars come on average every 15 minutes, but it’s never true that a shooting star is ever ‘’due” to come because you’ve waited so long. Your waiting time is memorylessness, which means that the time until the next shooting star comes does not depend on how long you’ve waited already. Example The waiting time until the next shooting star is distributed Expo(4). The 4 here is λ, or the rate parameter, or how many shooting stars we expect to see in a unit of time. The expected time until the next shooting star is (^) λ^1 , or 14 of an hour. You can expect to wait 15 minutes until the next shooting star. Expos are rescaled Expos
Y ∼ Expo(λ) → X = λY ∼ Expo(1)
Memorylessness The Exponential Distribution is the sole continuous memoryless distribution. This means that it’s always “as good as new”, which means that the probability of it failing in the next infinitesimal time period is the same as any infinitesimal time period. This means that for an exponentially distributed X and any real numbers t and s,
P (X > s + t|X > s) = P (X > t)
Given that you’ve waited already at least s minutes, the probability of having to wait an additional t minutes is the same as the probability that you have to wait more than t minutes to begin with. Here’s another formulation.
X − a|X > a ∼ Expo(λ)
Example - If waiting for the bus is distributed exponentially with λ = 6, no matter how long you’ve waited so far, the expected additional waiting time until the bus arrives is always 16 , or 10 minutes. The distribution of time from now to the arrival is always the same, no matter how long you’ve waited. Min of Expos If we have independent Xi ∼ Expo(λi), then min(X 1 ,... , Xk ) ∼ Expo(λ 1 + λ 2 + · · · + λk ). Max of Expos If we have i.i.d. Xi ∼ Expo(λ), then max(X 1 ,... , Xk ) ∼ Expo(kλ) + Expo((k − 1)λ) + · · · + Expo(λ)
Gamma Distribution
Let us say that X is distributed Gamma(a, λ). We know the following:
Story You sit waiting for shooting stars, and you know that the waiting time for a star is distributed Expo(λ). You want to see “a” shooting stars before you go home. X is the total waiting time for the ath shooting star.
Example You are at a bank, and there are 3 people ahead of you. The serving time for each person is distributed Exponentially with mean of 2 time units. The distribution of your waiting time until you begin service is Gamma(3, 12 )
Beta Distribution
Conjugate Prior of the Binomial A prior is the distribution of a parameter before you observe any data (f (x)). A posterior is the distribution of a parameter after you observe data y (f (x|y)). Beta is the conjugate prior of the Binomial because if you have a Beta-distributed prior on p (the parameter of the Binomial), then the posterior distribution on p given observed data is also Beta-distributed. This means, that in a two-level model:
X|p ∼ Bin(n, p) p ∼ Beta(a, b)
Then after observing the value X = x, we get a posterior distribution p|(X = x) ∼ Beta(a + x, b + n − x)
Order statistics of the Uniform See Order Statistics
Relationship with Gamma This is the bank-post office result. See Reasoning by Representation
Distribution
Let us say that X is distributed χ^2 n. We know the following:
Story A Chi-Squared(n) is a sum of n independent squared normals.
Example The sum of squared errors are distributed χ^2 n
Properties and Representations
E(χ^2 n) = n, V ar(X) = 2n, χ^2 n ∼ Gamma
n 2
χ 2 n =^ Z
2 1 +^ Z
2 2 +^ · · ·^ +^ Z
2 n, Z^ ∼
i.i.d. N (0, 1)
Discrete Distributions
DWR = Draw w/ replacement, DWoR = Draw w/o replacement
DWR DWoR
Fixed # trials (n) Binom/Bern HGeom (Bern if n = 1) Draw ’til k success NBin/Geom NHGeom (Geom if k = 1) (see example probs)
Bernoulli
The Bernoulli distribution is the simplest case of the Binomial distribution, where we only have one trial, or n = 1. Let us say that X is distributed Bern(p). We know the following:
Story. X “succeeds” (is 1) with probability p, and X “fails” (is 0) with probability 1 − p.
Example. A fair coin flip is distributed Bern( 12 ).
Binomial
Let us say that X is distributed Bin(n, p). We know the following:
Story X is the number of ”successes” that we will achieve in n independent trials, where each trial can be either a success or a failure, each with the same probability p of success. We can also say that X is a sum of multiple independent Bern(p) random variables. Let X ∼ Bin(n, p) and Xj ∼ Bern(p), where all of the Bernoullis are independent. We can express the following:
X = X 1 + X 2 + X 3 + · · · + Xn
Example If Jeremy Lin makes 10 free throws and each one independently has a 34 chance of getting in, then the number of free
throws he makes is distributed Bin(10, 34 ), or, letting X be the number of free throws that he makes, X is a Binomial Random Variable distributed Bin(10, 34 ).
Binomial Coefficient
(n k
is a function of n and k and is read n choose k, and means out of n possible indistinguishable objects, how many ways can I possibly choose k of them? The formula for the binomial coefficient is:
(n
k
n! k!(n − k)!
Geometric
Let us say that X is distributed Geom(p). We know the following:
Story X is the number of “failures” that we will achieve before we achieve our first success. Our successes have probability p.
Example If each pokeball we throw has a 101 probability to catch Mew, the number of failed pokeballs will be distributed Geom( 101 ).
First Success
Equivalent to the geometric distribution, except it counts the total number of “draws” until the first success. This is 1 more than the number of failures. If X ∼ F S(p) then E(X) = 1/p.
Negative Binomial
Let us say that X is distributed NBin(r, p). We know the following:
Story X is the number of “failures” that we will achieve before we achieve our rth success. Our successes have probability p.
Example Thundershock has 60% accuracy and can faint a wild Raticate in 3 hits. The number of misses before Pikachu faints Raticate with Thundershock is distributed NBin(3, .6).
Hypergeometric
Let us say that X is distributed HGeom(w, b, n). We know the following:
Story In a population of b undesired objects and w desired objects, X is the number of “successes” we will have in a draw of n objects, without replacement.
Example 1) Let’s say that we have only b Weedles (failure) and w Pikachus (success) in Viridian Forest. We encounter n Pokemon in the forest, and X is the number of Pikachus in our encounters. 2) The number of aces that you draw in 5 cards (without replacement). 3) You have w white balls and b black balls, and you draw b balls. You will draw X white balls. 4) Elk Problem - You have N elk, you capture n of them, tag them, and release them. Then you recollect a new sample of size m. How many tagged elk are now in the new sample?
PMF The probability mass function of a Hypergeometric:
P (X = k) =
(w k
)( (^) b n−k
(w+b n
Poisson
Let us say that X is distributed Pois(λ). We know the following: Story There are rare events (low probability events) that occur many different ways (high possibilities of occurences) at an average rate of λ occurrences per unit space or time. The number of events that occur in that unit of space or time is X. Example A certain busy intersection has an average of 2 accidents per month. Since an accident is a low probability event that can happen many different ways, the number of accidents in a month at that intersection is distributed Pois(2). The number of accidents that happen in two months at that intersection is distributed Pois(4)
Multivariate Distributions
Multinomial
Let us say that the vector X~ = (X 1 , X 2 , X 3 ,... , Xk ) ∼ Multk (n, ~p) where ~p = (p 1 , p 2 ,... , pk ). Story - We have n items, and then can fall into any one of the k buckets independently with the probabilities ~p = (p 1 , p 2 ,... , pk ). Example - Let us assume that every year, 100 students in the Harry Potter Universe are randomly and independently sorted into one of four houses with equal probability. The number of people in each of the houses is distributed Mult 4 (100, ~p), where ~p = (. 25 ,. 25 ,. 25 , .25). Note that X 1 + X 2 + · · · + X 4 = 100, and they are dependent. Multinomial Coefficient The number of permutations of n objects where you have n 1 , n 2 , n 3... , nk of each of the different variants is the multinomial coefficient. ( (^) n n 1 n 2... nk
n! n 1 !n 2!... nk!
Joint PMF - For n = n 1 + n 2 + · · · + nk
P ( X~ = ~n) =
( (^) n
n 1 n 2... nk
p n 1 1 p
n 2 2... p
nk k
Lumping - If you lump together multiple categories in a multinomial, then it is still multinomial. A multinomial with two dimensions (success, failure) is a binomial distribution. Variances and Covariances - For (X 1 , X 2 ,... , Xk ) ∼ Multk (n, (p 1 , p 2 ,... , pk )), we have that marginally Xi ∼ Bin(n, pi) and hence Var(Xi) = npi(1 − pi). Also, for i 6 = j, Cov(Xi, Xj ) = −npipj , which is a result from class. Marginal PMF and Lumping
Xi ∼ Bin(n, pi)
Xi + Xj ∼ Bin(n, pi + pj )
X 1 ,X 2 ,X 3 ∼Mult 3 (n,(p 1 ,p 2 ,p 3 ))→X 1 ,X 2 +X 3 ∼Mult 2 (n,(p 1 ,p 2 +p 3 ))
X 1 ,... , Xk− 1 |Xk = nk ∼ Multk− 1
n − nk ,
p 1 1 − pk
pk− 1 1 − pk
Multivariate Uniform
See the univariate uniform for stories and examples. For multivariate uniforms, all you need to know is that probability is proportional to volume. More formally, probability is the volume of the region of interest divided by the total volume of the support. Every point in the support has equal density of value (^) Total Area^1.
Multivariate Normal (MVN)
A vector X~ = (X 1 , X 2 , X 3 ,... , Xk ) is declared Multivariate Normal if any linear combination is normally distributed (e.g. t 1 X 1 + t 2 X 2 + · · · + tk Xk is Normal for any constants t 1 , t 2 ,... , tk ). The parameters of the Multivariate normal are the mean vector ~μ = (μ 1 , μ 2 ,... , μk ) and the covariance matrix where the (i, j)th^ entry is Cov(Xi, Xj ). For any MVN distribution: 1) Any sub-vector is also MVN. 2) If any two elements of a multivariate normal distribution are uncorrelated, then they are independent. Note that 2) does not apply to most random variables.
Distribution Properties
Important CDFs
Exponential F (X) = 1 − e−λx, x ∈ (0, ∞)) Uniform(0, 1) F (X) = x, x ∈ (0, 1)
Poisson Properties (Chicken and Egg Results)
We have X ∼ Pois(λ 1 ) and Y ∼ Pois(λ 2 ) and X ⊥⊥ Y.
- X + Y ∼ Pois(λ 1 + λ 2 )
- X|(X + Y = k) ∼ Bin
k, λ 1 λ 1 +λ 2
- If we have that Z ∼ Pois(λ), and we randomly and independently “accept” every item in Z with probability p, then the number of accepted items Z 1 ∼ Pois(λp), and the number of rejected items Z 2 ∼ Pois(λq), and Z 1 ⊥⊥ Z 2.
Convolutions of Random Variables
A convolution of n random variables is simply their sum.
- X ∼ Pois(λ 1 ), Y ∼ Pois(λ 2 ), X ⊥⊥ Y −→ X + Y ∼ Pois(λ 1 + λ 2 )
- X ∼ Bin(n 1 , p), Y ∼ Bin(n 2 , p), X ⊥⊥ Y −→ X + Y ∼ Bin(n 1 + n 2 , p) Note that Binomial can thus be thought of as a sum of iid Bernoullis.
- X ∼ Gamma(n 1 , λ), Y ∼ Gamma(n 2 , λ), X ⊥⊥ Y −→ X + Y ∼ Gamma(n 1 + n 2 , λ) Note that Gamma can thus be thought of as a sum of iid Expos.
- X ∼ NBin(r 1 , p), Y ∼ NBin(r 2 , p), X ⊥⊥ Y −→ X + Y ∼ NBin(r 1 + r 2 , p)
- All of the above are approximately normal when λ, n, r are large by the Central Limit Theorem.
- Z 1 ∼ N (μ 1 , σ^21 ), Z 2 ∼ N (μ 2 , σ^22 ), Z 1 ⊥⊥ Z 2 −→ Z 1 + Z 2 ∼ N (μ 1 + μ 2 , σ^21 + σ^22 )
Special Cases of Random Variables
- Bin(1, p) ∼ Bern(p)
- Beta(1, 1) ∼ Unif(0, 1)
- Gamma(1, λ) ∼ Expo(λ)
- χ^2 n ∼ Gamma
( (^) n 2 ,^
1 2
- NBin(1, p) ∼ Geom(p)
Reasoning by Representation
Beta-Gamma relationship If X ∼ Gamma(a, λ), Y ∼ Gamma(b, λ), X ⊥⊥ Y then
- (^) XX+Y ∼ Beta(a, b)
- X + Y ⊥⊥ (^) XX+Y
This is also known as the bank-post office result. Binomial-Poisson Relationship Bin(n, p) → Pois(λ) as n → ∞, p → 0, np = λ. Order Statistics of Uniform U(j) ∼ Beta(j, n − j + 1) Universality of Uniform For any X with CDF F (x), F (X) ∼ U
Formulas
In general, remember that PDFs integrated (and PMFs summed) over support equal 1.
Geometric Series
a + ar + ar^2 + · · · + arn−^1 =
n∑− 1
k=
ark^ = a
1 − rn 1 − r
Exponential Function (e
x
ex^ =
∑^ ∞
n=
xn n!
= 1 + x +
x^2 2!
x^3 3!
x n
)n
MGF - Finding Momemts
Find E(X^3 ) for X ∼ Expo(λ) using the MGF of X. Answer - The MGF of an Expo(λ) is M (t) = (^) λλ−t. To get the third moment, we can take the third derivative of the MGF and evaluate at t = 0:
E(X^3 ) =
λ^3
But a much nicer way to use the MGF here is via pattern recognition: note that M (t) looks like it came from a geometric series:
1 1 − (^) λt
∑^ ∞
n=
t λ
)n
∑^ ∞
n=
n! λn
tn n!
The coefficient of t
n n! here is the^ n
th (^) moment of X, so we have
E(Xn) = (^) λnn! for all nonnegative integers n. So again we get the same answer.
Markov Chains
Suppose Xn is a two-state Markov chain with transition matrix
Q =
0 1 − α α 1 β 1 − β
Find the stationary distribution ~s = (s 0 , s 1 ) of Xn by solving ~sQ = ~s, and show that the chain is reversible under this stationary distribution. Answer - By solving ~sQ = ~s, we have that
s 0 = s 0 (1 − α) + s 1 β and s 1 = s 0 (α) + s 0 (1 − β)
And by solving this system of linear equations it follows that
~s =
β α + β
α α + β
To show that this chain is reversible under this stationary distribution, we must show siqij = sj qji for all i, j. This is done if we can show s 0 q 01 = s 1 q 10. Indeed,
s 0 q 01 =
αβ α + β
= s 1 q 10
thus our chain is reversible under the stationary distribution.
Markov Chains, continued
William and Sebastian play a modified game of Settlers of Catan, where every turn they randomly move the robber (which starts on the center tile) to one of the adjacent hexagons.
Robber
a) Is this Markov Chain irreducible? Is it aperiodic? Answer - Yes to both The Markov Chain is irreducible because it can get from anywhere to anywhere else. The Markov Chain is also aperiodic because the robber can return back to a square in 2 , 3 , 4 , 5 ,... moves. Those numbers have a GCD of 1, so the chain is aperiodic.
b) What is the stationary distribution of this Markov Chain? Answer - Since this is a random walk on an undirected graph, the stationary distribution is proportional to the degree sequence. The degree for the corner pieces is 3, the degree for the edge pieces is 4, and the degree for the center pieces is 6. To normalize this degree sequence, we divide by its sum. The sum of the degrees is 6(3) + 6(4) + 7(6) = 72. Thus the stationary probability of being on a corner is 3/84 = 1/28, on an edge is 4/84 = 1/21, and in the center is 6/84 = 1/14. c) What fraction of the time will the robber be in the center tile in this game? Answer - From above, 1 /.
d) What is the expected amount of moves it will take for the robber to return? Answer - Since this chain is irreducible and aperiodic, to get the expected time to return we can just invert the stationary probability. Thus on average it will take 14 turns for the robber to return to the center tile.
Problem Solving Strategies
Contributions from Jessy Hwang, Yuan Jiang, Yuqi Hou
- Getting Started. Start by defining events and/or defining random variables. (”Let A be the event that I pick the fair coin”; “Let X be the number of successes.”) Clear notion = clear thinking! Then decide what it is that you’re supposed to be finding, in terms of your location (“I want to find P (X = 3|A)”). Try simple and extreme cases. To make an abstract experiment more concrete, try drawing a picture or making up numbers that could have happened. Pattern recognition: does the structure of the problem resemble something we’ve seen before.
- Calculating Probability of an Event. Use combinatorics if the naive definition of probability applies. Look for symmetries or something to condition on, then apply Bayes’ rule or LoTP. Is the probability of the complement easier to find?
- Finding the distribution of a random variable. Check the support of the random variable: what values can it take on? Use this to rule out distributions that don’t fit. - Is there a story for one of the named distributions that fits the problem at hand? - Can you write the random variable as a function of a r.v. with a known distribution, say Y = g(X)? Then work directly from the definition of PDF or PMF, expressing P (Y ≤ y) or P (Y = y) in terms of events involving X only. - For PDFs, find the CDF first and then differentiate. - If you’re trying to find the joint distribution of two independent random variables, just multiple their marginal probabilities - Do you need the distribution? If the question only asks for the expected value of X, you might be able to find this without knowing the entire distirbution of X. See the next item.
- Calculating Expectation. If it has a named distribution, check out the table of distributions. If its a function of a r.v. with a named distribution, try LotUS. If its a count of something, try breaking it up into indicator random variables. If you can condition on something, consider using Adam’s law. Also consider the variance formula.
- Calculating Variance. Consider independence, named distributions, and LotUS. If it’s a count of something, break it up into a sum of indicator random variables. If you can condition on something, consider using Eve’s Law.
- Calculating E(X^2 ) - Do you already know E(X) or Var(X)? Remember that Var(X) = E(X^2 ) − E(X)^2.
- Calculating Covariance If it’s a count of something, break it up into a sum of indicator random variables. If you’re trying to calculate the covariance between two components of a multinomial distribution, Xi, Xj , then the covariance is −npipj.
- If X and Y are i.i.d., have you considered using symmetry?
- Calculating Probabilities of Orderings of Random Variables Have you considered looking at order statistics? - Remember any ordering of i.i.d. random variables is equally likely. 10. Is this the birthday problem? Is this a multinomial problem? 11. Determining Independence Use the definition of independence. Think of extreme cases to see if you can find a counterexample. 12. Does something look like Simpson’s Paradox? make sure you’re looking at 3 events. 13. Find the PDF. If the question gives you two r.v., where you know the PDF of one r.v. and the other r.v. is a function of the first one, then the problem wants you to use a transformation of variables (Jacobian). You can also find the pdf by differentiating the CDF. 14. Do a painful integral. If your integral looks painful, see if you can write your integral in terms of a PDF (like Gamma or Beta), so that the integral equals 1. 15. Before moving on. Plug in some simple and extreme cases to make sure that your answer makes sense.
Biohazards
Section author: Jessy Hwang
- Don’t misuse the native definition of probability - When answering “What is the probability that in a group of 3 people, no two have the same birth month?”, it is not correct to treat the people as indistinguishable balls being placed into 12 boxes, since that assumes the list of birth months {January, January, January} is just as likely as the list {January, April, June}, when the latter is fix times more likely.
- Don’t confuse unconditional and conditional probabilities, or go in circles with Baye’s Rule - P (A|B) = P (B|A)P (A) P (B). It is^ not^ correct to say “P^ (B) = 1 because we know that B happened.”; P(B) is the probability before we have information about whether B happened. It is not correct to use P (A|B) in place of P (A) on the right-hand side.
- Don’t assume independence without justification - In the matching problem, the probability that card 1 is a match and card 2 is a match is not 1/n^2. - The Binomial and Hypergeometric are often confused; the trials are independent in the Binomial story and not independent in the Hypergeometric story due to the lack of replacement.
- Don’t confuse random variables, numbers, and events. - Let X be a r.v. Then f (X) is a r.b. for any function f. In particular, X^2 , |X|, F (X), and IX> 3 are r.v.s. P (X^2 < X|X ≥ 0), E(X), Var(X), and f (E(X)) are numbers. X = 2 and F (X) ≥ −1 are events. It does not make sense to write
−∞ F^ (X)dx^ because^ F^ (X) is a random variable. It does not make sense to write P (X) because X is not an event.
- A random variable is not the same thing as its distribution - To get the PDF of X^2 , you can’t just square the PDF of X. The right way is to use one variable transformations
- To get the PDF of X + Y , you can’t just add the PDF of X and the PDF of Y. The right way is to compute the convolution.
- E(g(X)) does not equal g(E(X)) in general. - See the St. Petersburg paradox for an extreme example. - The right way to find E(g(X)) is with LotUS.
Distributions
Distribution PDF and Support Expected Value Variance MGF
Bernoulli
Bern(p)
P (X = 1) = p
P (X = 0) = q p pq q + pet
Binomial
Bin(n, p)
P (X = k) =
(n
k
pk^ (1 − p)n−k
k ∈ { 0 , 1 , 2 ,... n} np npq (q + pet)n
Geometric
Geom(p)
P (X = k) = qk^ p
k ∈ {0, 1, 2,... } q/p q/p^2
p
1 −qet^ , qe
t < 1
Negative Binom.
NBin(r, p)
P (X = n) =
(r+n− 1
r− 1
pr^ qn
n ∈ {0, 1, 2,... } rq/p rq/p^2 (
p
1 −qet^ )
r , qet < 1
Hypergeometric
HGeom(w, b, n)
P (X = k) =
(w k
)( (^) b n−k
)
(w+b n
)
k ∈ { 0 , 1 , 2 ,... , n} μ = bnw+w
w+b−n
w+b− 1 n^
μ
n (1^ −^
μ
n )^ −
Poisson
Pois(λ)
P (X = k) = e
−λλk k!
k ∈ {0, 1, 2,... } λ λ eλ(e
t−1)
Uniform
Unif(a, b)
f (x) = 1
b−a
x ∈ (a, b) a+b
2
(b−a)^2 12
etb−eta t(b−a)
Normal
N (μ, σ^2 )
f (x) = 1
σ
√ 2 π
e−(x^ −^ μ)
(^2) /(2σ (^2) )
x ∈ (−∞, ∞) μ σ^2 etμ+^
σ^2 t^2 2
Exponential
Expo(λ)
f (x) = λe−λx
x ∈ (0, ∞) 1 /λ 1 /λ^2 λλ−t , t < λ
Gamma
Gamma(a, λ)
f (x) = Γ(^1 a) (λx)ae−λx^1 x
x ∈ (0, ∞) a/λ a/λ^2
λ λ−t
)a
, t < λ
Beta
Beta(a, b)
f (x) =
Γ(a+b)
Γ(a)Γ(b) x
a− 1 (1 − x)b− 1
x ∈ (0, 1) μ =
a a+b
μ(1−μ)
(a+b+1) −
Chi-Squared
χ^2 n
1 2 n/^2 Γ(n/2)
xn/^2 −^1 e−x/^2
x ∈ (0, ∞) n 2 n (1 − 2 t)−n/^2 , t < 1 / 2
Multivar Uniform
A is support
f (x) = |A^1 |
x ∈ A − − −
Multinomial
Multk (n, ~p)
P ( X~ = ~n) =
( n
n 1 ...nk
p
n 1
1... p
nk k
n = n 1 + n 2 + · · · + nk n~p
Var(Xi) = npi(1 − pi)
Cov(Xi, Xj ) = −npipj
k
i=1 pie
ti
)n
Inequalities
Cauchy-Schwarz Markov Chebychev Jensen
|E(XY )| ≤
E(X^2 )E(Y 2 ) P (X ≥ a) ≤
E|X|
a
P (|X − μX | ≥ a) ≤
σ X^2
a^2
g convex: E(g(X)) ≥ g(E(X))
g concave: E(g(X)) ≤ g(E(X))
