Download Markov and Chebyshev Inequalities: Proofs and Illustrations and more Study notes Probability and Statistics in PDF only on Docsity!
Markov and Cheybshev Inequalities and the Law of
Large Numbers—Proofs and Illustrations
Markov's Inequality
Suppose a random variable X takes only nonnegative values so that P{ X ≥ 0} = 1.
How much probability is there in the tail of the distribution of X?
More specifically, for a given value a > 0, what can we say about the value of P{ X ≥ a }?
Markov's inequality takes μ = E( X ) into account and
provides an upper bound on P{ X ≥ a } that
depends on the values of a and μ.
We give the derivation for a continuous random variable X with density function f ( x ):
μ = E( X ) = ⌠⌡(0 , ∞) x f ( x ) dx = ⌠⌡(0 , a ) x f ( x ) dx + ⌠⌡[ a , ∞) x f ( x ) dx ≥ ⌠⌡[ a , ∞) x f ( x ) dx ≥ ⌠⌡[ a , ∞) a f ( x ) = a ⌠⌡[ a , ∞) f ( x ) dx = a P{ X ≥ a },
from which we obtain Markov's Inequality:
P{ X ≥ a } ≤ μ/ a.
In the above:
! The first inequality holds because the integral ignored is nonnegative,
! The second inequality holds because a ≤ x , for x in [ a , ∞).
The proof for a discrete random variable is similar, with summations replacing integrals.
Markov's Inequality gives an upper bound on P{ X ≥ a } that applies to any distribution with positive support.
Practical consequences.
For most distributions of practical interest,
the probability in the tail beyond a is
noticeably smaller than μ/ a for all values of a.
Below, for several continuous and discrete distributions,
each with μ = 1, we use R to show that the nonincreasing "reliability function" R ( a ) = 1 – F ( a ) = P{ X > a } is
bounded above by μ/ a = 1/ a.
Exponential distribution with mean 1
a <- seq(0, 4, length=1000)
aa <- seq(1, 4, length= 500)
plot(a, 1 - pexp(a, rate=1), type="l", ylim=c(0,1), ylab="R", main="Markov Bound for EXP(1)")
lines(aa, 1/aa, col="red")
Uniform distribution on (0, 2).
a <- seq(0, 4, length=1000)
aa <- seq(1, 4, length= 500)
plot(a, 1 - punif(a,0,2), type="l", ylim=c(0,1), ylab="R", main="Markov Bound for UNIF(0,2)")
lines(aa, 1/aa, col="red")
Binomial distribution with n = 2 and p = 1/2.
a <- seq(-.01, 4, by=.001)
aa <- seq(1, 4, length= 500)
plot(a, 1 - pbinom(a,2,.5), type="l", ylim=c(0,1), ylab="R", main="Markov Bound for BINOM(2,.5)")
lines(aa, 1/aa, col="red")
Poisson distribution with λλλλ = 1.
a <- seq(-.01, 4, by=.001)
aa <- seq(1, 4, length= 500)
plot(a, 1 - ppois(a,1), type="l", ylim=c(0,1), ylab="R", main="Markov Bound for POIS(1)")
lines(aa, 1/aa, col="red")
Copyright © 2004 by Bruce E. Trumbo. All rights reserved. Department of Statistics, California State University, Hayward.
The following table compares the some of the information from Chebyshev's Inequality with exact information about a normal distribution (upon which the Empirical Rule is based).
Interval Exact Chebyshev
( μμμμ - σσσσ , μμμμ + σσσσ ) 0.681 Uninformative
( μμμμ - 1.5 σσσσ , μμμμ + 1.5 σσσσ ) 0.866 ≥≥≥≥ 1 - 4/9 = 0.
( μμμμ - 2 σσσσ , μμμμ + 2 σσσσ ) 0.950 ≥≥≥≥ 1 - 1/4 = 0.
( μμμμ - 3 σσσσ , μμμμ + 3 σσσσ ) 0.997 ≥≥≥≥ 1 - 1/9 = 0.
Below for several continuous and discrete distributions, each with μ = 1 and σ = 1, we use R to show that the nondecreasing function Q ( k ) = P{| Y – μ| ≤ k σ} = P{| Y – 1 | ≤ k } is bounded above by 1 – 1/ k^2.
Problems:
(c) In the normal example below, verify the values in the table above as closely as you can by reading the graph.
(d) In the remaining examples, verify that the variance is 1 and find the functional forms of Q. Draw a similar sketch of Q compared with the Chebyshev bound for the distribution that places probability 1/8 at each of the points 0 and 2 and probability 3/4 at 1.
Normal distribution with μμμμ = 1 and σσσσ = 1.
k <- seq(-.01, 4, by=.001)
kk <- seq(1, 4, length= 500)
plot(k, pnorm(1+k,1,1)-pnorm(1-k,1,1), type="l", ylim=c(0,1), ylab="Q", main="Chebyshev Bound for NORM(1,1)")
lines(kk, 1 - 1/kk^2, col="red")
Uniform distribution on (1 – √√√√ 3, 1 + √√√√ 3).
k <- seq(-.01, 4, by=.001)
kk <- seq(1, 4, length= 500)
plot(k, punif(1+k,1-sqrt(3),1+sqrt(3))-punif(1- k,1-sqrt(3),1+sqrt(3)), type="l", ylim=c(0,1), ylab="Q", main="Chebyshev Bound for UNIF(-0.732,2.732)")
lines(kk, 1 - 1/kk^2, col="red")
Poisson distribution with λλλλ = 1.
k <- seq(-.01, 4, by=.001)
kk <- seq(1, 4, length= 500)
plot(k, ppois(1+k,1)-ppois(1-k-.001,1), type="l", ylim=c(0,1), ylab="Q", main="Chebyshev Bound for POIS(2)")
lines(kk, 1 - 1/kk^2, col="red")
Law of Large Numbers for Coin Tossing.
An important theoretical use of Chebyshev's Inequality is to prove the Law of Large Numbers for Coin Tossing.
If a coin with P(Heads) = p is tossed n times, then the heads ratio Zn = #(Heads)/ n has mean μ = E( Zn ) = p and σ^2 = V( Zn ) = p (1 – p )/ n.
Thus for arbitrarily small ε = k σ > 0, Chebyshev's inequality gives
1 ≥ P{| Zn – p | < ε} ≥ 1 – p (1 – p )/ n ε^2 → 1, as n → ∞.
Thus P{| Zn – p | < ε} → 1.
Note: Here ε = k σ = k [ p (1 – p )/ n ]1/2, so 1/ k^2 = p (1 – p )/ n ε^2.
We say that Zn converges to p in probability and write Zn
prob → p.
In words, as the number of tosses increases to a sufficiently large number, we see that the
heads ratio is within ε of p with probability as near 1 as we please.
As a practical matter, Zn is nearly normally distributed for large n. Thus the normal distribution is a better way than the Chebyshev bound to assess the accuracy of Zn as an estimate of p.
Roughly speaking, this amounts to using the Empirical Rule.
As a specific example: in 10 000 tosses of a fair coin, 2 SD( V 10000) = 2( pq /10 000) 1/2^ = 2(40 000) –1/2^ = 2/200 = 0.01.
So we can expect the heads ratio to be within 0.01 of 1/ with probability 95%.
Fifty thousand tosses would allow approximation of p with 2-place accuracy.
Problem:
(e) What does Chebyshev's inequality say about P{| Z 12500 – p | < 1/150}? What does the normal approximation say about this probability?
The graph on the next page illustrates Chebyshev and CLT bounds for tossing a fair coin.
