This post is about some understandings of the Central Limit Theorem and (un)related stuff. First of all, a very good lecture Lecture 29: Law of Large Numbers and Central Limit Theorem | Statistics 110.

Law of Large Numbers

According to Wikipedia, \[ \bar{x}_n-\mu \rightarrow 0\] LLN says that the sample average converges to the expected value as \(n\) approaches infinity. But that doesn’t tell how fast it converges.

Central Limit Theorem

According to Wikipedia, \[ \sqrt{n}(\bar{x}_n-\mu) \rightarrow N(0,\sigma^2)\] CLT says that as \(n\) approaches infinity, the random variable \( \sqrt{n}(\bar{x}_n-\mu) \) converges in distribution to a normal \( N(0,\sigma^2) \).

The interesting part is the scaling factor \( \sqrt{n} \), for a smaller scaling factor the whole term \( \sqrt{n}(\bar{x}_n-\mu) \) always goes to zero, for a larger scaling factor the term will blow up, only for \( \sqrt{n} \) it converges to a distribution with constant variance.

For instance, for \(n\) i.i.d variables with zero mean, as \(n\) gets larger, their mean goes to zero (LLN), their sum blows up to positive/negative infinity, but this term \( \sqrt{n}(\bar{x}_n-\mu) \) converges in distribution to a normal.

CTL is useful in many areas including the Wiener process, which is often used to model the stock price, see this blog and this question.

Variance of Sum of IID Random Variables

The \( \sqrt{n} \) scaling factor also applies when summing a finite number of i.i.d random variables (ref: sum of uncorrelated variables). For example sum of normally distributed random variables, and Irwin–Hall distribution.

Note that this is different from applying a change of variable \(z=2x\), which gives a variance of \((2\sigma)^2\) instead of \(2\sigma^2\).

For \(n\) normally distributed random variables with the same variance, it follows \(\sum (x_n-\mu_n) \sim N(0, n\sigma^2)\), which is equivalent to \[ \sqrt{n}(\bar{x}_n-\mu) \sim N(0,\sigma^2)\] the Central Limit Theorem.

Sampling Normal Distributions

Using the facts above we can approximate samples from any normal distribution given samples from a uniform distribution \(U(0,1)\) (though in fact Box–Muller transform can do that accurately).

According to Central Limit Theorem, the sum of \(n\) uniformly distributed variables approximately follows a normal distribution, \(\sum x \sim N(n/2,\sigma^2)\). According to sum of uncorrelated variables, we can work out the variance \(\sigma^2 = nVar(x)=n/12\), therefore \(\sqrt{n}(\sum x/n-1/2)\) converges in distribution to \( N(0,1/12) \).