# Central Limit Theorem and Law of Large Numbers

This post is about some understandings of the Central Limit Theorem and (un)related stuff. First of all, a very good lecture Lecture 29: Law of Large Numbers and Central Limit Theorem | Statistics 110.

## Law of Large Numbers

According to Wikipedia, $\bar{x}_n-\mu \rightarrow 0$ LLN says that the sample average converges to the expected value as $$n$$ approaches infinity. But that doesn’t tell how fast it converges.

## Central Limit Theorem

According to Wikipedia, $\sqrt{n}(\bar{x}_n-\mu) \rightarrow N(0,\sigma^2)$ CLT says that as $$n$$ approaches infinity, the random variable $$\sqrt{n}(\bar{x}_n-\mu)$$ converges in distribution to a normal $$N(0,\sigma^2)$$.

The interesting part is the scaling factor $$\sqrt{n}$$, for a smaller scaling factor the whole term $$\sqrt{n}(\bar{x}_n-\mu)$$ always goes to zero, for a larger scaling factor the term will blow up, only for $$\sqrt{n}$$ it converges to a distribution with constant variance.

For instance, for $$n$$ i.i.d variables with zero mean, as $$n$$ gets larger, their mean goes to zero (LLN), their sum blows up to positive/negative infinity, but this term $$\sqrt{n}(\bar{x}_n-\mu)$$ converges in distribution to a normal.

CTL is useful in many areas including the Wiener process, which is often used to model the stock price, see this blog and this question.

## Variance of Sum of IID Random Variables

The $$\sqrt{n}$$ scaling factor also applies when summing a finite number of i.i.d random variables (ref: sum of uncorrelated variables). For example sum of normally distributed random variables, and Irwin–Hall distribution.

Note that this is different from applying a change of variable $$z=2x$$, which gives a variance of $$(2\sigma)^2$$ instead of $$2\sigma^2$$.

For $$n$$ normally distributed random variables with the same variance, it follows $$\sum (x_n-\mu_n) \sim N(0, n\sigma^2)$$, which is equivalent to $\sqrt{n}(\bar{x}_n-\mu) \sim N(0,\sigma^2)$ the Central Limit Theorem.

## Sampling Normal Distributions

Using the facts above we can approximate samples from any normal distribution given samples from a uniform distribution $$U(0,1)$$ (though in fact Box–Muller transform can do that accurately).

According to Central Limit Theorem, the sum of $$n$$ uniformly distributed variables approximately follows a normal distribution, $$\sum x \sim N(n/2,\sigma^2)$$. According to sum of uncorrelated variables, we can work out the variance $$\sigma^2 = nVar(x)=n/12$$, therefore $$\sqrt{n}(\sum x/n-1/2)$$ converges in distribution to $$N(0,1/12)$$.