Lesson 3

The empirical distribution gives a general recipe known as the plug-in principle: if the true distribution is unknown, replace it by the empirical distribution and compute the quantity you need. For example, if we want the probability of a tail event, we count how often it happened: \[\widehat{\Pr}_{N}[\alpha \ge t] = \frac{|\{i \colon x_i \ge t\}|}{N}.\] If we want the typical value of a function \(g(\alpha)\), we average its observed values: \[\frac{g(x_1)+g(x_2)+\dotsb+g(x_N)}{N}.\] The indicator function \(g(x)=[x\in A]\) gives exactly the empirical probability above. Later, when we discuss expectation and deviation inequalities, this point of view will become one of the main bridges between probability theory and data.