Lesson 2

Now assume that we have a training sample \[(x_{1},y_{1}),(x_{2},y_{2}),\dotsc,(x_{n},y_{n}),\] interpreted as independent observations of the same random pair \((X,Y)\). For a fixed parameter value \(\theta\), the model assigns the following probability (or density) to the observed answers: \[L(\theta) = p(y_{1},\dotsc,y_{n} \mid x_{1},\dotsc,x_{n},\theta) = \prod_{i=1}^{n}p(y_{i} \mid x_{i},\theta).\] This quantity is called the likelihood. The product appears because the observations are treated as independent once the inputs and the parameter are fixed.

One small warning about terminology: likelihood is not a probability distribution over \(\theta\). In this chapter, \(\theta\) is just a candidate parameter value. The likelihood is a score telling us how plausible the observed data would look if this value of \(\theta\) were used.