Lesson 3

The maximum likelihood estimate is a parameter value that maximizes the likelihood: \[\widehat{\theta}_{\mathrm{MLE}}\in \operatorname*{arg\,max}_{\theta \in \Theta}\prod_{i=1}^{n}p(y_{i} \mid x_{i},\theta).\] The abbreviation \(\mathrm{MLE}\) stands for maximum likelihood estimator. If several parameter values give the same maximum, the MLE is not unique. In some models it may fail to exist at all, for example when the likelihood can be made arbitrarily large but never reaches its maximum.

The idea is simple: among all parameter values in the family, choose one under which the data we actually observed is as unsurprising as possible. This is one of the most common ways probabilistic models are fitted in practice.