Lesson 4

Directly maximizing a product of many small probabilities is inconvenient. Instead, one usually maximizes the log-likelihood \[\ell(\theta) = \log L(\theta) = \sum_{i=1}^{n}\log p(y_{i} \mid x_{i},\theta).\] The logarithm is a monotone function, so it does not change which value of \(\theta\) is best. But it turns a product into a sum, which is much easier to optimize numerically. It also avoids multiplying many tiny numbers, which can quickly become too small for a computer to store accurately.