Lesson 5

In simple cases, the maximum likelihood estimate can be found analytically. Suppose there are no features \(x\), and the observations are zeros and ones: \[y_{1},\dotsc,y_{n} \in \{0,1\}.\] We model them by a Bernoulli distribution with unknown parameter \(\theta\): \[\Pr[Y=1]=\theta, \qquad \Pr[Y=0]=1-\theta.\] If the sample contains \(s\) ones, then the likelihood is \[L(\theta)=\theta^{s}(1-\theta)^{n-s}.\] Therefore, \[\ell(\theta) = s\log \theta+(n-s)\log(1-\theta).\] Assuming \(0<s<n\), the derivative is \[\ell'(\theta) = \frac{s}{\theta}- \frac{n-s}{1-\theta}.\] The maximum is reached when \[s(1-\theta)=(n-s)\theta,\] and hence \[\widehat{\theta}_{\mathrm{MLE}}=\frac{s}{n}.\] Thus, for a Bernoulli model, maximum likelihood chooses exactly the empirical success rate. If \(s=0\) or \(s=n\), the maximum is attained at the boundary \(\theta=0\) or \(\theta=1\) if the boundary is allowed. If the parameter space is open, for example \(\Theta=(0,1)\), then in these extreme cases the maximum is not reached.