Lesson 4

This example is the beginning of a simple probabilistic classifier. Suppose an email contains several words \(w_{1},\dotsc,w_{k}\), and we want to compare two hypotheses: \[H_{1}=\text{“the email is spam”}, \qquad H_{0}=\text{“the email is not spam”}.\] Bayes' formula tells us that the posterior probability of spam is proportional to \[\Pr[H_{1}]\Pr[w_{1},\dotsc,w_{k} \mid H_{1}].\] The difficult part is the joint probability of all words appearing together. The naive Bayes classifier makes a very strong simplifying assumption: after the class is fixed, the features are treated as independent. Under this assumption, \[\Pr[w_{1},\dotsc,w_{k} \mid H_{1}] \approx \Pr[w_{1} \mid H_{1}]\cdots \Pr[w_{k} \mid H_{1}].\] The same expression is computed for \(H_{0}\), and the larger posterior wins. The assumption is usually false, but the method is simple, fast, and often surprisingly effective.

The example above is simple, but it illustrates a general principle behind automated prediction systems. Many such systems estimate conditional probabilities from data: \[\Pr[\text{label}\mid \text{features}], \qquad \Pr[\text{next token}\mid \text{context}], \qquad \Pr[\text{user clicks}\mid \text{shown item}].\]