Lesson 3

Monte Carlo simulation becomes especially useful when the event itself is defined by a program. Suppose we want to evaluate a system that produces random outputs: for example, a text generation system tested on prompts from a fixed benchmark. One run of the experiment consists of choosing a prompt, running the system with its random choices, and checking whether the answer has some property: maybe it violates a rule, maybe it contains a factual error, or it solves the task correctly. The event may be difficult to describe by a closed formula, but it is easy to check after the run. Repeating the experiment gives an estimate of the corresponding failure rate or success rate.

There is, however, a serious warning: Monte Carlo is not magic. Rare events require many trials. If an event has probability about \(10^{-6}\), then a simulation with \(10\ 000\) trials will usually not see it even once. Thus, simulation is a good companion to theory, but not a replacement for it. Later, when we discuss variance and deviation inequalities, we will learn how to put explicit confidence guarantees on such estimates.