Lesson 4

The empirical distribution can also be used to estimate how unstable an estimate is. Suppose a system was tested on ten tasks, and the observed outcomes were \[1,1,0,1,1,1,0,1,0,1,\] where \(1\) means success and \(0\) means failure. The observed success rate is \(0.7\). But ten tasks is a small sample: if we had chosen another ten tasks, the rate could have been different.

A simple computational trick is called the bootstrap. We pretend that the empirical distribution is the real distribution, sample ten observations from it with replacement, and recompute the success rate. Repeating this many times gives a rough picture of the variability of the estimate.

import random

import matplotlib.pyplot as plt

outcomes = [1, 1, 0, 1, 1, 1, 0, 1, 0, 1]

random.seed(7)
rates = []
for _ in range(2000):
    sample = random.choices(outcomes, k=len(outcomes))
    rates.append(sum(sample) / len(sample))
rates.sort()

print("empirical success rate:", sum(outcomes) / len(outcomes))
print("bootstrap 90% interval:", (rates[100], rates[1900]))

plt.hist(
    rates,
    bins=[k / 10 for k in range(11)],
    density=True,
    edgecolor="black",
    color="#8ecae6",
)
plt.axvline(sum(outcomes) / len(outcomes), color="#d62828", linewidth=2)
plt.xlabel("bootstrap success rate")
plt.ylabel("density")
plt.tight_layout()
plt.savefig("empirical_bootstrap.png")

empirical success rate: 0.7
bootstrap 90% interval: (0.5, 0.9)

The output says that, under this empirical distribution, another ten-task test would often give a success rate somewhere between \(0.5\) and \(0.9\). In other words, the estimate \(0.7\) is a very rough summary.