Lesson 5

Bootstrap becomes more useful when we compare two systems. Suppose two classifiers, search engines, or prompts are tested on the same set of tasks. System B may have a higher score than system A, but is the difference meaningful, or did B simply get lucky on this particular test set?

The right observation is now a pair \((a_{i},b_{i})\), where \(a_{i}\) and \(b_{i}\) are the results of the two systems on the same task. We will resample these pairs, not the two lists separately. After each resampling, we compute the difference \[\text{success rate of B}-\text{success rate of A}.\]

import random

import matplotlib.pyplot as plt


# Each pair is (result of system A, result of system B) on the same task.
tasks = (
    [(1, 1)] * 22
    + [(1, 0)] * 3
    + [(0, 1)] * 9
    + [(0, 0)] * 6
)


def success_rate(system):
    return sum(task[system] for task in tasks) / len(tasks)


random.seed(11)
trials = 5000
diffs = []
for _ in range(trials):
    sample = random.choices(tasks, k=len(tasks))
    rate_a = sum(a for a, b in sample) / len(sample)
    rate_b = sum(b for a, b in sample) / len(sample)
    diffs.append(rate_b - rate_a)

diffs.sort()
observed_difference = success_rate(1) - success_rate(0)
low = diffs[int(0.05 * trials)]
high = diffs[int(0.95 * trials)]
probability_b_better = sum(diff > 0 for diff in diffs) / trials

print("system A success rate:", round(success_rate(0), 3))
print("system B success rate:", round(success_rate(1), 3))
print("observed difference B-A:", round(observed_difference, 3))
print("bootstrap 90% interval:", (round(low, 3), round(high, 3)))
print("bootstrap Pr[B > A]:", round(probability_b_better, 3))

plt.hist(diffs, bins=17, density=True, edgecolor="black", color="#8ecae6")
plt.axvline(0, color="black", linestyle="--", linewidth=1)
plt.axvline(observed_difference, color="#d62828", linewidth=2)
plt.xlabel("difference in success rate (B - A)")
plt.ylabel("bootstrap density")
plt.tight_layout()
plt.savefig("paired_bootstrap.png")

system A success rate: 0.625
system B success rate: 0.775
observed difference B-A: 0.15
bootstrap 90% interval: (0.025, 0.275)
bootstrap Pr[B > A]: 0.953

Here system B is better on the observed test set by \(0.15\). The bootstrap distribution of differences is mostly to the right of zero, and the \(90\%\) bootstrap interval is approximately \([0.025,0.275]\). If the interval were centered near zero, the data would be saying: “the test set is too small to distinguish these systems reliably.”