Search DevTools

Jump to any tool or page

Eval Confidence Interval

Put uncertainty around an AI or LLM evaluation pass rate. Balance sample size, observed rate, and baseline targets to determine whether an apparent improvement is statistically credible or noise.

Newwilson interval
Evaluation Parameters3 inputs

Number of evaluation cases evaluated

200 runs

Percentage of evaluation items passed

80%

Benchmark or incumbent rate to beat

70%
Pass rate clears the 70% baseline with 95% confidence.
80.0%160 passes of 200
Range: 73.9% – 85.0%
Observed Rate80.0%
Lower Bound73.9%
Upper Bound85.0%
Baseline Delta+10.0%
±6.1%Larger half-width

At 95% confidence, the true pass rate lies between 73.9% and 85.0%.

Uncertainty spans 11.0 percentage points across 200 examples. To cut this margin of error in half, roughly 4× as many evaluation samples (~800) would be needed.

Wilson score interval provides reliable coverage across both large and small sample sizes and remains properly bounded without collapsing at the extremes.

Method & Statistical Principles

Evaluating model performance on a finite test set is a binomial sampling experiment. An apparent 80% pass rate on 20 examples yields an interval of 58.4% to 91.9% at 95% confidence—meaning you cannot confidently assert it outperforms a 70% baseline. Only by scaling sample size or observing substantial margins can true model improvements be distinguished from stochastic variance.

Why is an eval score without sample size meaningless?

A pass rate is a point estimate of an underlying probability. Without sample size, there is no way to tell whether a reported 80% score represents 4 out of 5 tests (where uncertainty spans 35 percentage points) or 800 out of 1000 (where uncertainty spans less than 3 points).

Why does the Wald interval break down near 0% and 100%?

Wald assumes the sampling distribution is symmetric and normal. When all samples pass (k = n), the estimated variance collapses to zero, causing Wald to claim an interval width of exactly 0% (absolute certainty from finite samples). When k is close to n, Wald can even produce bounds greater than 100%.

What makes the Wilson score interval the recommended default?

Wilson inverts the score test under the true binomial variance. It never escapes [0, 100%], behaves correctly when k = 0 or k = n, and provides well-calibrated average coverage across both small and large sample sizes.

When should Clopper-Pearson be used?

Clopper-Pearson is the exact interval derived by inverting the binomial CDF test. Because of the discrete nature of binomial trials, its true coverage is guaranteed to never drop below the nominal level, making it the most conservative and widest interval.

How many samples are needed to cut the confidence interval in half?

Because the standard error scales inversely with the square root of n, narrowing the confidence interval by half requires four times as many evaluation examples (4× n).

Developer Utilities

About Eval Confidence Interval

Put a confidence interval around an AI or LLM eval pass rate and see whether an observed improvement clears a baseline benchmark. Pass rates reported without sample size hide stochastic noise; this tool calculates intervals under Wilson, Wald, and Clopper-Pearson methods to evaluate statistical credibility.

Related Developer Utilities tools

Token CounterZod GeneratorBase64 Encoder/DecoderSecret ScannerHash Generator

Frequently asked questions

Why is an eval score without a sample size not a result?
Because a percentage score without sample size conveys zero information about variance. An 80% pass rate from 20 evaluation examples has a 95% confidence interval spanning from 58.4% to 91.9%, which cannot reliably demonstrate superiority over a 70% baseline. The exact same 80% score from 1,000 examples narrows the interval to 77.4%–82.3%, clearly separating from the baseline. Sample size determines whether an apparent gain is signal or stochastic noise.
Why does the Wald interval break down at the extremes?
The Wald formula assumes the sampling distribution of a proportion is symmetric and normally distributed. When all examples pass (k = n) or all fail (k = 0), the observed variance p(1-p)/n collapses to zero, leading Wald to claim an interval of zero width (100.0%–100.0%) and asserting absolute certainty from finite evidence. Near the boundaries, Wald intervals can also extend past 100% or below 0%, which is mathematically impossible for binomial probabilities.
What does Clopper-Pearson buy and what does it cost?
Clopper-Pearson inverts the exact binomial test rather than approximating via a normal curve, so its actual coverage probability is mathematically guaranteed to never fall below the nominal confidence level (e.g. 95%). The cost is conservatism: because the binomial distribution is discrete, achieving the guaranteed minimum coverage requires widening the interval beyond the nominal rate, making Clopper-Pearson the widest and most conservative of the standard methods.
Why show a confidence interval instead of just a p-value?
A p-value only tells you whether you can reject the null hypothesis at an arbitrary threshold, saying nothing about effect size or practical uncertainty. A confidence interval communicates both the point estimate and the plausible range of true performance on the same scale as the metric itself (percentage points). It immediately shows whether a model's true capability might still be unacceptably low even if a null-hypothesis test passes.
How many examples are needed to detect a given gain?
Because the standard error of a proportion is proportional to 1/sqrt(n), halving the uncertainty interval requires multiplying the evaluation sample size by four. To reliably detect a modest 5% performance difference between a model and a baseline, several hundred to a few thousand test cases are typically required depending on the chosen confidence level.