Lift and Confidence Intervals
— How to Tell If a Signal Is Real
Which is more trustworthy: "60% hit rate, a sample of 20" or "42% hit rate, a sample of 1,400"? Understanding lift and confidence intervals lets you answer this. We break it down with no formulas.
Lift — How Much Better Than the Baseline
Lift is simple.
Lift = Hit rate with the signal − the base rate
If the base rate is 30% and a signal's hit rate is 42%, the lift is +12%p. That +12%p is the signal's actual contribution — not the raw number 42% itself.
If the lift is close to 0, that signal is giving you essentially no information. If it's negative, you'd actually be better off without it.
Confidence Interval — Can You Trust That Number?
The problem is that with a small sample, lift can come out large from luck alone.
If you flip a coin 10 times and get heads 7 times, you don't conclude "this coin lands heads 70% of the time." With only 10 flips, that kind of spread is common. A hit rate works exactly the same way.
A confidence interval tells you "accounting for sampling error, the true value is likely somewhere in this range." The smaller the sample, the wider the range.
| Observation | Hit Rate | 95% CI (approx.) | Verdict |
|---|---|---|---|
| 12 hits out of a 20-stock sample | 60% | 39% ~ 78% | Could overlap with the 30% base rate → hold judgment |
| 588 hits out of a 1,400-stock sample | 42% | 39% ~ 45% | Clearly beats the 30% base rate |
Looking at the hit rate alone, 60% looks better — but the trustworthy one is 42%. A 60% from a 20-stock sample wouldn't be strange to see land at 35% over the next 20.
DawnScan's Decision Criteria
The base-rate proof page displays these three things together.
- Sample size — under 100 gets flagged as "insufficient sample" and grayed out. It's shown for reference only and isn't used as grounds for a judgment.
- 95% CI lower bound — a conservative lower bound computed using the Wilson method. This value has to be above the base rate for it to be treated as more than chance.
- ✓BH — shows whether it also passed the multiple-testing correction.
All three conditions have to pass for something to be treated as a "significant signal." Passing just one makes it only a candidate.
How Large a Sample Is Enough?
There's no single right answer, but you can get a feel for it. To reliably distinguish a difference of roughly +10%p at a 30% base-rate level, you generally need a sample in the hundreds.
That's why DawnScan sets its minimum bar at 100, and even that isn't used to mean "enough" — it's used as the line meaning "you at least need to clear this before the discussion even starts." Once you also apply deduplication of repeat observations on top of that, the actual effective sample shrinks further.