Multiple Testing
— Run 30 Indicators and One Wins by Chance
When you test a single indicator, "a 5% chance it's random" looks low. But test 30 of them at once and the story changes. This is why so many "we found a significant signal" claims are an optical illusion.
The Structure of the Problem
A common threshold in statistical testing is "less than a 5% chance this result happened by chance." For a single indicator, that's a reasonable bar.
But what happens if you test 20 completely useless indicators against that same bar? Each one has a 5% chance of passing purely by luck, so on average, 1 of them passes — even though none of the indicators actually carries any information at all.
| Number Tested at Once | Chance at Least One Useless Indicator Looks "Significant" |
|---|---|
| 1 | 5% |
| 10 | 40% |
| 20 | 64% |
| 30 | 79% |
Run 30 tests and 8 times out of 10, at least one fake finding sneaks in. And people tend to remember and publish only the ones that "hit."
Common Forms This Takes in Investment Decisions
- Searching indicator combinations — tweaking RSI × moving average × volume conditions every which way and picking whichever combination performed best. With hundreds of combinations, one of them is bound to look good.
- Parameter tuning — "returns go up if you use 28 instead of an RSI of 30." Continually adjusting a value is also repeating the test.
- Trying different time windows — if it doesn't fit, adjust the window. This is also a way of increasing the number of attempts.
A rule found this way looks perfect in the past and falls apart in the future. That's because it was fitted to a random pattern in the data.
The Fix — Correction and Pre-Registration
① Multiple-Comparison Correction (BH/FDR)
This is a method that tightens the bar the more tests you run. DawnScan applies the Benjamini-Hochberg (BH) procedure to control "the share of fakes among everything published as significant." The ✓BH mark on the base-rate proof page is exactly this.
A signal that only passes ✓Wilson (its confidence-interval lower bound exceeds the base rate) and a signal that also passes the BH correction carry different levels of trust.
② Measure First, Apply Later
When DawnScan adds a new signal, it sets its weight to 0 and only logs it. Only once enough of a sample has accumulated and it passes testing does it get folded into the actual score. This breaks the cycle of "looking at the result and then rewriting the rule."
③ Requiring Consistency Across Multiple Labels
Looking at just one criterion makes it easy to pass by chance. That's why we require the same direction across all three of the peak, path, and final-return labels. It's much harder for chance to line up three times in a row.