🏠 Home 🔍 Today's Scan 📰 Daily Report 📈 Hit Rate 📊 Base Rate ❓ Methodology 📚 Learn 🪙 Crypto Scanner 📋 All Tools ⏪ Investment Simulator 🧾 Tax Calculator🧮 Pension vs. Direct ⚡ Leverage ⚖️ Rebalancing 💹 DCA 📉 Averaging-Down B/E 💰 Dividend Calendar
← Back to the Learn hub
📉 Backtesting · Verification

Backtest Overfitting
— Why a Strategy That Was Perfect in the Past Falls Apart

You see a backtest result like "400% return over the last 3 years." But run it for real and the performance is completely different. That's because explaining the past well and predicting the future are two different things.

Written by Dawn · IT Engineer · Published
💡 Key takeaway — Keep tuning a rule to fit the data, and you eventually memorize even that data's random noise. New data doesn't have that same noise, so the rule falls apart.

What Overfitting Is

It's a state where a rule has been tuned to fit historical data too precisely.

Think of it like studying for an exam by memorizing the answers to past exam questions outright. You'd get a perfect score on that exact test, but you couldn't solve a new question. Data mixes together genuine regularity and randomness unique to that particular stretch, and fitting too finely ends up learning the randomness too.

How It Happens — the Common Path

  1. You build a strategy and run it on historical data. The result is mediocre.
  2. You tweak a parameter slightly. RSI 30 → 28. Returns go up.
  3. You add a condition. "Only when volume is 2x or more." It goes up more.
  4. You adjust the stop-loss width. It goes up again.
  5. The finished strategy's historical return is excellent.

The problem is steps 2 through 4. You looked at the result and changed the rule. From that moment on, the backtest stops being verification and becomes a fitting exercise. The more attempts you make, the more chances you get for something to look good purely by luck (see multiple testing).

⚠️ Warning Signs — The more parameters there are · the more elaborate the conditions are · the smoother the return curve is, the more you should suspect overfitting. A rule like "when RSI stays under 27.5 for 3 or more days and volume is 2.3x" is usually the result of something carved out to fit one specific stretch.

In-Sample and Out-of-Sample

  • In-sample — the data you looked at while building the rule. Performance here can always be made to look good, so it isn't evidence.
  • Out-of-sample — data you're seeing for the first time after the rule is locked in. Performance here is the real thing.

But out-of-sample data also gets contaminated if you look at it more than once. Repeating "that didn't fit, let me tweak the rule and try again" eventually turns that stretch into in-sample data too. It only means something if you use it exactly once.

The Most Reliable Verification — Forward Testing

Lock in the rule publicly first, then confirm it against data that arrives afterward. Since data that doesn't exist yet can't be fitted to, self-deception becomes impossible.

This is why DawnScan logs a new signal at a weight of 0 only, and decides whether to fold it in only after a sample has built up. The measurement standard gets set first, and whether the data clears that bar gets checked afterward.

  • The decision criteria (minimum sample size, confidence interval, correction) get locked in in advance.
  • If it falls short, it's held back, not passed by lowering the bar.
  • Held-back signals and the reason are published too.

What to Ask When Looking at a Backtest Result

  1. How many times was a parameter adjusted? — the more times, the more chance is mixed in.
  2. Is there performance from after the rule was locked in? — if not, it hasn't actually been verified yet.
  3. Did it account for trading costs and slippage? — for a small-cap strategy, this alone can flip the result.
  4. Did it include delisted stocks? — excluding them inflates the return.
📌 Summary — A historical return is a hypothesis, not evidence. Evidence only comes from what happens after the rule is locked in. So "when was the rule locked in" is a more important question than "how much did it make."
📮 Daily US Market Morning Brief — We send an analysis of the previous day's top 10 US gainers (TOP10) and what they had in common, every day at 8am (KST). Telegram @dawnbrief · Free · No ads · Not stock recommendations.

Frequently Asked Questions

What is overfitting?

It's a state where a rule has been tuned to fit historical data too precisely, ending up learning even that data's random noise. It's like memorizing the answers to past exam questions — great performance on the past, but it falls apart on new data.

Is adjusting a parameter not allowed at all?

Adjusting itself isn't the problem — the problem is looking at the result, adjusting based on it, and then presenting that result as your track record. Every adjustment amounts to a new test, increasing the chance of something looking good purely by luck. If you did adjust it, you need to re-confirm it using new data from after the rule was locked in.

What's the difference between in-sample and out-of-sample?

In-sample is the data you looked at while building the rule — performance here can always be made to look good, so it isn't evidence. Out-of-sample is data you see for the first time after the rule is locked in. That said, out-of-sample data also turns into in-sample if you repeatedly look at it and tweak the rule, so it should only be used once.

What's the most trustworthy verification method?

Forward testing. You lock in the rule and the decision criteria first, then confirm them against data that arrives afterward. Since a rule can't be fitted to data that doesn't exist yet, self-deception is fundamentally impossible.
🎲
Multiple Testing
The illusion created by the number of attempts
📊
Base-Rate Proof
Results measured after locking in criteria in advance
❓
Methodology
The measure → verify → apply order

Related Reading