Your Backtest Might Be Lying to You

A strategy that returns 200% in backtesting and loses money in live trading is not unusual. It's actually common. The reason is that backtesting introduces several biases that inflate performance, and if you don't account for them, you end up with a strategy that's perfectly optimized for the past but useless for the future. Understanding these pitfalls is the difference between building a robust strategy and building a mirage.

The good news is that once you know what to look for, these biases are avoidable. The bad news is that most traders don't know they exist until they've already lost money trading a "proven" backtest.

Curve Fitting: The Most Dangerous Trap

Curve fitting (also called overfitting) happens when you optimize your strategy's parameters so precisely that they match the historical data perfectly but have no predictive power for future data. It's like tailoring a suit to fit one specific mannequin: it looks perfect on that mannequin and terrible on every actual human.

Here's how it typically happens. You test a moving average crossover strategy and find that the 17-period MA crossing the 43-period MA with a 2.3 ATR stop and a 3.7 ATR target produced the best results on EUR/USD from 2019 to 2023. Those suspiciously specific parameters are a red flag. The strategy isn't capturing a market principle. It's memorizing the specific price movements of that period.

A robust strategy uses round, logical parameters. The 20-period and 50-period moving averages work because they approximate one month and one quarter of trading days. A 17/43 combination has no logical basis. If your backtested parameters are oddly specific, you've likely curve-fitted.

The fix: test your strategy with slightly different parameters. If a strategy works with a 20-period MA but falls apart with an 18-period or 22-period MA, it's curve-fitted. A robust strategy should work across a range of similar parameters, not just one magic combination.

Look-Ahead Bias: Using Information You Wouldn't Have

Look-ahead bias occurs when your backtest uses information that wasn't available at the time of the trade. This is sometimes obvious (using tomorrow's closing price to make today's decision) and sometimes subtle (using an indicator that needs several bars of future data to stabilize).

A common form of look-ahead bias is testing a strategy on a list of stocks that you selected knowing they performed well during the test period. "I'll test my strategy on Apple, Tesla, and Nvidia from 2020 to 2023" is cheating because you already know those stocks went up. Your strategy gets credit for a selection bias, not for its actual entry and exit rules.

Another subtle form: some indicators repaint. They calculate their values using future data and retroactively change their historical values. A supertrend indicator that repaints might show perfect signals on a historical chart, but in real time, those signals looked very different. Always check whether the indicators you use repaint before trusting your backtest.

The fix: test on stocks you selected before the test period (you can use constituent lists from a prior date) and verify that your indicators don't repaint. When in doubt, do your backtesting in real time by walking forward through the charts bar by bar.

Survivorship Bias: The Stocks That Disappeared

Survivorship bias happens when your test data only includes assets that survived to the present day. If you test a strategy on the current S&P 500 constituents going back 10 years, you're excluding all the companies that were in the S&P 500 ten years ago but have since been removed (due to bankruptcy, acquisition, or performance decline).

This bias inflates your results because the survivors are, by definition, the winners. The losers have been quietly removed from the dataset. A strategy that "would have bought" Enron in 2000 based on the current S&P 500 list wouldn't include Enron because it no longer exists.

Survivorship bias is particularly dangerous in stock screening strategies. If your strategy says "buy stocks above their 200-day moving average in the S&P 500," backtesting it on today's S&P 500 ignores all the stocks that fell below their 200-day MA and eventually got delisted. Your real-time results will include those eventual losers, but your backtest didn't.

The fix: use survivorship-bias-free databases (some data providers offer these) or, at minimum, be aware that your results are likely inflated by 1 to 3% annually due to this bias.

Small Sample Sizes: When 30 Trades Aren't Enough

A strategy that wins 8 out of 10 trades in a backtest has an 80% win rate, right? Technically yes, but statistically, 10 trades tells you almost nothing. Random chance alone could produce an 80% win rate over 10 coin flips. You need a much larger sample to know whether your edge is real.

The minimum meaningful sample size depends on your strategy's expected win rate. For a strategy that should win about 50% of the time, you need at least 50 trades to have reasonable confidence. For strategies with lower win rates (trend-following strategies often win 30 to 40% of the time), you need 100 or more trades.

Small sample sizes also hide drawdown risk. A 30-trade backtest might not include the worst-case losing streak that your strategy can produce. Over 300 trades, you're much more likely to encounter the kind of adverse conditions that stress-test your risk management.

How to Build a Backtest You Can Trust

Start with clear, simple rules and logical parameter choices. Test across multiple instruments and multiple time periods, including bear markets and choppy markets. Use at least 50 trades, preferably 100 or more. Hold back the most recent data as an out-of-sample test, meaning you don't use it during optimization and only test on it once, as a final validation.

Expect your live results to underperform your backtest by 20 to 30%. This is normal and accounts for slippage, execution delays, and the biases I've described. If your backtest shows a 30% annual return, plan for 20 to 25% in live trading. If a strategy only barely works in backtesting, it will likely lose money in real trading.

Document everything. Keep your backtest spreadsheets, your parameter choices, and your notes about why you chose specific rules. When you eventually trade the strategy live, track your results in TruthAlpha and compare them to your backtest expectations. If live results diverge significantly, revisit your backtest for the biases described here. Start free with TruthAlpha and maintain a clear performance record from backtest through live execution.