Backtesting Abuse: When Historical Data Lies

Backtesting is a legitimate and standard part of strategy development — but Felix Bick’s How Not to Get Scammed by AI Trading Apps dedicates Chapter 25 to explaining why it’s also one of the easiest tools in finance to abuse.

Bick explains the core problem clearly: a strategy can be adjusted repeatedly until it produces an attractive-looking historical result, a process that guarantees good-looking past performance while providing no assurance about future performance, since the strategy was effectively reverse-engineered to fit data that has already happened.

The book identifies specific ways this abuse compounds: backtests run over short historical windows, cherry-picked to exclude major drawdown periods, or run without accounting for realistic transaction costs, slippage, and liquidity constraints that would apply in live trading. Each of these choices can make a mediocre or genuinely unprofitable strategy look considerably better on paper than it would ever perform with real money.

Bick offers readers a genuinely useful heuristic to apply immediately: any backtest presented without corresponding out-of-sample, forward testing on data the strategy wasn’t developed against should be treated as marketing material rather than evidence of a working strategy, regardless of how sophisticated the underlying analysis appears.

This chapter is one of the more technically substantive in the book, and it rewards careful reading for anyone who takes trading products seriously. Understanding the distinction between a backtest and genuine forward-tested performance is exactly the kind of knowledge that separates readers who can meaningfully evaluate a trading product’s claims from those who can only react to how professional the presentation looks.

How Not to Get Scammed by AI Trading Apps treats backtesting with real technical seriousness in this chapter, giving Felix Bick’s readers a genuinely sophisticated tool for cutting through one of the fraud industry’s most common tactics.