Backtesting Systems & Biases

What is backtesting?

A trading idea is not evidence. Before risking capital, a quant must answer one question objectively: does this signal actually work? Backtesting is how we answer it.

The point is to replace gut feeling with measurement: estimate the return, study the drawdowns, and surface weaknesses early — while they are cheap to fix.

Research vs production

These are two different worlds with different standards. Research is experimentation: cheap, fast, tolerant of failure. Production trades real capital, so it demands far stricter controls, monitoring and reproducibility.

The big misconception

This is the single most important lesson of the module: a stunning historical result can be entirely an artifact of bias. The rest of this page is a catalogue of the biases that fabricate performance — and how to defend against them.

flowchart TB
  BT([Beautiful Backtest])
  BT --> LA[Look-ahead bias]
  BT --> SV[Survivorship bias]
  BT --> OF[Overfitting]
  BT --> DL[Data leakage]
  BT --> FR[Ignored frictions]
  LA & SV & OF & DL & FR --> FALSE([Inflated, false confidence])
  classDef s fill:#eef2ff,stroke:#4f46e5,stroke-width:1px,color:#3730a3;
  classDef d fill:#ecfdf5,stroke:#0d9488,color:#0d9488;
  class LA,SV,OF,DL,FR s
  class BT,FALSE d
      

Look-ahead bias

This is the most dangerous backtest bias because it is the easiest to introduce by accident and it produces spectacular, completely fake results.

A concrete trading example

Suppose a stock closes Monday at 100 and Tuesday at 110. If your backtest "buys on Monday because it knows Tuesday is higher," it has used Tuesday's price to make Monday's decision. On Monday that price did not exist yet. The 10% gain is invalid — it could never be captured live.

Look-ahead bias in code

In practice this bias usually hides in a single misaligned line. When a signal is computed from the same bar's data, multiplying it directly by that bar's return lets the position "know" the outcome it is supposed to predict.

# BAD — signal and return share the same timestamp (peeks into the present)
pnl = signal * returns

# GOOD — act on the PREVIOUS bar's signal, only after it is known
pnl = signal.shift(1) * returns

How it happens and how to prevent it

Industry blowups trace back to mundane causes: timestamp mistakes, data-availability errors, and the false confidence they breed. The defenses are equally concrete.

Survivorship bias

Overfitting

Markets contain genuine randomness, and humans see patterns everywhere — even in pure noise. A model with enough freedom will happily memorize that noise.

Train vs test: checking generalization

The defense is to split the data: train the model on one sample, then evaluate on an unseen sample. If performance holds up on data the model never saw, the edge is more likely real. The professional pattern uses three splits.

flowchart LR
  D([Historical Data]) --> TR([Train
fit the model])
  TR --> VA([Validate
tune parameters])
  VA --> TE([Test
final unseen check])
  TE -->|Holds up| OK([Generalizes ✓])
  TE -.->|Collapses| BAD([Overfit ✗])
  classDef s fill:#eef2ff,stroke:#4f46e5,stroke-width:1px,color:#3730a3;
  classDef d fill:#ecfdf5,stroke:#0d9488,color:#0d9488;
  class D,TR,VA,TE s
  class OK,BAD d
      

Data leakage

The reality problem: market friction

Even a perfectly clean, bias-free strategy faces a final hurdle. Backtest profit is not real profit, because trading in the real world costs money.

Transaction costs

Slippage

Slippage is tied to the order book: liquidity sits at different price levels, and a large order eats through several levels, getting progressively worse fills. This drives two further constraints:

The backtest checklist & professional workflow

Every credible backtest clears the same gauntlet before anyone trusts its number.

flowchart LR
  A([Idea]) --> B([Research])
  B --> C([Backtest])
  C --> D([Validate])
  D -->|Passes checklist| E([Deploy & Monitor])
  D -.->|Fails checks| B
  classDef s fill:#eef2ff,stroke:#4f46e5,stroke-width:1px,color:#3730a3;
  classDef d fill:#ecfdf5,stroke:#0d9488,color:#0d9488;
  class A,B,C,D s
  class E d
      

Key takeaways