Ten thousand configurations against the same history and the best of them will look extraordinary whether or not a single one of them contains an edge. This is the central problem of quantitative research, and overfitting a trading strategy is what happens when you mistake the winner of that search for a discovery.
We have run more than 65,000 backtest configurations across equity indices, Bitcoin, gold, volatility products and FX. The overwhelming majority were archived with a written verdict. This article is about the machinery that does the archiving — because the metric that predicts the quality of a research process is not its best curve. It is its rejection rate.
The anecdote that explains everything: the model that learned the clock
Early in the project, one model Run enough backtests and you will find a beautiful equity curve. That is not an opinion about markets; it is arithmetic. Search looked exceptional in validation. Its feature importance told the real story: a single input — a cyclical encoding of the hour of day — was absorbing around two thirds of the model’s total importance.
The model was not predicting the market. It was memorising when profitable trades had historically happened in that specific sample, and dressing the memory as a signal. Out of sample it collapsed, exactly as it had to.
Removing that one feature raised validation SQN from 0.88 to 1.66 in a single change. The lesson generalises far past the specific input: any feature that lets the model index the training set rather than describe market state is an overfitting vector, and importance charts are where they hide in plain sight. Calendar encodings are the classic case. They are now permanently excluded from our tree models.
Why the top of the ranking is almost never a system
The most expensive habit in retail quant research is picking the top row of a sorted results table.
In our latest equity index campaign, the best trial by validation out of 5,880 showed runaway in-sample memorisation — a discriminative power of 22.8 on training data — with a stellar validation score of 4.89. On the hidden test block it scored 0.58.
Nothing about that is a surprise once you understand what a ranking is: with thousands of configurations searched against one validation split, the extreme of the distribution is chosen partly for skill and mostly for noise. It is a lottery with a leaderboard. We discard rank-1 by rule, not by intuition.
The inverse trap is just as lethal and much better disguised: scanning the test column and “discovering” trials that were mediocre in validation but brilliant out of sample. We measured this profile precisely — trials with validation 0.3–0.9 and test 1.5+ shared the same ascending gradient toward the test block, the statistical signature of having won the lottery in the one partition nobody used for selection. Choosing them turns the test into a second validation and leaves you with no virgin data at all.
The house rule admits no exceptions: selection is always by validation; the test score of a trial chosen for its test score is not evidence.
Four defences against backtest overfitting
Backtest overfitting is not solved by one technique. It is survived in layers, and a candidate has to pass all of them.
1. Hunt plateaus, not peaks
An isolated maximum in parameter space is almost always an artifact. A region where many neighbouring configurations work is a property of the market. Our neighbourhood gate takes a candidate and examines every trial in the study within ±25% of its barrier parameters and ±0.03 of its decision threshold, requiring at least 60% of those dense neighbours to be positive in validation.
The candidate that eventually passed everything sat in a cluster of four neighbouring configurations sharing the same eight features — atr_ratio, dist_session_open, dist_swing_high_12, dist_swing_high_24, rs_plus, rsj, rv_ratio, upper_wick_norm — and its neighbourhood delivered 54 of 61 positive neighbours — 89% — with a median of +1.33. When the same test was applied months earlier to candidates that later failed, the numbers were nowhere close.
2. Reproduce before you publish
This one we learned from a scare. A candidate’s grid cell recorded a test SQN of 2.23. When the frozen model was rebuilt to generate its system sheet, the same configuration produced 1.11.
We investigated everything. Training non-determinism? No — twelve retrains identical to the third decimal. The partition boundary? Sliding it across an entire quarter never produced the recorded figure. The cause was traced to data-boundary sensitivity in how the original cell had been evaluated. The model was fine. The number was not.
The published figure for that system is the reproducible one, around 1.17 — not 2.23. A methodology that preaches reproducibility and then publishes its unreproducible best number has no methodology at all.
3. Monte Carlo the sequence
Historical order is one draw. We resample it: skip trades at random, reorder blocks, and require the 5th percentile of the resulting distribution to remain positive with no negative outcomes at the tail we care about. A strategy whose profit depends on three trades landing in a particular sequence is not a strategy; it is an anecdote with a Sharpe ratio.
4. Stress the costs
Every result is re-run at triple the modelled transaction costs and must survive. This defence has killed more candidates than any other, and it is the one retail backtests skip most often. One volatility-product campaign was closed on this alone: the platform’s modelled spread was 0.018% against a real 0.61% — a factor of thirty-four that turned an apparently excellent system into an arithmetic impossibility. On another instrument, a one-hour daily window carried spreads 20 to 50 times normal; the search had been quietly harvesting that artifact until it was masked.
The graveyard is the asset
We keep a written cemetery of everything that failed, and it is genuinely the most valuable document in the project.
Entire instrument campaigns are in it: gold across three independent campaigns with no deployable edge; a volatility product across five variants; Bitcoin across nine timeframes and roughly 30,000 trials, all archived. So are individual features — momentum oscillators, fractal exponents, band positions, wick asymmetries, various z-scores — each with a note on how it was measured and why it lost.
The cemetery does two things. It stops the project relitigating settled questions, and it makes the survivors mean something. A candidate that has passed a filter which has rejected 60,000 alternatives is a different object from a candidate that is simply the best thing you happened to try.
What a candidate looks like when it finally passes
For completeness, the profile of the one that got through: a cluster of four neighbouring configurations on an equity index, the eight features named above, 89% positive dense neighbours, 13 of 15 walk-forward windows positive with pooled SQN between 1.04 and 1.80, Monte Carlo 5th percentile positive with zero negative outcomes, and survival at triple modelled costs. Published figures are the reproducible ones.
That is not a promise of returns. It is a candidate that has failed to be eliminated by every test we know how to run — which is the most any honest process can claim, and considerably more than a good-looking equity curve claims.
FAQ
What is the difference between overfitting and curve fitting?
In practice they describe the same failure from different angles. Curve fitting usually refers to tuning parameters until the historical equity curve looks good; overfitting is the statistical consequence — a model that has learned the sample rather than the process. Data mining bias is the third face: the distortion introduced by the number of things you tried.
How many backtests are too many?
There is no fixed number, because the problem is not the count but whether the selection is corrected for it. Search widely and then demand that survivors clear plateau, reproducibility, Monte Carlo and cost-stress gates, and a large search becomes an asset instead of a liability.
Does walk-forward analysis solve overfitting on its own?
No. It is the most important single filter, but it can still be gamed by re-tuning inside the windows or by running it after the search has already consumed the same data.
Where can I see the results this process produced?
The metrics, the rejected candidates and the live figures are in the methodology overview.
This article describes methodology and aggregate results. Full configurations, trained models and exact execution parameters are not published. All backtest and walk-forward figures include the execution venue’s real commissions and constitute neither investment advice nor a guarantee of future performance.