The Difference Between Backtesting and Reality
A backtest is a necessary tool and an insufficient one. The trader who mistakes it for proof has not tested their system. They have tested their assumptions about their system on data that cannot challenge those assumptions.
A map is not the territory.
This is one of the oldest observations in epistemology, and one of the most consistently ignored in practice. The map is drawn from observation of the territory at a specific time, under specific conditions, by an observer whose instruments and methods captured some features of the landscape and omitted others. It is a representation, not a reproduction. The traveller who treats it as a reproduction will eventually walk into a discrepancy between what the map shows and what the ground contains, and the discrepancy will be most dangerous precisely where the map looks most authoritative.
A backtest is a map. The live programme is the territory. The distinction matters in every domain where maps are used, but it matters with particular severity in systematic trading, because the backtest is often the only evidence available for committing capital to a programme before the live results exist. The temptation to treat it as the territory rather than a representation of it is enormous, and the consequences of yielding to that temptation are specific and predictable.
This Foundation does not argue against backtesting. Backtests are necessary. A programme that has not been tested against historical data has no evidence at all about its statistical properties. The argument is about what a backtest can tell you and what it cannot, about the class of errors that consistently arise when the boundary between those two things is crossed, and about the discipline required to use the tool without being used by it.
What a Backtest of a Non-Predictive Structural System Is Actually Testing
Before discussing what a backtest can and cannot tell you, the question that determines those answers must be addressed: what is the backtest of an Outlier Hunting programme actually a test of?
Foundation 7 established a distinction that runs through everything that follows. A predictive system attempts to forecast where prices will be at some future moment, and its backtest is testing whether those forecasts were accurate. A non-predictive structural system reads structural features of how markets actually work as complex adaptive systems and responds mechanically to those features, and its backtest is testing something different: whether the structural mechanism the rules are designed to capture was operating in the historical period in the way the rules assume, and whether the rules’ response to specific terrain features produced the expected statistical properties.
A positive return on historical data therefore demonstrates two different things in the two cases. For a predictive system, it is evidence of forecasting accuracy. For a non-predictive structural system, it is evidence that the structural mechanism was present in the data and that the rules captured it. The two backtests look superficially similar but support different kinds of inference about the future.
The Outlier Hunting framework is a non-predictive structural system. Its rules read terrain features such as breakouts from compressed states, the development of trending regimes, and the alternation between feedback states described in the Fractals research. Every subsequent claim in this essay about what backtests can and cannot do refers specifically to backtests of this kind. The arguments would be different for predictive systems, and Foundation 8 is not addressed to traders running them.
What a Backtest Can Legitimately Tell You
A backtest applied correctly to a non-predictive structural system answers three questions that matter.
The first is whether the rules produce a positive geometric return on historical data. This is the minimum threshold of plausibility. A set of rules that cannot demonstrate a positive return across a substantial historical period, one covering multiple market regimes and at least one major adverse event, is not worth committing capital to. The backtest is the filter that eliminates the approaches that cannot meet this basic standard. The Fractals of Finance research, drawing on forty-one years of daily data across sixty-eight global futures contracts, provides the kind of historical sample against which this filter can be applied seriously, because it includes the trending regimes, the ranging regimes, and the regime transitions that any honest filter requires.
The second is the shape of the historical return distribution: the depth and duration of historical drawdowns in closed balance equity, the frequency and magnitude of winning periods, the relationship between the two. This information is not predictive in the sense of telling you exactly what the live programme will produce. It is diagnostic. It tells you the range of outcomes the rules have historically generated, which gives you a basis for the position sizing and Cut Back Rule calibration covered in Foundation 2. A programme whose historical record contains a forty percent maximum drawdown in closed equity should be sized and managed accordingly, not as though its worst outcome will be twenty percent because that is more comfortable to contemplate. The closed-equity figure is what matters here, because it is the realised compounding base. Drawdowns in total equity that include unrealised retracements on winning positions are the structural mid-life of outlier captures, as Foundation 7 established, and they are not what the position sizing is calibrated against.
The third is the sensitivity of the results to parameter variation. A programme whose results are robust across a wide range of parameter values, one where modifying the entry lookback from twenty days to twenty-five days produces similar return characteristics, is making a more durable claim about structural market behaviour than one whose results depend critically on a specific parameter setting. Fragile parameter dependence is a warning that the backtest has found a historical artefact rather than a structural property the system can rely on as the historical record extends.
What a Backtest Cannot Tell You
A backtest cannot tell you whether the rules will produce the same return characteristics on future data. This is the boundary that is most consistently crossed, and the crossing of it is so normalised in systematic trading practice that it often passes without comment.
The reason is straightforward. The backtest is constructed using the same data on which the rules were developed, either explicitly through parameter optimisation or implicitly through the iterative process of testing, observing, and refining that produces most systematic approaches. The rules have been selected, to some degree, because they worked on that data. Their performance on that data is therefore not a clean out-of-sample test. It is, to a greater or lesser degree, a measure of how well the rules fit the historical period they were developed on.
This is curve-fitting, and it exists on a spectrum. At one extreme, a programme optimised to maximise performance on a specific historical dataset with many free parameters has almost certainly found patterns that are specific to that dataset and will not persist. At the other extreme, a programme built from first principles on a structural theory of market behaviour, with few free parameters and robust performance across many historical periods, has a much weaker curve-fitting problem. But the problem never disappears entirely. Every backtest is conducted on finite historical data, and every historical period contains only the trunk events that happened to fall within it. Foundation 7 established that the trunk events of the market’s fractal structure progressively reveal themselves as the sample grows. The future will contain trunk events drawn from the same fractal distribution that produced the historical ones, but the specific events the future contains will not be the events the backtest contains. The backtest cannot test the system on events the historical sample did not include.
“The backtest tells you what the rules would have done. It cannot tell you what they will do. The distance between those two statements is where most systematic programmes eventually disappoint their builders.”
The Memory Problem
Markets have memory. This single fact invalidates the most widely used method of assessing backtest robustness.
Monte Carlo simulation, in the form most commonly implemented in retail backtesting software, takes the historical trade results of a system and re-randomises their sequence thousands of times, generating a distribution of possible outcomes from which confidence intervals around the backtest results are derived. If the worst simulated outcome across ten thousand re-randomisations is a drawdown of thirty-five percent in closed equity, the conclusion drawn is that the programme is unlikely to produce a drawdown worse than approximately that level.
This conclusion rests on an assumption that the trade results are independent of each other: that the outcome of trade number forty-seven carries no information about the likely outcome of trade number forty-eight. In a system with memory, this assumption is wrong. Trade results cluster. Winning trades arrive in sequences, driven by the trending regime that produced them. Losing trades arrive in sequences, driven by the ranging regime that produced them. The naive re-randomisation that this form of Monte Carlo performs destroys precisely the sequential structure that makes the distribution of outcomes in a real programme different from a shuffled version of the same outcomes.
The Fractals of Finance research provides the empirical case directly. The autocorrelation of absolute returns averages 0.353 at lag one across the universe and remains positive and significant for more than a year. A violent day leaves a trace that persists for months. A quiet day suppresses activity long after the calm has ended. The Hurst exponent averages 0.866 across the same universe, far above the 0.5 baseline that would describe an independent sequence. These are the statistical signatures of memory. The memory is not faint or anomalous. It is structural. And it is exactly what naive Monte Carlo simulation destroys when it re-randomises the sequence of trade outcomes.
A naive Monte Carlo simulation of a trend following programme will systematically understate the depth and duration of realistic drawdowns in closed equity, because it disperses the losing sequences that actually arrive in concentrated runs across many imagined simulations where they are diluted by randomly interspersed winning trades. The trunk-event framing from Foundation 7 makes this concrete. The trunk events that produce the largest drawdowns in real programmes are events in which losing trades cluster across markets and weeks because the same regime is producing the same kind of result across the universe simultaneously. Naive Monte Carlo takes those clustered events and scatters their components across thousands of randomised sequences in which the clustering is broken. The output looks more comforting than reality. The protection it describes is not real.
The validation method must preserve the order, clustering, and regime structure of returns. Walk-forward analysis does this directly, by testing the rules on genuinely out-of-sample data, one period at a time, in the sequence those periods actually occurred. Block bootstrap and stationary bootstrap are sophisticated alternatives that attempt the same preservation through other means. The naive per-trade reshuffling that most retail backtesting software performs is not in this category, and it is the version of Monte Carlo this essay is critiquing. The walk-forward, however, comes with its own discipline, and the next section addresses why that discipline matters.
Walk-Forward Is a Test, Not a Loop
Walk-forward analysis has a specific structure. The historical data is divided into a development period and one or more validation periods that the development process has not seen. The system is built on the development period, with all parameters set and all rules finalised. The validation periods are then used to test the system as built, in chronological order, with no further modification of the system based on what the validation reveals.
This is not how walk-forward is most commonly used in systematic trading practice. The common pattern is to run the walk-forward, observe the result, return to the development phase, modify the system in response to what the walk-forward revealed, and run the walk-forward again. Each iteration of this loop is presented as a test of the modified system on out-of-sample data, but it is not. The structural principle is simple: once validation results influence development, that validation data is no longer out-of-sample. The bias is structural rather than behavioural. It does not require the trader to deliberately overfit. It requires only that they iterate, and the iterative use of the validation set is itself the contamination.
This is the cardinal error of systematic trading practice. The trader has not validated the system on held-out data. They have used the held-out data as additional development data, under the appearance of validation. By the second iteration, the developer has learned from the first walk-forward’s results. By the third, the validation data has been substantially absorbed into the development process. By the fourth, there is very little genuinely uncontaminated data left.
In practice, this means the walk-forward functions as a single test rather than a development loop. Its purpose is to give the developer confidence that the system, as it currently exists, has acceptable properties on data the development process has not seen. The result is information about the likely behaviour of the system as it stands. It is not information to be acted on through further system modification.
If the walk-forward result is unacceptable, the appropriate response is to recognise that the system as developed does not have the properties the developer hoped for, and either commit to the system with realistic expectations of its actual behaviour or abandon the system entirely. There is no third option that involves further development on the same validation data. The walk-forward sample is now used data. It cannot be re-used as out-of-sample for a modified system, because the developer has already learned from it.
There is also a connection to Foundation 7’s trunk-events framing worth making here. The trunk events the historical sample contains are finite. Each iteration of test-modify-test consumes more of the structural information in those trunk events for development purposes. A trader who has iterated extensively has very little genuine out-of-sample test material left, even if they have a long historical dataset, because the trunk events in that dataset have been effectively absorbed into the development process. The walk-forward as a single test preserves the structural information. The walk-forward as an iterative loop destroys it.
Non-Stationarity and the Regime Problem
Markets are non-stationary. The statistical properties that describe their behaviour in one period do not reliably describe their behaviour in the next.
Volatility regimes shift. Correlation structures change. The specific markets that trend most powerfully in one decade are not necessarily the ones that trend most powerfully in the next. The relationships between asset classes that held during one monetary regime are restructured when that regime ends. These are not random fluctuations around a stable mean. They are genuine changes in the generative process that produces market returns.
The Fractals research on the post-2009 period demonstrates this concretely. The decline in simple symmetric trend following returns during the 2009 to 2020 quantitative easing era was not a structural decay of the underlying feedback mechanism. The structure that produces trends remained intact across the period. What changed was the macro environment: the variance ratio compressed during QE and ZIRP, starving divergent strategies of the raw material they require. When the macro environment delivered genuine trends again during the 2020 to 2022 inflation, every divergent strategy in the test recovered to pre-QE quality. A backtest covering only the pre-QE period would not have warned the trader about the QE-era performance suppression. A backtest covering only the QE era would have led the trader to abandon a strategy whose underlying mechanism was undamaged. Each backtest was true on its sample. Neither was sufficient on its own.
A backtest conducted on data from a specific historical period has, by construction, been exposed only to the regimes that occurred during that period. The programme’s properties in regimes that did not occur during the backtest period are unknown. This is not a problem that can be solved by extending the backtest to more historical data, though that helps at the margin. It is a fundamental feature of applying rules derived from a finite historical sample to a non-stationary system that will continue to produce new regimes indefinitely.
The practical response is not despair. It is appropriate humility about what the backtest has established, combined with programme design that is robust to regime change rather than optimised for a specific historical regime. A programme with few parameters, structural rather than empirical justification for its rules, asymmetric calibration of long and short signals as Foundation 3 established, and genuine diversification across asset classes and time frames is more robust to regime change than one with many parameters, historically fitted rules, and concentrated exposure. Foundation 1’s emphasis on simplicity, Foundation 3’s emphasis on maximum diversification, and Foundation 4’s distinction between structural edge and specific implementation are all, in part, responses to the non-stationarity problem.
The Correct Use of the Backtest
Given all of this, what should a backtest actually be used for?
It should establish the plausibility of the approach: confirm that the rules produce a positive return across a substantial historical period that includes adverse conditions, and characterise the shape of the return distribution well enough to calibrate position sizing and drawdown management on the closed-equity figure that Foundation 2 named as the compounding base.
It should map the tails: understand, as well as possible from the available data, what the worst outcomes of the approach have historically looked like. Not to treat those worst outcomes as the ceiling of future risk, which Foundation 7 addressed directly, but to ensure the programme is designed to survive outcomes of at least that severity.
It should test robustness: confirm that the results do not depend critically on specific parameter settings, specific historical periods, or specific markets. A robust backtest is one where the results degrade gracefully as parameters are varied, rather than collapsing sharply at the boundaries of the optimised range.
It should be used once for final validation: a single walk-forward on data the development process has not seen, with the result accepted as information about the likely properties of the system as built rather than as feedback for further development.
It should not be used as a performance guarantee. The live programme will deviate from the backtest. The deviation is not a sign of failure. It is the expected consequence of applying rules derived from finite historical data to a non-stationary system that will continue generating new regimes and new trunk events indefinitely. The live programme that performs differently from the backtest is behaving normally. The trader who expected the live programme to replicate the backtest has misunderstood what the backtest was.
A backtest does not prove that a programme will work. It shows whether the programme’s rules were compatible with the structural conditions contained in the historical sample. That is a useful piece of evidence. It is not a guarantee, and treating it as one is the mistake this essay has been about.
“The backtest is a stress test for your assumptions, not a preview of your results. Use it as the former and it will serve you. Use it as the latter and it will deceive you.”
What the Bears Know
The Trader and the Three Bears, developed in the site’s own framework, offers a useful checklist for three categories of backtest optimism. Each category represents a specific way the map has been drawn that systematically misrepresents the territory in a comforting direction.
The first bear is overfitting: rules that work on the historical data because they fit its specific contours rather than because they capture structural properties of market behaviour that will persist. The map records features that are accidents of the survey rather than properties of the ground. The warning sign is strong performance that requires precise parameter values and fails to generalise.
The second bear is favourable sample selection: a backtest conducted on a period that happened to be particularly good for the approach, then presented as representative. The map accurately represents one favourable patch of terrain that was selected because it was favourable. The rest of the territory may not resemble it.
The third bear is survivorship: building the programme on markets that exist today and omitting the markets that existed during the period in question but no longer do. The surviving markets have already demonstrated a property the full universe did not share. The map shows only the surviving terrain, not the territory as it actually was.
Each error produces a backtest more encouraging than the live programme will be. Each is avoidable with care.
What Comes Next
Foundation 8 has established what backtesting is and what it is not: a necessary test of plausibility, a tool for mapping risk in closed equity, a diagnostic for robustness, a single validation of out-of-sample behaviour, and an insufficient guide to future performance. The programme that has been correctly backtested has passed the minimum threshold of seriousness. It has not been proved.
Foundation 9 turns from the historical simulation to the live record. The equity curve that the programme produces in real time is a different kind of document from the backtest chart. It contains information that the backtest cannot carry: about how the rules perform in the specific regime of their live operation, about how the trader is managing the programme, and, for the Outlier Hunter specifically, about whether the trunk events the entire architecture was designed to capture are actually arriving and whether the architecture is capturing them as designed.
READ DEEPER
→ The Death of Backtesting: Why Historical Simulations Are Not What You Think
→ The Illusion of Time: Why Backtests Lie About the Future
→ The Trader and the Three Bears: Three Backtest Errors That Will Cost You
→ The Pitfalls of Monte Carlos: Why Re-randomisation Misleads Trend Followers
→ The Fractals of Finance: The Nile River’s Secret
→ The Fractals of Finance: The Paradox
Previous: Foundation 7: What a Drawdown Actually Means | Next: Foundation 9: How to Read an Equity Curve
Want the theoretical foundation for why trend following works?
The Fractals of Finance: Determinism, Adaptation and the Geometry of Markets
The book explores the full architecture of feedback, fat tails, and fractal structure in financial markets, and what it means for how we trade, invest, and understand risk.
Available now on Amazon in paperback, hardcover, and Kindle.
Want a practical field manual for trading trends and capturing outliers?
The Aussie Turtles Trend Following Guide: A Field Manual for Hunting Outliers adapts the timeless principles of the original Turtle traders into a systematic, rules-based approach for modern markets. Co-authored with Adam Havryliv.
Available now on Amazon in paperback, hardcover, and Kindle.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.Lorem ipsum dolor sit amet consectetur adipiscing elit dolor