In today’s efficient marketplace the retail trader is spoilt for choice. What was previously the domain of the ‘chosen few’ is now the domain of literally every man/woman and their dog, thanks to improving technology, education and affordability. I spend a lot of my time down ‘t’ut pit in the world of data mining overseeing my strategy factories that pump out thousands of automated strategies per week from the interrogation of market data using random generation and genetic algorithms replete with an entire library of statistical performance metrics. The process is akin to my prior work as a geologist assaying thousands of core samples to extract any pertinent information that may lead to a lucky strike.

The programmatic skills of the developers of this software are a site to behold, but it is very easy to simply lie back on the couch and accept that these processes are simply going to generate your fortunes without engaging that wonderful brain of  yours to sift the chaff from the wheat…..and therein lies the problem.

Despite the power of technology today…. sometimes the old ways are best. Like using that most magnificent tool that we have in our skulls that is taken for granted and left wanting as we salivate over the next technological release of data-mining software with new metrics and new features…’cause eleventy billion iterations is just not enough to find that signal in the noise.  Ok…I might be a bit tongue in cheek here…..but you get the point.

Most of us, even with this technology at our disposal are still left scratching our heads finding the robust solutions that can confidently project into an uncertain future and after all is said and done are left with a sifted array of strategies that simply fall off the edge of despair as they are taken to the live trading environment…….however ‘most of us’ in our strategy selection process fall into the very traps of human bias that we are trying to avoid through the application of these systematic processes.

How can this be? We have let the data-mining software make all the decisions and all we have done is select the ‘best performers’ with the optimal equity curve? I have highlighted the elephant in the room tucked within this statement that is riddled with selection bias. We have fallen into the trap of focusing on the outcome of the processes involved as opposed to understanding what is trying to be achieved and how the processes give rise to that outcome.

Guess what? Data-mining is all about ‘curve fitting the data’ so if you complain that your chosen strategy is ‘curve fitted’ but from random data points and subsequently has no future forecasting potential ….then you are yourself to blame. You have simply forgotten to use that brain of yours and spend a bit of effort looking under the hood and understanding what is trying to be achieved here. If you had mapped your return streams to the market condition you would have instantly seen that the results had no correlation with the strategy design logic.

We don’t need to navigate into current complex measures of ‘curve fitting treatment’ such as Walk forward optimisation or even Monte Carlo testing when data-mining for divergent strategies (eg. momentum/trend following). These intensive statistical techniques are more suited for detecting convergent strategies and lurking within convergence lies the inevitable tail risk that we as trend followers want to avoid. In detecting divergence, you need to use the visual power of that overlooked piece of software called your brain.

What do I mean by this? Let’s dig into this a bit deeper.

We can classify any trading strategy into two broad classes. Convergent strategies or divergent strategies. Nearly all the retail traders on this planet focus on the former class of strategy as it makes intuitive sense to them and the guise of apparent success reinforces convergent behaviour.

Convergent Strategies – The problems of assuming predictability

Convergent strategies relate to the risk profile associated with ‘predictable outcomes’. It is a backwards looking philosophy that assumes that the current market condition will persist into the future. For example mean reverting strategies fall into this category based on the principle that there is a predictable historic mean around which future price will oscillate. As a result, when price has overextended itself either above or below this predictable average, then it will tend to revert in the future towards this average. Now in applying this philosophy in the real world, it certainly does work for extended periods of time given the leptokurtic skew of real market distributions and we harness this market principle in our data mining efforts looking for strategies with a high win rate with an associated nice linear equity curve of predictability….but we tend to pay tittle attention to the associated adverse fat tail of that same distribution.

Why the high win rate? Well there can be a few factors at play here that you need to be aware of. The intent of the high win rate in convergent philosophy is to find the outcome that is more repeatable than a random normal distribution would imply. That makes sense as this philosophy assumes that price is more likely to revert to a mean than continue on in it’s excursion away from this condition or simply random walk, hence should be more predictable than a pure random outcome…..but this philosophy is also riddled with assumptions.

Convergent traders like to be more right than wrong as this is a sign of predictability as opposed to uncertainty. It also feels good psychologically to be more right than wrong which helps to influence or bias the trader towards convergence….but convergent signatures can also be created in purely random data. For example, if you establish a short profit target with a large stop far away, the convergent signature is guaranteed however you can probably now see why your data mining selection of that great strategy with the linear equity curve falls off the cliff the minute you deploy it.

So when we have an equity curve signature that is nice and linear and has a high win rate we still have not been able to “assay this data sample” correctly. It can still be riddled with fools gold. There may be some equity curve signatures that are truly representative of convergent market conditions, but it is far more likely that they are still full of chaff. By far the greatest number of convergent outcomes arising in the vast collections of datamined samples are simply a result of relative entry and exit condition placement as opposed to their faithful representation of trading convergent market conditions.

The way around this little nugget of advice is to firstly test for convergence with no initial stop. That allows you to more faithfully data-mine for strategies that hold up in convergent market conditions. Once found, then you need to get out of your armchair and apply risk mitigation measures such as stops of last resort onto these winnowed solutions…..however despite this effort, this then kicks the issue further down the road where you will then step into the problem of negative skew.

Applying stops of last resort in convergent solutions cannot protect you from a series of unfavourable events. Given the need to place these stops with a wide berth to allow the strategy to more faithfully ride the convergent market condition, they need to allow for negative skew to deliver positive expectancy…..and without going into the pitfalls of negative skew again….I defer the reader to my prior article on this subject.

Furthermore the assumption of convergence is couched in terms of predictable market conditions. What happens when those conditions unpredictably change? For example, you have a convergent strategy that trades mean reverting market conditions that are as predictable as clockwork. Your win rate is soaring, your equity curve is rising like a ruler and you are “king of the world”. You start to believe that you have supernatural predicting abilities and you start eyeing off ‘mentalist job applications’ in the local rag……..and then…..”Bang, Bang……. he shot me down….Bang, Bang, I hit the ground……Bang, Bang, this market shot me down….RIP !!!!!!”

It was the unpredictable move to a new market equilibrium that threw the spanner in the works.  That reliable historic average was no more. The stops of last resort started kicking in….not once….not twice but persistently until very quickly my hedonistic bliss was snuffed out and you started to use the local rag as shelter on a park bench as opposed to a source of career opportunity. You have just been measured by Mr Market and found wanting.

Be warned…if you decide to play with convergence, then ensure it represents a small allocation of your total divergent portfolio. You success in the market is typically defined by a very small fraction of trades. That is what defines your edge. Convergence takes no heed of this principle and assumes the opposite…..that your success is defined by the vast majority of your trades. If it smells to good to be true…..then you know the rest of that story.

If I said to you that the most successful traders on this planet assign their success to a very few outliers, then you will probably nod your head in agreement. The occasional windfall interspersed with a swag of bad outcomes where you simply haven’t lost much.   This just demonstrates that success and long term sustainability is derived from the unpredictable as opposed to the predictable.

If I said to you that the vast majority of traders that have account blowups can assign their fate to a single event or a very small sequence of very unfavourable events, then you will also probably nod your head in agreement. The large number of successful outcomes with the occasional unforeseen horrific event that’s sets you back to ground zero.

The central takeout from this is that these prior statements of success and failure are dominantly assigned to the unpredictable as opposed to the predictable.

So given that we may agree to this point about the issues surrounding convergence, why are most traders attracted to the principle of convergent predictability? There is a bit of incoherence here and stems from our desire to think we can crack these markets and that they are predictable in nature.

Here is a brief overview of some of the statistics, graphics, beliefs and behaviours that may help to identify if you have a bias towards convergence.

Convergence:

• High win rates eg. 60-70 plus % win rate. You are right more times than you are wrong. This leads to reinforcing behaviour and the perception that you can beat this market.
• Very popular technique in the trading game due to the deceptive lure of perceived predictability and consistent cashflow.
• Your success is defined by how many times you are right and the degree of ‘rightness’ provided that total score exceeds how many times you are wrong and their ‘wrongness’
• You spend great effort treating results to eliminate ‘curve fit’ random results through intensive statistical measures such as Walk Forward optimisation and Monte Carlo testing as you need to be right the majority of the time.
• Negatively skewed – Low Average Win$ and High average Loss $
• Linear equity curves over the short term representative of periods of market equilibrium. You see these curves a lot in the trading forums….and then they just ‘disappear’.
• Jagged departures from linearity over the longer term attributed to adapting market conditions and occasional market re-equilibration…..you just don’t see these curves on the forums or in the fine print of the black box ‘holy grails’ as they usually lie well below total risk of ruin.
• Nice tight and linear Monte Carlo signatures across strategy parameters – attributed to those robust strategies that successfully deploy convergence over a single market condition’
• Stable equity curves arising from Walk forward optimization where the predictability of the return stream leads to the impression that these strategies suit periodic optimisation with a fine print caveat…”provided that current market conditions persist”.
• An emphasis placed on getting the best return with little heed played towards matters of risk;
• The assumption that using more recent shorter term data sets to predict the short term future is preferable to using longer term data sets as you want to be trading the current market condition and assume that conditions persist;
• Strategy diversification is less important that strategy ‘concentration’. Given that a stable predictable market condition defines convergence, only a few static optimised solutions meet your required performance criteria and most are vastly different. You need to therefore ‘concentrate’ your portfolio selection towards these optimised outcomes that reflect the current market condition to prevent capital deterioration.
• The need to continuously strategy hop and pick new winners as old strategies no longer work with market conditions changing. This introduces the issues of market timing into your decision making and the nagging question of …..when should I turn this strategy off or on?
• Convergent players like to predict price and devise complex processes to attempt to crack open the secrets of the market.
• Style drift is far more prominent in this camp as system predictions can be very accurate or spectacularly wrong and when conditions change, you don’t/can’t immediately adapt and respond  accordingly.
• Efforts are progressively turned towards statistical techniques as opposed to logical design considerations to sift the signal from the noise. Data-mining enthusiasts who seek convergence are progressively drawn further away from the logic of simply mapping the system performance to the market condition in preference for finding that elusive data signal in the noise through statistical measures. This is the major cause of failure in a convergent mindset  where the emphasis is directed towards the cause of ‘predictability’ in an attempt to find the holy grail to predict the future as opposed to the mere act of making money by observing emergent complexity at work. The Statistical processes at work in data-mining make us detached from the reality of the market itself.

So let’s now take a step into the world of the less popular divergent trader and see what floats their boat.

Divergent Strategies – The benefits of assuming uncertainty

So from this point on the data-mining story gets better for those that simply accept that markets are complex, non-stationery and adaptive moving feasts. You can now get rid of that noise of overwhelming statistics associated with each generated strategy and the complex and intensive crunching of walk forward optimisation, and Monte Carlo testing. These are all techniques that are more suited to convergent styles of trading. They are all designed to derive an outcome that detects stability hence future predictability. They have little to say in the world of divergence which is characterised by a moving feast of uncertainty. In fact when describing convergence and divergence in descriptive terms they are polar opposites.

Now I have written at length on divergence in previous articles so there is no point in regurgitating this but I have provided a summary of the characteristics of a divergent trader below to demonstrate how poles apart these categorizations of trading style are.  So compare and contrast the following summary with the convergent trader above.

Divergence:

• Low win rates eg. 30-40% win rate or less. You are wrong more times than you are right. This leads to self-deflating egos where you accept that the market is always right while you may be frequently very wrong.
• An unpopular technique for the masses in the trading game due to the long periods of stagnation and slow building drawdowns while waiting patiently for the unpredictable. A wealth builder but certainly not a cashflow generator.
• Your success is defined by the very few outliers, often referred to as anomalies that simply mean that you happened to be in the right place at the right time as opposed to the rightness or wrongness of your decision making for each trade event.
• You spend most of your time invested under the hood in your system design ensuring it can capture divergence with most effort assigned to plugging the weaknesses of your design in navigating a variety of different market conditions.  You pay scant heed to Walk Forward optimisation and Monte Carlo testing as you don’t need to be right about the future as your emphasis on risk management ensures you will survive to fight another day. You just need to ensure you are not hopelessly wrong about it and out of the game so you can no longer participate in it.
• Positively skewed – High Average Win$ and Low average Loss $
• Volatile equity curves over the long term that reflect that the market decides your fate in this game as opposed to the predictive ability of your strategy. You emphasis is placed on eliminating adverse volatility. Forget the Sharpe ratio as some forms of volatility and correlation are favorable in this game. You just don’t want unfavourable volatility or unfavourable correlations. You are investing your efforts in ensuring that during adverse market conditions, you equity curve does not rapidly deteriorate. It is all about capital protection with the occasional non predictive bonanza to make the game worth it.
• Slowly building drawdown profile with rapid return to high water marks associated with ‘cutting losses short and letting profits run’. Many long term equity curves are available to demonstrate the sustainability of this approach over the long term under a broad range of different market conditions. It is not ‘risk of ruin’ that get’s you in the divergent game but the impact of long periods of torture by a thousand cuts and overall frustration. You may decide to move onto different pastures under frustration, but you are unlikely to be permanently incapacitated by the market.
• Splayed Monte Carlo signatures with the occasional large ‘step up’ leading to a stairway effect – attributed to the splay of nonperformance when conditions are not favorable with the strongly correlated step up across variables when conditions become favorable. This leads to a pattern of many landscapes with plateaus across a large data set and multiple market conditions. The steps are where the money is made….all else is just the noise of the game that contributes to the drawdown profile
• Erratic equity curves that defy optimisation making walk forward techniques a fools errand. The market decides when to depart from equilibrium not the desire of the system. When market divergence occurs….if it occurs….then the equity curve rises…..not before. During periods of market equilibrium divergent strategies spin their wheels or go slightly backwards.   Your prowess as a divergent trader is defined by how well you can preserve your hard earned capital. Profits are just a symptom of being a participant with a divergent system on hand.
• An emphasis placed on cutting losses short with little heed played towards matters of taking profit. You simply unbound the profit condition and let profits run with a trailing exit condition;
• The assumption that using longer term data sets ensures the strategy can navigate a broader range of market conditions and is the way to improve strategy robustness. The short term current market condition is simply used to detect the degree of drawdown incurred in this condition. There is no assumption that the current market condition will continue or not.
• Strategy ‘diversification’ is more important than strategy ‘concentration’. Given that market instability defines divergence, a vast collection of logically designed asymmetrical strategies with a relatively tight trailing stop can capture divergence. Furthermore given that divergence is more an anomaly than a predictable event, you need to diversify far and wide to capture these infrequent and unpredictable events. You need to design each divergent strategy to activate only during market extremes and avoid the noise of the everyday churn. The filters you use to capture divergence significantly reduce your trade frequency for each strategy so to bring home the bacon, you need lots of them across assets, timeframes and systems.
• The need to stay the course with your strategy when conditions are unfavourable and avoid strategy hopping at all costs. Risk of ruin is not your enemy here in ‘divergent land’ as your losses are never great, but drawdowns are a fact of life so get used to them. Simply have faith that if your strategy design captures divergent market conditions, then that is enough. Endure the pain to win this game. Avoid strategy hopping and associated symptoms of market timing. Your decision to turn a strategy off should be a design feature that is built into the system itself…not a psychological response associated with frustration. If you risk adjust your strategies to respond to adapting market conditions, as you should, then your under-performing strategies dilute rather than being simply turned off. Beware any forms of predictive decision such as ‘when to turn off your strategy?’.
• Divergent players like to follow price and devise very simple design solutions to restrict trading to more exotic market conditions and cut losses short and let profits run. They never predict, they just operate off the principle that trends tend to persist more frequently than a normal distribution would imply.
• Style drift is far more concentrated in this camp as simple systems are deployed to catch the divergent market condition. When markets diverge, nearly all trend followers and momentum chasers do well with large leaps in equity. When they don’t, style drift becomes more evident but the disparity in returns is less acute. It is only when we have long periods of stability that you start to see drift in this camp.
• Efforts are progressively turned towards design considerations that simply predate on a particular class of market condition as opposed to an attempt to crack the holy grail and seek predictive certainty in the future. Data-mining enthusiasts who seek divergence are progressively drawn further away from the statistics and associated  ‘curve fitting treatment’ of data streams towards design logic considerations where strategy performance is ‘mapped to the market condition’. It is the market that matters. The system is simply the method used to capture the alpha available when market conditions are favorable and at all other times operate to protect the capital base.

So the moral of this tale is this. If you want to make money in these markets, then you do not need to be a quant to do so. Only use intensive statistical tools if you understand what is trying to be achieved in their use. The maths is not wrong….but the application of that maths is for a different theoretical world that might depart from the reality.

All you need to do is to focus on what really matters in protecting and building your wealth. Use that brain of yours to map your system performance against market price.  A system will make money when market conditions are favorable for that system and lose money or simply not be active during unfavourable market conditions. You do not need intense statistical treatment to assess this. If markets are trending, then your system should be active and performing.

You do not need a Monte Carlo simulation or Walk Forward Optimisation to do this. Simply map the equity curve to the market condition and see if the logic of the relationship makes sense in accordance with your design rules. If it does, and your assessment has been made across a large data set comprising an array of varying market conditions, then the chances are that you have found a good divergent strategy that also is robust when market conditions are ‘not divergent’……..but just don’t settle on one of them…..find many different uncorrelated design solutions and then compile them together.

Capture as many different forms of market trend condition that you can with your suite of different design styles to increase your trade frequency and …..hey presto….alacazam….your equity curve will miraculously reach for the stars without you having to contrive it.

A divergent trader will make money when markets diverge either up or down…..but if they don’t, then you are not permanently crippled and can stay in the game until they do so. Do not add the baggage that predictability brings to your strategy development processes for in doing so, you will be confounded by complexity and false leads and forever lost in a statistical morass of confusion and intractability. Simply ensure that your system design bakes in the principle of ‘cutting losses short and letting profits run’. Survival is key in this game and the ability to participate in those rare events that make this game worth it.

Trade well and prosper

Rich B