The Physicist Who Beat the House Twice
From roulette wheels to Wall Street to simulating entire economies, and what one physicist's journey reveals about the limits of prediction and the power of responsive strategy
The idea of using physics to beat roulette did not begin with J. Doyne Farmer. It began with Edward O. Thorp.
In 1960, Thorp, a mathematics professor at MIT, partnered with Claude Shannon, the father of information theory, to build what is generally considered the first wearable computer. It was an electromechanical device, roughly the size of a cigarette pack, designed to predict the outcome of a roulette wheel by timing the ball and rotor. They tested it briefly in Las Vegas in the summer of 1961. It worked. Thorp estimated a 44% edge over the house. But the hardware was unreliable, and the pair decided their time was better spent elsewhere.
Thorp went elsewhere spectacularly. He wrote Beat the Dealer, proving blackjack could be beaten through card counting. He founded Princeton Newport Partners, one of the earliest quantitative hedge funds, and pioneered statistical arbitrage strategies that would define a generation of Wall Street quant trading. His trajectory ran from the casino to the trading floor: a straight line from finding exploitable structure in games of chance to finding it in financial markets.
But there was a second physicist who took the roulette project and followed it somewhere else entirely. Somewhere that leads, by a path that is neither straight nor obvious, to the deepest scientific foundations of what we do as trend followers.
In the late 1970s, nearly two decades after Thorp and Shannon’s experiment, a graduate physics student at the University of California, Santa Cruz, built a new roulette computer and hid it inside his shoe.
J. Doyne Farmer had grown up in Silver City, New Mexico, in a Boy Scout troop led by a young physicist named Tom Ingerson, whose adventures included searching for an abandoned Spanish gold mine to fund a mission to Mars. Farmer went to Stanford for his undergraduate degree, then moved down the California coast to Santa Cruz, where he was supposed to be studying physical cosmology. Instead, he and his childhood friend Norman Packard bought a roulette wheel and spent two years studying the physics of its motion.
Inspired by Thorp and Shannon’s earlier work, Farmer and Packard built what was arguably the first wearable digital computer. Data was entered by tapping a micro-switch with the big toe. Output was delivered by three solenoid actuators strapped to the stomach, vibrating to indicate which octant of the wheel to bet on. The entire system ran on a three-kilobyte programme that Farmer had hand-coded in machine language: a floating-point arithmetic package, a real-time sequencer, and a basic operating system, all designed for toe-based inputs and vibrotactile outputs.
They called themselves the Eudaemons, after the Aristotelian concept of enlightenment derived from a life lived in accordance with reason. Their goal was to beat the casino and use the winnings to fund a scientific commune. The name was not ironic. These were physicists who genuinely believed that understanding the dynamics of a system could give you an edge over the house.
The conventional wisdom about roulette was that it was random. The wheel spins. The ball bounces. The outcome is unpredictable. Thorp had already proved this wrong. Farmer and Packard took the proof further, but not in the direction you might expect.
Roulette is a deterministic physical system governed by Newton’s equations of motion. The ball follows a path shaped by its initial velocity, the angular speed of the wheel, friction, gravity, and the geometry of the pockets. In that narrow technical sense, the outcome is not random. It is determined by the starting conditions.
But here is the crucial distinction that Farmer, as a trained physicist, understood precisely: deterministic is not the same as predictable.
The motion of the roulette ball is chaotic in the mathematical sense. It is exquisitely sensitive to initial conditions. Tiny differences in the ball’s release, differences too small to measure, compound exponentially as the ball travels. The system is deterministic in principle and unpredictable in practice beyond a very short time horizon. This is not a statement about human measurement limits that will one day be overcome. It is a fundamental property of the mathematics governing the system.
Farmer and Packard did not attempt to predict the exact pocket. That was computationally and physically impossible. They tried to predict only the octant: which eighth of the wheel the ball would land in. By timing the ball’s passage relative to the wheel’s rotation, they could narrow the outcome from thirty-eight equally likely pockets to a region of roughly five. A coarse probabilistic edge, not a precise deterministic forecast.
This distinction matters enormously, because the roulette wheel is as close to a purely mechanical system as you will find outside a physics laboratory. It has a fixed number of states. Its dynamics are governed by classical mechanics, not by adaptive agents responding to each other. No one inside the system changes their behaviour based on where the ball landed last time. There are no feedback loops, no co-evolving strategies, no ecology of participants whose interaction generates the very dynamics being observed. It is, in the language of complexity science, a closed deterministic system, not a complex adaptive one.
The success of the shoe computer in a casino proves that deterministic chaotic systems can be nudged toward probabilistic advantage under very specific conditions. It does not prove anything of the kind about financial markets. The analogy between a roulette wheel and an equity market is seductive and wrong.
Here is where the two roulette stories diverge, and where it matters most for this series.
Thorp’s path ran from roulette to blackjack to Wall Street. He became one of the great quantitative investors, applying mathematical rigour to identifying mispricings in options and convertible bonds. His approach was fundamentally about locating specific predictive edges: situations where price differed from the mathematically correct value, and exploiting the difference.
Farmer’s path ran somewhere entirely different. After the roulette project, he switched his dissertation topic to chaotic dynamics. He joined Norman Packard, James Crutchfield, and Robert Shaw to found the Dynamical Systems Collective at Santa Cruz, a group later called the Chaos Cabal. Their most important contribution was a method for reconstructing the geometry of a chaotic system from a single time series: a way to visualise the hidden structure beneath apparently random data.
This technique, called state-space reconstruction, showed that behind apparently random fluctuations, there could be a low-dimensional attractor, a geometric shape in an abstract space describing the system’s underlying dynamics. The fluctuations were not noise. They were the shadow of structure.
Farmer completed his doctorate in 1981, took a postdoctoral appointment at Los Alamos National Laboratory, and received an Oppenheimer Fellowship. In 1988, he founded the Complex Systems Group in the Theoretical Division, recruiting a generation of researchers who would become leaders in the field.
In 1991, Farmer resigned from Los Alamos, reunited with Norman Packard and their graduate school classmate James McGill, and co-founded the Prediction Company in Santa Fe, New Mexico.
The founding thesis was audacious: the same techniques used to find structure in chaotic physical systems could be used to find structure in financial markets. If roulette was not random despite appearing random, then perhaps markets were not entirely random either.
They set up shop in an adobe bungalow furnished with plastic lawn chairs and high-end Sun workstations. They signed an exclusive contract with O’Connor and Associates, a Chicago derivatives firm, to provide investment advice and technology. When O’Connor merged with Swiss Bank Corporation, the relationship continued. The strategy they developed was an early form of statistical arbitrage, processing essentially every quantitative input related to the US stock market. From 1996 onward, all trading was fully automated.
Prediction Company was sold to UBS in 2006 and later resold to Millennium Management, where it became one of the firm’s largest funds. The strategy ran profitably for over two decades before being shut down in 2018.
But the name itself is instructive. The Prediction Company. And the eventual shuttering of the strategy is equally instructive. Farmer had found exploitable statistical structure in markets. But markets are not roulette wheels. They adapt. The agents inside them observe what is working and crowd toward it. The very act of deploying a strategy changes the environment that makes the strategy profitable. The prediction window closes. This is not a temporary problem to be solved by better computing power. It is a structural feature of any complex adaptive system populated by diverse, learning agents.
Farmer’s career after the Prediction Company follows the arc of a scientist who understood this distinction deeply, and spent the next two decades building the theory to explain it.
In 1999, Farmer left the Prediction Company for the Santa Fe Institute, and his work took a turn that connects directly to the framework of this entire series.
His experience with automated trading had led him to think carefully about the similarities between financial market strategies and biological species. At the Santa Fe Institute, he began developing what he called a theory of market ecology: a formal framework for understanding how trading strategies interact with each other the way species interact in an ecosystem.
The parallel is precise. In a biological ecosystem, the population of one species depends on the populations of all the others. Predators and prey co-evolve. A strategy that works when prey is abundant fails when the predator population grows too large. Niches open and close. No single species dominates permanently, because the very success of a species changes the environment in ways that undermine its advantage.
Financial markets work the same way. Farmer and his colleagues showed formally what Arthur’s El Farol problem had demonstrated intuitively: the profitability of any trading strategy depends on what other strategies are deployed simultaneously. A value strategy works until too many participants run it. A momentum strategy works until the capital it attracts overwhelms the signal. A volatility-selling strategy works until the premiums collected attract so much capital that the first shock produces catastrophe. Every strategy is a species. Every market is an ecology. And the ecology never reaches a permanent equilibrium.
In a 2021 paper in the Proceedings of the National Academy of Sciences, Farmer and co-authors demonstrated that market malfunctions, bubbles, crashes, liquidity crises, can be explained as ecological imbalances: moments when the population of one type of strategy grows too large relative to the others, destabilising the system exactly as an overpopulation of predators destabilises a food web.
This is the El Farol problem given mathematical teeth. And it confirms, from a completely independent line of research, the central distinction this series has drawn: predictive strategies are species that compete inside the ecology, subject to crowding and collapse. Responsive strategies are meta-strategies that observe the ecology itself and harvest the dynamics the competition produces.
But it is here that we need to be precise about what the word ‘responsive’ actually means, because this is not merely a practical trading observation. It is a logical consequence of the mathematics of complex adaptive systems.
A roulette wheel can be nudged toward probabilistic advantage because it contains no agents who respond to being observed, no learning, no adaptation, no co-evolution. The ball does not adjust its trajectory because too many physicists are timing it.
A financial market is the opposite of this. Every participant observes price. Every participant conditions their behaviour on what they observe. The output of the system, price, is a direct input to the decisions that produce the next price. This feedback loop is not incidental to how markets work. It is what markets are.
The structural consequence of this feedback is precisely what our earlier research series documented empirically. In the Fractals of Finance research programme, we built an agent-based model of a financial market to test whether feedback was merely correlated with the statistical signatures of real markets, or causally responsible for them. The experiment was decisive.
In the null world, a pure noise process with no agents and no feedback, every signature of real market behaviour vanished. Autocorrelation was zero. Hurst exponents sat at 0.53, indistinguishable from a random walk. Fat tails disappeared entirely. Five-sigma events: zero. The world that traditional finance assumed was the correct description of markets produced a statistical flatline.
When we introduced a realistic ecology of agents, divergent participants whose behaviour amplifies price moves alongside convergent participants who fade them, everything transformed. The autocorrelation of absolute returns leapt from zero to 0.287. The Hurst exponent jumped to 0.609. Excess kurtosis surged from zero to 27. Five-sigma events appeared at thirteen per ten thousand days, where the null world produced none. Every signature returned simultaneously, driven by a single variable: the presence of feedback between participants and price.
Furthermore, in Episode 6 of that series, we swept the dial from zero to maximum divergent participation and discovered that the transition was not gradual. It was a phase transition: a critical threshold at approximately twenty-five to thirty percent divergent agents, below which markets behave essentially like random walks, and above which the full fingerprint of volatility clustering, fat tails, and persistent memory erupts simultaneously. The random walk does not erode as feedback increases. It shatters.
This is the causal mechanism. Feedback between adaptive agents is not one explanation among several for why markets exhibit the statistical signatures they do. It is the only mechanism required. Our simulations contained no external shocks, no macroeconomic news, no earnings announcements, no central bank interventions. Just participants observing price and reacting, convergent and divergent forces in permanent tension. That alone was sufficient to reproduce the statistical DNA of sixty-eight real futures markets across eight asset classes and four decades.
This is the context in which Farmer’s later work at the Santa Fe Institute takes on its full significance, and why it matters so directly for how we think about systematic trend following.
The ecology he described is not static. It is permanently out of equilibrium. The population of strategies shifts continuously. Niches open and close as participants enter, grow, crowd, and collapse. The feedback that produces the statistical signatures we documented is not a temporary feature of immature markets that will fade as information technology improves and participants become more sophisticated. It is the inevitable output of any market populated by diverse, adaptive, boundedly rational agents. The more sophisticated the participants, the more elaborate the co-evolutionary dynamics, but the feedback loop does not disappear. It deepens.
In 2024, Farmer published Making Sense of Chaos: A Better Economics for a Better World. The argument is sweeping. Standard economics is built on assumptions that bear no resemblance to how economies actually work. Rational agents, perfect information, and equilibrium are mathematical conveniences, not descriptions of reality. What is needed is a computational approach: agent-based models that simulate millions of interacting adaptive agents and observe what patterns emerge.
These models do not assume equilibrium. They do not require the future to resemble the past in any specific way. They place adaptive agents in an environment, give them rules for learning, and let them interact. What emerges, spontaneously and without being programmed in, is a market that produces fat tails, clustered volatility, speculative bubbles, and sudden crashes. Every pattern that a trend follower knows by heart appears naturally from the interaction of adaptive agents in a non-equilibrium system. Farmer’s models are now used by central banks and financial regulators around the world.
For anyone who trades systematically, the implications are profound. If the dynamics that trend followers exploit, momentum, volatility clustering, fat tails, emerge naturally and necessarily from the interaction of adaptive agents, then these dynamics are not anomalies waiting to be arbitraged away. They are structural features of any market populated by diverse, adaptive, boundedly rational agents. They are as permanent as the agents themselves.
Let me draw the line that runs through Farmer’s entire career, because it points somewhere precise.
At the roulette wheel, Farmer showed that a deterministic chaotic system contains enough short-term structure to support a coarse probabilistic edge. Not prediction of outcomes. A reduction in uncertainty across a narrow horizon, in a closed mechanical system with no adaptive agents.
At the Prediction Company, he found that financial markets contain exploitable statistical structure. But the structure was not stable in the way that the roulette wheel’s physics was stable. Markets adapt. Strategies crowd. Edges close. The prediction window, always short, narrows further as capital flows toward whatever is working.
At the Santa Fe Institute and Oxford, he built the theory that explains both observations. Markets are ecologies of competing adaptive strategies. The co-evolution of these strategies produces characteristic dynamics, trends, reversions, fat tails, clustered volatility, that are not noise but signal. They are permanent features of the system. But they cannot be predicted in advance. They can only be detected and responded to.
This is the through-line of Farmer’s career, and it is not the one that superficially reads as a story about prediction. It is a story about the fundamental limits of prediction in complex adaptive systems, and about the rational response to those limits.
Thorp proved you could beat the house with mathematics, in a closed deterministic system. Farmer proved you could find statistical structure in a financial market. Then he went to Oxford and proved that the structure itself is generated by a mechanism, feedback between adaptive agents, that makes the kind of precise prediction Thorp exercised in blackjack permanently unavailable in markets.
The Prediction Company was not the endpoint of Farmer’s thinking. It was a chapter in a longer argument that concluded somewhere different: that markets are living systems, not mechanical ones, and that the only rational posture toward a living system is not prediction but alignment with its structural dynamics.
Farmer’s journey is the bridge between the science and the practice.
Brian Arthur gave us the theoretical framework in earlier articles in this series: non-equilibrium dynamics, increasing returns, ecologies of strategies that never settle. Farmer took those ideas and built systems that traded real money in real markets, found their limits, and returned to academia to prove mathematically why those limits exist. He showed that the patterns trend followers have known empirically for decades are not quirks of historical data. They are the inevitable output of any complex adaptive system populated by diverse, adaptive agents.
The work documented in the Fractals of Finance research series, our empirical investigation across sixty-eight futures markets and forty-one years of data, arrives at the same destination from the other direction. We measured the fingerprint in real market data. We proved causation by building the null world and watching every signature disappear. We demonstrated that feedback between divergent and convergent agents is sufficient, and necessary, to produce every statistical signature simultaneously. Farmer’s ecology framework is the theory. The Fractals of Finance research is the empirical confirmation.
The structural implication of both is identical: you cannot predict your way through a complex adaptive market. The feedback loops that produce the dynamics you seek to capture are the same feedback loops that close the prediction window. The only rational response is to build systems that detect and align with structural dynamics as they emerge, not systems that forecast where those dynamics will appear next.
Observe the structure. Follow the signal when it appears. Respond to the system rather than predict it.
Farmer’s career already proved why that is the only viable strategy.
Next in the series: “The Artificial Stock Market.” When physicists built a computer model of a stock exchange from scratch, it spontaneously produced every pattern that trend followers know by heart. What that tells us about where returns come from.
Want the theoretical foundation for why trend following works?
The Fractals of Finance: Determinism, Adaptation and the Geometry of Markets
The book explores the full architecture of feedback, fat tails, and fractal structure in financial markets, and what it means for how we trade, invest, and understand risk.
Available now on Amazon in paperback, hardcover, and Kindle.
Want a practical field manual for trading trends and capturing outliers?
The Aussie Turtles Trend Following Guide: A Field Manual for Hunting Outliers adapts the timeless principles of the original Turtle traders into a systematic, rules-based approach for modern markets. Co-authored with Adam Havryliv.
Available now on Amazon in paperback, hardcover, and Kindle.