Regime Discovery
When considering market regime, many traders think first of macro principles: business cycle, risk appetite, interest rates, liquidity, market volatility, etc. This perspective is motivated by applying herd behavior to market dynamics: regimes arise when many people either trade in harmony (trends) or disharmony (range). In this sense, herd behavior is an unobservable for regimes.
Yet, this explanation is not terribly insightful for trading: either one must predict the future or settle for exploiting established trends (i.e. macroeconomic forecasting or trend following). Although both strategies are actively traded, profits tend to be unpredictable.
An alternative approach is trying to build regime discovery models which quantitatively identify regimes early in their formation, and probabilistically inform how to trade ahead of the herd (whether by seconds or days). For example, how price or volatility of HP, VGT, and DJIA change in relation to a price change in IBM. Alternatively, how USD crosses change in response to an unobserved Fed intervention.
How to formulate this type of discovery model is not obvious. What follows is a sketch, which readers are encouraged to comment upon.
Begin by positing:
Regime discovery drives prices and volatility via second-order causal information diffusion.
In other words: a price change in one security will cause one or more other securities to correspondingly change price. Alternatively, by analogy; think of a pebble thrown in a lake:
- Lake: the universe of observable variables, consisting of all time-series observable from worldwide exchanges and OTC (although not accessible to retail traders, this universe is available to quant HFs with good connectivity)
- Pebble: event which causes a price change for a single instrument at a single point in time
Framed this way, regime discovery requires a model for how prices across multiple instruments sequentially change over time in response to a price change of a single instrument (i.e. an inter-instrument sequential price model). Thus, a corresponding model should be potentially applicability to any market which trades multiple securities whose prices are empirically believed to be driven by information diffusion:
- Equities: price change in one instrument resulting in price/vol changes in others due to ETF composition, pairs trading, hedging, etc.
- FX: central bank intervention resulting in updated quotes/trades being propagated across dealers
- IR: changes in central bank rates resulting in price/vol changes across credit and equity instruments
Models for expressing inter-instrument sequential price/vol changes warrant research. Tr8dr recently authored several interesting posts applying clustering and correlation, including minimum spanning trees for inter-asset correlations and equity clusters. Older literature includes interesting articles by Zoran Obradovic (mid-1990s), Takayuki Mizuno, and Laszlo Gillemot.
Quantivity 
Disclaimer: I’m an algo-trading newbie, just beginning to learn the ropes so I hope you can answer this question:
I can see how USD crosses change in relation to Fed interventions (i.e. unusual volatility in crosses when the rates aren’t as expected), but what do you mean by “unobserved” Fed intervention? Do you mean, the retail algo traders won’t have direct (i.e. computational) access to this information?
Thanks for writing the blog – it’s an interesting read so far!
@Michael: thanks for your question, and complement. Although Quantivity is the first to admit having never worked on a fx desk, the mechanics are pretty well documented. Primary dealers intervene in the market on behalf of sovereigns, in ways which are unobserved (e.g. Lyons [2001]); interdealer trades or bid/ask spreads may partially leak this behavior, although they too remain unobservable by average retail traders (e.g. p.45 of Lyons [2001]).
Is this because the average retail investor does not have access to “Level 2” market depth information, i.e. the list of all buy/sell orders? Surely there are services which provide this data for a reasonable price (especially for equities)?
@Michael: if you are referring to fx, then no: institutional and sovereign fx order flow is never made public (except a few survey academic studies), as it’s OTC; and, thus, there is no “level 2” equivalent.
Having thought about this post in more depth, have you considered an algorithm known as Non-Negative Matrix Factorisation? There’s a good write-up in Toby Segaran’s book “Programming Collective Intelligence” on how he applies it to correlations in Yahoo and Google’s share price.
You could use this method to see how multiple securities change in response to one security change, I would imagine.
Click to access TanakaYamawakiPaper.pdf
@at: very interesting paper (see HF Basket Prediction via RMT / PCA for an initial comment on this paper when its abstract was published, prior to the conference). Coincidentally, I was just reading Plerou [2002] last week.