This topic is known as direction-of-change forecasting in the literature. Needless to say, successful prediction of the sign for future returns is quite interesting from a trading perspective. Traditionally, only univariate return series were considered; Anatolyev (2008) is an exception, modeling two or more interrelated markets via dependence ratios. This literature tends to be a bit obtuse, due to commonly unstated stylistic assumptions regarding conditional return dynamics.

The traditional formulation for this topic considers the estimation of the probabilities of returns exceeding an upper or lower threshold , optionally conditioned on an information set from the previous time step:

If , the probabilities reduce to forecasting positive or negative returns; for trading, a natural choice for is roundtrip transaction costs:

Estimating these probabilities can be undertaken via several techniques. One approach is to use a logit model, based upon the logistic function:

Where are explanatory variables from the previous time step. Challenge of this model is proper selection of explanatory variables.

An alternative approach is to consider the following functional decomposition for univariate return series:

Where is the conditional expected value, is the conditional variance, and is a martingale with zero mean, unity variance, and conditional distribution function . From which the direction of change probabilities can be expressed:

With corresponding conditional expectations:

These expectations simplify to the following when , assuming (otherwise, expectation is constant and thus uninteresting):

These expectations can be evaluated explicitly via calculating the empirical distribution function (requiring assumption of a parametric distribution), where is the indicator function:

Alternatively, this decomposition suggests one potential formulation for the logit parameters from the above model, where is estimated by the logit and are historical observations:

Of course, the non-trivial work is generating forecast estimates for next-step average conditional return and conditional variance .

An alternative way to model the logit parameters is to apply ARMA intuition with a binary autoregression (BARMA) due to Startz (2006), including lags for both autoregressive parameters and past indicator values, due to Anatolyev (2008):

A survey of estimation techniques may be considered in a subsequent post, depending on reader interest.

One exploratory analysis technique relevant to sign forecasting is visualizing up/down runs, signed difference (i.e. up-down), and corresponding averages for a return series.

returnRuns <- function(r, bound=0, doPlot=TRUE, startAvg=5, avgLen=-1)
{
  # Generate up/down runs and average runs for a return series, optionally
  # plotting them.
  #
  # Args:
  #   r: return series
  #   bound: symmetric upper and lower bound, aka c
  #   doPlot: flag indicating whether plots should be generated for runs
  #   startAvg: Number of average runs which should be excluded for
  #             eliminating unstable average with few leading observations
  #   avgLen: number of periods over which to generate average; of -1 for
  #           entire period
  #
  # Returns: none
  
  up <- cumsum(ifelse(r > bound, 1, 0))
  down <- cumsum(ifelse(r < -bound, 1, 0))
  
  if (doPlot)
  {
    plot(up, main='Signed Runs: Up & Down', ylim=range(up,down))
    lines(down, col='red')
    legend("topleft",legend=c("Up","Down"), fill=colors, cex=0.5)
    
    plot(up-down, main="Signed Run Difference (up-down)")
  }
  
  if (avgLen == -1)
  {
    avgUp <- xts(sapply(c(1:length(up)), function(i) {
      up[i]/i
    }), order.by=index(up))
    avgDown <- xts(sapply(c(1:length(down)), function(i) {
      down[i]/i
    }), order.by=index(up))
  } else
  {
    avgUp <- xts(sapply(c(avgLen:length(up)), function(i) {
      start <- i - avgLen + 1
      last(cumsum(ifelse(r[start:i] > bound, 1, 0))) / avgLen
    }), order.by=index(up[avgLen:length(up)]))
    avgDown <- xts(sapply(c(avgLen:length(down)), function(i) {
      start <- i - avgLen + 1
      last(cumsum(ifelse(r[start:i] < bound, 1, 0))) / avgLen
    }), order.by=index(up[avgLen:length(up)]))
  }
  
  if (doPlot)
  {
    n <- length(avgUp)
    plot(avgUp[startAvg:n], main=paste("Average Runs: Up & Down (",avgLen," periods)",sep=""), type='l', ylim=range(avgUp,avgDown))
    lines(avgDown[startAvg:n], col='red')
    legend("topleft",legend=c("Up","Down"), fill=colors, cex=0.5)
  }
      
  return (list(up=up, down=down, avgUp=avgUp, avgDown=avgDown))
}

For example, the following plots illustrate CRM run dynamics from 2005 to present. First plot illustrates the running sums for both up and down returns, indicating negative returns are more prevalent:

Second plot illustrates the difference in signed sums, showing time-dynamics for the difference in up and down returns. Not surprising, this difference mirrors the CRM price curve closely:

Third plot illustrates the average probabilities for both up and down, running incrementally over the entire timeframe:

The following are representative papers from the direction-of-change literature, ignoring the early papers focused on evaluating market efficiency (e.g. run tests):

• Forecasting Stock Indices: A Comparison of Classification and Level Estimation Models, by Leunga, Daoukb, and Chenc (2000)
• Financial, Asset Returns, Market Timing, and Volatility Dynamics, by Christoffersen and Diebold (2002)
• Direction-of-Change Forecasts Based on Conditional Variance, Skewness and Kurtosis Dynamics: International Evidence, by Christoffersen et al (2004)
• Are the Directions of Stock Price Changes Predictable? Statistical Theory and Evidence, by Hong and Chung (2003)
• Evaluating Direction-of-change Forecasting: Neurofuzzy Models vs. Neural Networks, by Bekiros and Georgoutsos (2005)
• Modeling Financial Return Dynamics via Decomposition, by Anatolyev and Gospodinov (2007)
• Direction-of-change Forecasts and Trading Strategy Profitability at Intra-Day Horizons, by Deliya (2007)
• Multi-Market Direction-of-Change Modeling Using Dependence Ratios, by Anatolyev (2008)
• Forecasting the Direction of the U.S. Stock Market with Dynamic Binary Probit Models, by Nyberg (2008)
• Direction-of-Change Financial Time Series Forecasting using Bayesian Learning for MLPs, by Skabar (2008)
• A Kernel-Based Technique for Direction-of-Change Financial Time Series Forecasting, by Skabar (2008)
• Optimal Probabilistic and Directional Predictions of Financial Returns, by Thomakos and Wang (2009)
• Directional Prediction of Returns under Asymmetric Loss: Direct and Indirect Approaches, by Anatolyev and Kryzhanovskaya (2009)

Finally, Kinlay briefly surveyed this topic in two posts on volatility sign prediction.

This focus is not behavioral finance, in search of anomalies driven by cognitive biases divergent from equilibrium (although majority do that too). Rather asking inferential sociological questions, such as: was the market “efficient”, in the Fama sense, during the post-war decades prior to 2000 because people expected it to be (blissfully ignoring a few hiccups); in contrast to how it is commonly understood and formalized, with reverse causality: market is assumed to be efficient, thus people understand it as such.

Similarly, have the past 15 years been “inefficient”, in the bubble and anomaly sense, because cultural faith among investors in such “efficiency” was lost; or, did they lose faith because the market became inefficient? Big difference.

In other words: is finance governed by physics, biology, or Peltzman?

The traditional answer of market hypothesis, provided by financial economics via microeconomic principles of equilibrium and efficiency: causality flows from market to investor. This explanation comes in two variants, known by their colloquial analogical fields:

• Physics: market is governed by immutable mathematical principles and can be formalized into coherent predictive models, either in favor or contradiction of excess returns; exemplified by classic weak/strong EMH theory
• Biology: market is governed by evolutionary principles ala Darwin, as exemplified by Lo’s 2004 AMH article: “Very existence of active liquid financial markets implies that profit opportunities must be present. As they are exploited, they disappear. But new opportunities are also constantly being created as certain species die out, as others are born, and as institutions and business conditions change.” (p. 24)

Yet, both these explanations suffer from implicitly begging the question: conjure “a market” with desired attributes and then derive conclusions. The physics perspective assumes immutability, conceivability, and mathematical expressiveness for its hypothesized market. While the biology perspective endows the hypothesized market with even more sophisticated Darwinian traits, presumably driven by underlying physical principles so inscrutable as to defy mathematical formalization.

An alternative explanation is to apply the self-fulfilling Peltzman effect to financial markets, and reverse causality: markets behave as they do because of investor sociology, rather than arising emergent from implicit cooperation of equilibrium-seeking rational microeconomic agents.

In other words: when investors believe the market is rational (irrespective of whether that belief is well-founded), then they embody Dunning-Kruger by ex ante faithfully dumping money into their 401K each month; in doing so collectively, the investment management industry undertakes its rent seeking activity resulting in the market possessing ex post “efficient” characteristics. Conversely, when investors believe the market is irrational, they either: go to cash, pursue uninformed non-collective trading, or both. Both of which result in anomalous market behavior, uncontrollable by the industry, either due to decreased liquidity or absence of predictable momentum.

If the market is indeed Peltzmanian, then the real question is how to best quantify and model primary and spillover effects resulting from investor sociology as they unfold ephemerally.

Recall is unobserved, and thus the model cannot be directly estimated via MLE. Thus, need to decide how to approach estimation for this latent variable. One way is to be naive, and simply assume is the deterministic difference in return between stock and index (technically, this generates a profile likelihood as formalized by Severini and Wong [1992], which Murphy and Van Der Vaart [2000] verify is well-behaved consistent with exact likelihood):

This assumption permits focus on estimating , providing insight into the mixing behavior of the return being decomposed: if a stock return behaves like its index, then mixing is low with small (in the limit, when a stock behaves identical to its index, as no mixing is required); in contrast, the stock return behaves independent from its index on a regular basis, then mixing is high with a large .

Autocorrelation of and is worth consideration, as that helps determine whether time indexing is required for . For returns with insignificant autocorrelation (common for signed equity returns), the time index is dropped and a single is estimated. Yet, conditional dependence often exists between and , consistent with previous posts in the Proxy / Cross Hedging series (illustrating r-z copula for CRM / QQQ example below):

Use of this identity for transforms the decomposition model into:

The model is further simplified into a familiar independent mixture model by dropping sign and , and estimating via MLE using density and return distribution functions :

MLE estimation requires assumption of parametric distributions for , of which common choices from the literature are normal, student-t, skew-t, or skew hyperbolic student-t (Aas and Haff [2006]). Next question is how to estimate the parameters: and family of parameters (e.g. and if is assumed to be normal). As is observed, one way to proceed is via two-step estimation:

1. Estimate parameters via MLE from
2. Jointly estimate and parameters via MLE on the mixture, holding parameters constant

For both, recall the likelihood , and log likelihood , are defined as:

From which MLE of the mixture is maximization of the likelihood over , where log is chosen for numeric stability:

This optimization can be performed numerically in R via minimization using DEoptim of the negative log likelihood negLogLikeFun (negative is due to minimization in DEoptim versus maximization of ). DEoptim is chosen due to rapid convergence on non-smooth global optimizations.

For example, continuing the example of CRM / QQQ introduced in the previous posts on Proxy / Cross Hedging generates the results:

> symbols <- c("CRM", "QQQ")
> endDate <- Sys.Date()
> startDate <- endDate - as.difftime(52*5, unit="weeks")
> quoteType <- "Close"
> p <- do.call(cbind, lapply(symbols, get.hist.quote, start=startDate, end=endDate,  quote=quoteType))
> colnames(p) <- symbols
> doReturnDecomp(p)
normal mix likelihood: -3485.55 phi1 params: 0.0003471366 0.01673634 params 0.2546208 -0.001208877 0.0113988
skew-t mix likelihood: -3566.512 phi1 params: 0.003844969 0.01079941 -0.3252923 2.893228 params 0.2357737 -0.004188977 0.01099266 0.4157174 26.5643
skew-hyp-t likelihood: -3083.700 phi1 params: 0.01051675 0.1485529 -3.945452 10.10836 params 0.8295289 -0.0003636940 0.03071332 -0.5 5

These results correspond to the following density functions for the skew-t mixture:

One interesting observation of these densities is their location parameters are on opposing side of zero: has positive location, while has negative location. One interpretation of this is positive returns from CRM disproportionately originate from the idiosyncratic , while negative returns originate from the index . Economically, this is plausible: positive news is often idiosyncratic, while negative news is often market-wide.

Several additional inferences can be drawn from these results:

• Model selection: likelihood suggests skew-t is the preferred model, indicating long tails and skewness (matching stylized facts)
• Mixing: indicating that over 75% of CRM returns are determined by the corresponding QQQ index; remaining 25% are determined by the unobserved return series
• Tails: CRM df = 2.89 which indicates significantly thicker tails than QQQ df = 26.56 (matching stylized facts for individual stocks versus indices)

Subsequent posts may consider alternative estimation techniques for this model.

library("MASS")
library("stats")
library("DEoptim")
library("sn")
library("SkewHyperbolic")

normalMixtureIndexDecomp <- function(r, i)
{
  # Two-step MLE estimation of return decomposition model, assuming both
  # return distributions are normal.
  #
  # Args:
  #   r: return series being decomposed
  #   i: index series used for decomposition
  #
  # Return value: MLE parameter estimates
  
  z <- r - i
  id <- fitdistr(i, "normal")$estimate
  negLogLikeFun <- function(p) {
    a <- p[1]; mu1 <- p[2]; s1 <- p[3];
    ll <- (-sum(log(a * dnorm(z,mu1,s1) + (1 - a) * dnorm(i, id[1], id[2]))));
    return (ll); 
  }
  mle <- DEoptim(negLogLikeFun, c(0, -0.5, 0), c(1, .5, .5), control=list(trace=FALSE))
  
  cat("normal mix likelihood:", last(mle$member$bestvalit), "phi1 params:",id, "params", last(mle$member$bestmemit),"\n")
  mle <- last(mle$member$bestmemit)
  
  x <- seq(-.25,.25,length.out=500)
  dnorm1 <- dnorm(x, id[1], id[2])
  dnorm2 <- dst(x, mle[2], mle[3])
  plot(x, dnorm1, type='l', ylim=c(0, max(dnorm1,dnorm2)), ylab="Density", main="Normal Mixture")
  lines(x, dnorm2, col='red')
  abline(v=id[1], lty=2)
  abline(v=mle[2], col='red', lty=2)
  legend("topleft",legend=c("phi1", "phi2"), fill=colors, cex=0.5)
    
  return (mle)
}

mixtureSkewTIndexDecomp <- function(r, i)
{
  # Two-step MLE estimation of return decomposition model, assuming both
  # return distributions are skew-t.
  #
  # Args:
  #   r: return series being decomposed
  #   i: index series used for decomposition
  #
  # Return value: MLE parameter estimates

  z <- r - i
  idp <- st.mle(y=i)$dp
  negLogLikeFun <- function(p) {
    a <- p[1]; mu1 <- p[2]; s1 <- p[3]; s2 <- p[4]; df1 <- p[5]
    ll <- (-sum(log(a * dst(z,location=mu1,scale=s1,shape=s2,df=df1) + (1 - a) * dst(i, dp=idp))));
    return (ll); 
  }
  mle <- DEoptim(negLogLikeFun, c(0, -0.5, 0, 0, 2), c(1, .5, .5, 5, 50), control=list(trace=FALSE))
  
  cat("skew-t mix likelihood:", last(mle$member$bestvalit), "phi1 params:", idp, "params", last(mle$member$bestmemit),"\n")
  mle <- last(mle$member$bestmemit)

  
  x <- seq(-.25,.25,length.out=500)
  dst1 <- dst(x, dp=idp)
  dst2 <- dst(x, dp=mle[2:5])
  plot(x, dst1, type='l', ylim=c(0, max(dst1,dst2)), ylab="Density", main="Skew T Mixture")
  lines(x, dst2, col='red')
  abline(v=idp[1], lty=2)
  abline(v=mle[2], col='red', lty=2)
  legend("topleft",legend=c("phi1", "phi2"), fill=colors, cex=0.5)
  
  return (mle)
}

mixtureSkewHypTIndexDecomp <- function(r, i)
{
  # Two-step MLE estimation of return decomposition model, assuming both
  # return distributions are skew hyperbolic student-t.
  #
  # Args:
  #   r: return series being decomposed
  #   i: index series used for decomposition
  #
  # Return value: MLE parameter estimates

  z <- r - i
  iparam <- skewhypFit(i,plots=FALSE,printOut=FALSE)$param
  negLogLikeFun <- function(p) {
    a <- p[1];
    ll <- (-sum(log(a * dskewhyp(z,param=p[2:5]) + (1 - a) * dskewhyp(i, param=iparam))));
    return (ll); 
  }
  mle <- DEoptim(negLogLikeFun, c(0, -5, 0, -0.5, 0), c(1, 5, .5, -0.5, 5), control=list(trace=FALSE))
  
  cat("skew-hyp-t likelihood:", last(mle$member$bestvalit), "phi1 params:",iparam,"params", last(mle$member$bestmemit),"\n")
  mle <- last(mle$member$bestmemit)

  
  x <- seq(-.25,.25,length.out=500)
  dskewhyp1 <- dskewhyp(x, param=iparam)
  dskewhyp2 <- dskewhyp(x, param=mle[2:5])
  plot(x, dskewhyp1, type='l', ylim=c(0, max(dskewhyp1,dskewhyp2)), ylab="Density", main="Skew Hyperbolic Student-T")
  lines(x, dskewhyp2, col='red')
  abline(v=iparam[1], lty=2)
  abline(v=mle[2], col='red', lty=2)
  legend("topleft",legend=c("phi1", "phi2"), fill=colors, cex=0.5)

  return (mle)
}

doReturnDecomp <- function(p)
{
  # Decompose return of two series, using several parametric distributions.
  #
  # Args:
  #   p: p[,1] is return being decomposed; p[,2] is index returns
  #
  # Return value: none

  r <- ROC(p[,1], type="discrete", na.pad=FALSE)
  i <- ROC(p[,2], type="discrete", na.pad=FALSE)
  
  normalMixtureIndexDecomp(r, i)
  mixtureSkewTIndexDecomp(r,i)
  mixtureSkewHypTIndexDecomp(r,i)
}

Moreover, it is misunderstood—even by many who have smelled it up close personally via big trading loses on hedged positions. Aaron Brown’s most recent text, Red-Blooded Risk, explains why.

In doing so, it is simultaneously brilliant and flawed. For the former, Brown deserves credit; for the latter, the publisher presumably deserves most of the blame.

First, the brilliance; summarized in one word, with two intended meanings: pragmatics. Oh yeah, the book includes 漫画-style comic strips, helpfully provided as idiot self-detectors.

First, well-known meaning is the philosophical tradition linking practice and theory. Arguably unique for risk books, Brown builds deep intuition around the concept of risk and its manifestation from structural to marked positions to historical roots in tulips. Brown has clearly lived and breathed risk management for many years (self-proclaimed from its modern origin in the 1980s), and that wisdom shines through via first-person prose combining insight, intuition, arrogance, pride, greed, humility, regret, condescension, and insecurity. Perhaps the first book ever to make VaR sound geek-sexy—with nary a technical definition.

Second, lesser-known meaning is the subfield of linguistics which investigates ways in which context contributes to meaning. Unlike technical risk texts, Brown spends much of the book deep diving into context and letting meaning emanate from therein. As he states, “different aspects are easier to understand from different vantages” (p. 57). As a reader who never personally worked at a big bank, this is deeply informative for attuning mental models (akin to Harris for trading mechanics, Rebonato for derivatives, and Taleb for hedging). Explaining the lineage of quant hedge funds through the corresponding frequentist versus Bayesian disposition of their founders is fascinating, and indeed makes sense in retrospect. Slicing away the credibility mystique and exposing the raw underbelly of banks, down to explaining front, middle, and back office in depth. For readers familiar with disciplined metrics-driven tech companies, the apparent intellectual and technical sloppiness of big banks is simply jaw dropping.

One excerpt simply must be quoted, beginning of Chapter 4, as it is perhaps the most accurate and beautiful summary of self-entitlement believed by geeks with -IQ time immemorial:

The rocket scientists came together on Wall Street in the 1980s and began the process that eventually explained the modern concept of probability and reconstructed the global financial system. We were not individually ambitious. All we wanted was to make more money than any rational person could spend, without ever putting on a tie or being polite to anyone we didn’t like. We didn’t have any use for the money, except for maybe some books and cool computer equipment. We didn’t want to throw (or go to) fancy parties or buy political power—and we didn’t spend it on cars, jewelry, or places to live, and least of all on clothes. We’d probably give the money away, but until then, it would give us the power to say “f- you” to anyone, except that we were mostly pretty soft-spoken and civil in our expressions

Now, the flawed parts.

One reviewer caveat worth advance mention is 9 books out of 10 read by Quantivity have content dense with math, code, or both. On the positive side, reading of Brown’s book indicates positive noteworthiness due to its statistical abnormality (given it has neither); on the negative side, any review is biased through such lens.

First, lots of effort was expended by the publisher making this text appeal to a mass audience, clearly rushing to fill the void of perceived post-financial crisis publishing opportunity. From the ridiculous title (and cover) to the silly use of “secret history” meme to hilarious tongue-in-cheek back cover reviewer comments by Gatheral, Taleb, Wilmott, and Thorp. Parts of the book are prone to hyperbole, which read like they were edited in for sales effect. While these nuisances detract credibility, such can be ignored and arguably contributes the positive benefit of reducing its purchase price to mass market (i.e. under $25).

Second, the book lacks unifying organization. While Brown provides the following disclaimer for such, the editor equally deserves some blame (p. 57):

If I had all the theory worked out, I could write a textbook organized in logical sequence. Instead, I’m going to intersperse theoretical discussions with accounts of the development of the ideas.

While this makes sense, it is admittedly a bit jarring to see that disclaimer juxtaposed alongside supposition of having “explained the modern concept of probability”. Thus, the reader is left wondering whether perhaps either of the following may be true:

• Brown has a theory of risk, but was refused in editing due to overabundance of equations
• Brown has a new theory of probability, but could not muster a theory of risk

Either are intriguing, although perhaps the former seems more likely as Brown includes the following tongue-in-cheek disclaimer regarding use of a tiny bit of high school-level math included in Chapter 5 (p. 73):

Warning, this chapter contains a little math. It’s nothing intimidating, mostly multiplication and some simple algebra, but I know a lot of people don’t like it. If that describes you, I urge you to read the chapter anyway. It’s one of the most important in the book. You can skip the math and get the ideas anyway.

Having never met Brown and thus unfamiliar with his personality, cannot escape the sense that he is making gentle fun of readers which possess such bias. Either way, it’s amusing.

While modest disorganization is a textual flaw, astute readers may perhaps perceive it as subtle financial opportunity: if such theory was sufficiently well-defined to warrant standard textbook treatment, then there would undoubtedly be much less juice possible from doing it really well.

In either case, remedy for this shortcoming is to read the book in as few distinct sittings as possible. Having read it in two sittings, the wisdom was able to percolate together and expand personal mental models nicely. Well worth the read.

This technique finds surprisingly often use in quant models.

Ongoing analysis and trading based on proxy hedging, exemplified by series beginning with Proxy / Cross Hedging, suggests potential for an equity decomposition model based on the relationship between returns of a stock and its corresponding index :

To explain this model, let’s build it up from intuition.

To begin, consider a trading observation: interday returns of individual stocks have a subtle relationship with their corresponding index. On some days, return for a given stock follows its index; other days, returns of stock and index diverge strongly. This distinction in behavior is commonly attributed to stock-specific “news”, interpreted broadly—whether known publicly or only privately.

This intuition can be formalized into two-state regime:

• Uninformed regime: stock return follows an index , scaled by a proportional factor
• Informed regime: stock return follows an idiosyncratic path , conditionally independent of its index

Relationship between regimes can be modeled in two ways via . A switching model arises when regimes are binary: . An ensemble model arises when regimes are smooth: . For the latter, can be understood as proportional decomposition weighting of the respective return series, and thus can provide smooth mixing between the regimes. Finally, sign of returns are explicitly decomposed as , acknowledging greater regularity of absolute-valued return series.

Worth noting is the following are latent variables: idiosyncratic path from the informed regime, proportional factor , and regime parameter . Obviously, challenge of this model lies in their estimation. One potential trick is to exploit triangular relationships, as described below.

One stylized fact not explicitly accommodated by this model is well-known asymmetry of uninformed regimes, arising from analysis of market breadth: stocks uniformly go down together (think big down days), but much less often uniformly go up together (majority of rallies). Unclear whether this fact naturally arises via or needs to be explicitly modeled.

Readers familiar with machine learning (ML) may recognize how to reformulate this as an additive model:

Where .

This model can be interpreted in numerous ML ways, depending on the desired objective. For example, and can be interpreted as basis functions. Alternatively, boosting can be applied by interpreting them as weak classifiers. Graphical models can be applied by introducing conditional dependence between , , and . Hierarchical models and decision trees naturally arise when and are further functionally decomposed.

Given this model, an interesting question is how to use it predicatively—whether directional or not. For example, combining models for two stocks which share a common index to introduce the notion of equity triangle arbitrage on the joint .

Head back to curation and watch new algos emerge on top of that next-gen curation again. Think of Twitter as a new stab at curation. Curated sites will re-seed a new generation of algorithmic search sites. In short, curation is the new search.

Indeed, intent of curation here is to maintain high signal-to-noise ratio for a mix of preprint and classics in a highly-specialized literature (i.e. combo of retail and prop ) for which strong motivation exists elsewhere to obfuscate; and search over the stream provides ability to both rewind time and to integrate conceptual connectivity spanning time.

One addition being contemplated is keyword search over all literature cited in feed, providing deep content search over the feed. Although, unclear yet what is the best technical avenue to implement this (please comment, if you have suggestions).

So, with this positive start, curation input set is being modestly expanded to coincide with increased personal research activity and availability of several new quant sources—while maintaining the same focus and high signal-to-noise goal. Specifically, curation is expanding to include the following SSRN working papers: ARPM Series and JEL Codes G11 (Portfolio Choice), G12 (Asset Pricing), G13 (Contingent Pricing; Futures Pricing), G14 (Information and Market Efficiency), C21 (Cross-Sectional Models), C22 (Time-Series Models), C51 (Model Construction and Estimation), and C53 (Forecasting and Other Model Applications). Selection of JEL codes is data-driven: feed links were ranked by JEL classification and most cited classifications were chosen.

Authors are encouraged to ensure correct use of JEL codes, to ensure your articles are picked up.

Curious what readers think? Are there other high-value sources worth adding to curation input set? What else could make this more useful?

Early evidence suggests potential for significant intellectual cross-fertilization with finance, both mathematical and computational. Geometrically, richer modeling and analysis of latent geometric structure than available from classic linear algebraic decomposition (e.g. PCA, one of the main workhorses of modern finance); for example, cumulant component analysis. Topologically, more effective qualitative analysis of data sampled from manifolds or singular algebraic varieties; for example, persistent homology (see CompTop).

As evidence by Twitter followers, numerous Quantivity readers are familiar with these fields. Thus, perhaps the best way to explore is to seek insight from readers.

Readers: please use comments or twitter to suggest applied literature from these fields; ideally, although not required, that of potential relevance to finance modeling. All types of literature are requested, from intro texts to survey articles to preprint working papers on specific applications.

These suggestions will be synthesized into one or more subsequent posts, along with appropriate additions to People of Quant Research.

The classic discrete-time models for capturing such statistical conditionality are ARMA (see Box et. al (1994)) and GARCH (see Engle (1982) and Bollerslev (1986)), for returns and volatility respectively. Yet, therein lies a practical problem faced by hedge analysis: necessity to select a model with optimal parameters and error distribution for underlying and hedge. This post describes and implements such model selection for choosing a model from the universe of standard parameters and non-normal error distributions.

To illustrate this methodology for proxy hedging, begin by reviewing required statistical machinery. Next, illustrate results and graphical visualization with ongoing example of a well-known equity and QQQ. Finally, finish with R code.

Statistical Models

Three pieces of statistical machinery are required, briefly reviewed here: ARMA, GARCH, and BIC.

Conditional returns can be described by the ARMA(p, q) model:

The are autoregressive polynomial and is moving average polynomial. Recall from Why Moving Averages that any stationary stochastic process can be represented by an , due to Wold decomposition. Worth recalling is any model whose absolute parameter values are less than 1 is weakly stationary.

Conditional volatility can be described by the GARCH(m, s) model:

From this formulation, GARCH is nicely illustrated as ARMA applied to the squared series . The is sequence of iid variables, drawn from one of the following distributions: student-t, skew-t, or skew normal. Choice of student-t and skew are motivated by previous posts (and voluminous literature), which illustrate residuals following such distributions. Readers may wish to note relationship between this squared series and previous autocopula post.

Finally, the Bayesian Information Criteria (BIC) provides the statistic for ranking comparative model fitness of two more ARMA+GARCH models:

where is model-maximized likelihood value, is number of observations, and is number of free parameters. Thus, the best model is that with the highest log likelihood penalized by number of estimated parameters (i.e. ).

Proxy Models

The following performs model selection for CRM and QQQ, over the 5-year observation period:

crmModel <- selectProxyModel(ROC(p[,1], type="discrete", na.pad=FALSE))
qqqModel <- selectProxyModel(ROC(p[,2], type="discrete", na.pad=FALSE))

CRM is fitted as GARCH(1,1) with student-t errors:

       Estimate  Std. Error  t value Pr(>|t|)
omega  1.219e-05   5.781e-06    2.108    0.035 *
alpha1 3.552e-02   9.004e-03    3.945 8.00e-05 ***
beta1  9.544e-01   1.115e-02   85.600  < 2e-16 ***
shape  4.247e+00   5.227e-01    8.124 4.44e-16 ***

Thus, autoregressive coefficient is order-1, significant, and equal to 0.0355; variance coefficient is order-1, significant, and equal to 0.9544. Sum of coefficients is 0.9899, thus model possesses either IGARCH unit root or long-memory (e.g. Andersen (1999)). The residuals are student-t with 4.25 df, which is broadly consistent with the previous distribution fit undertaken in Empirical Copulas and Hedge Basis Risk.

QQQ is fitted as GARCH(1,1) with skew-t errors:

        Estimate  Std. Error  t value Pr(>|t|)
omega  2.833e-06   1.238e-06    2.288   0.0222 *
alpha1 1.005e-01   1.840e-02    5.464 4.64e-08 ***
beta1  8.956e-01   1.748e-02   51.228  < 2e-16 ***
skew   8.516e-01   2.971e-02   28.669  < 2e-16 ***
shape  6.620e+00   1.410e+00    4.694 2.68e-06 ***

Thus, autoregressive coefficient is order-1, significant, and equal to 0.1; variance coefficient is order-1, significant, and equal to 0.8956. The residuals are skew-t with df = 6.62 and skew = 0.852 (recall student-t is limit of skew-t with skew=0; see McNeil, Frey, and Embrechts [2005]). Note autoregression coefficient is nearly 3x larger for QQQ than CRM. Sum of coefficients is 0.9956, thus also possesses either IGARCH unit root or long-memory.

The preceding models can be visualized in several ways. To begin, visualize the conditional volatility for both instruments:

Volatility broadly matches intuition and plots from previous posts, with significant spikes during the financial crisis. As observed in previous posts, CRM volatility is significantly larger than QQQ. Note CRM has numerous significant volatility spikes not experienced by QQQ in 2009 and 2010, which correspond to earnings announcements (late-August and mid-November). Kendall correlation of conditional volatility between CRM and QQQ is 0.529, thus there exists a modest volatility dependence relationship:

cor(crmModel[3][[1]]@h.t, qqqModel[3][[1]]@h.t, method="kendall")

The copula of the GARCH volatility dependence exhibits large probability of absolute changes conditional on previous large changes, consistent with Lag Dynamics with Autocopulas:

Next, GARCH residuals exhibit some shortcoming in model fitness, given they retain modest heteroskedasticity:

Finally, consider plot of fitted GARCH residual distributions:

Which illustrates the significant difference in distribution, both versus each other and normality. Large positive skew for QQQ is particularly interesting. This plot is generated by, where p is the price matrix:

pROC <- ROC(p, type="discrete", na.pad=FALSE)
x <- seq(-6, 6, length=100)
cnames <- colnames(pROC)
x1 <- dnorm(x)
x2 <- dt(x,df=4.25)
x3 <- dst(x,df=6.62,shape=0.852)
plot(x, x1, type='l',lty=2, ylim=c(0,max(x1,x2,x3)), ylab="", main="GARCH Residual Distributions")
lines(x, x2, col=colors[2])
lines(x, x3, col=colors[3])
legend("topleft",legend=c("Normal",cnames[1],cnames[2]), fill=colors, cex=0.5)

library("fGarch")

selectProxyModel <- function(p, useMean=FALSE)
{
  # ARMA+GARCH model selection based on BIC, using the following error
  # distributions: student-t, skew-t, and skew normal. ARMA
  # may be either (1,1) or (2,2); GARCH is only (1,1).
  #
  # Args:
  #   p: single vector of instrument price data
  #   useMean: flag indicating whether mean should be fitted in GARCH
  #
  # Returns: selected GARCH model
  
  g11Std <- garchFit(~ garch(1,1), data = coredata(p), trace = FALSE, cond.dist="std", include.mean=useMean)
  g11SStd <- garchFit(~ garch(1,1), data = coredata(p), trace = FALSE, cond.dist="sstd", include.mean=useMean)
  g11SNorm <- garchFit(~ garch(1,1), data = coredata(p), trace = FALSE, cond.dist="snorm", include.mean=useMean)
  
  g11A11Std <- garchFit(~ arma(1,1) + garch(1,1), data = coredata(p), trace = FALSE, cond.dist="std", include.mean=useMean)
  g11A11SStd <- garchFit(~ arma(1,1) + garch(1,1), data = coredata(p), trace = FALSE, cond.dist="sstd", include.mean=useMean)
  g11A11SNorm <- garchFit(~ arma(1,1) + garch(1,1), data = coredata(p), trace = FALSE, cond.dist="snorm", include.mean=useMean)

  g11A22Std <- garchFit(~ arma(2,2) + garch(1,1), data = coredata(p), trace = FALSE, cond.dist="std", include.mean=useMean)
  g11A22SStd <- garchFit(~ arma(2,2) + garch(1,1), data = coredata(p), trace = FALSE, cond.dist="sstd", include.mean=useMean)
  g11A22SNorm <- garchFit(~ arma(2,2) + garch(1,1), data = coredata(p), trace = FALSE, cond.dist="snorm", include.mean=useMean)
  
  gModel <- list(g11Std, g11SStd, g11SNorm, g11A11Std, g11A11SStd, g11A11SNorm, g11A22Std, g11A22SStd, g11A22SNorm)
  gBIC <- data.frame(g11Std@fit$ics[2], g11SStd@fit$ics[2], g11SNorm@fit$ics[2], g11A11Std@fit$ics[2], g11A11SStd@fit$ics[2], g11A11SNorm@fit$ics[2], g11A22Std@fit$ics[2], g11A22SStd@fit$ics[2], g11A22SNorm@fit$ics[2])
  colnames(gBIC) <- c("g11Std-BIC", "g11SStd-BIC", "g11SNorm-BIC", "g11A11Std-BIC", "g11A11SStd-BIC", "g11A11SNorm-BIC", "g11A22Std-BIC", "g11A22SStd-BIC", "g11A22SNorm-BIC")
  
  minBIC <- order(gBIC,decreasing=FALSE)[1]
  
  return (list(gBIC, colnames(gBIC)[minBIC], gModel[[minBIC]]))
}

visualizeGarchModels <- function (p, m1, m2, cnames)
{
  # Visualize two garch models in overlapping plots
  #
  # Args:
  #   p: matrix of instrument price data, including valid colnames
  #   m1: first GARCH model to visualize
  #   m2: second GARCH model to visualize
  #   cnames: list of column names
  #
  # Returns: none
  
  t <- index(p)[2:nrow(p)]
  
  plot(xts(m1[3][[1]]@h.t, order.by=t), main="GARCH Conditional Volatility", type='l', ylab="Volatility", ylim=c(min(crmModel[3][[1]]@h.t, qqqModel[3][[1]]@h.t),max(crmModel[3][[1]]@h.t, qqqModel[3][[1]]@h.t)))
  lines(xts(m2[3][[1]]@h.t, order.by=t), col=colors[2])
  legend("topleft",legend=c(cnames[1],cnames[2]), fill=colors, cex=0.5)
  
  plot(xts(m1[3][[1]]@residuals, order.by=t), main="GARCH Residuals", type='p', ylab="Residuals")
  points(xts(m2[3][[1]]@residuals, order.by=t), col=colors[2])
  legend("topleft",legend=c(cnames[1],cnames[2]), fill=colors, cex=0.5)
}

Yet, while indeed true, this wisdom is not terribly helpful in practice for hedging well-known equities, as described in previous posts—as no instrument exists with such high correlation. This motivated revisiting the role of dependence in hedging, uncovering what may perhaps be an interesting result: multi-period asymptotically perfect hedges exist with .

To derive this result, begin by asking a simple-sounding question: over what range of correlation between underlying and hedge can a perfect hedge conceivably be built? When evaluated for a single period, the answer is obviously a single value: , as for any other correlation, given:

   .

Yet, this question becomes more interesting when generalized to multiple periods. Specifically, consider the temporal series of proxy errors over multiple contiguous periods:

Can intuition be built by modeling the temporal evolution of directly, rather than describing the characteristics of and ? Consider modeling using elementary trigonometry, say a familiar sin curve:

This model is interesting, as it captures the two desirable properties for optimal proxy hedging:

• Zero crossings: error crosses zero a positive number of times during its path (determined by zerocross parameter below), providing frequent opportunity to exit hedge with zero loss (i.e. )
• Absolute bounds: error is bounded, above and below, to not exceed some maximum threshold

These two properties should be familiar to readers who trade relative value strategies.

Given the above evolution of over time, next question is asking what range of is possible given arbitrary paths of and . As always, a picture is worth a thousand words.

Define to follow a stochastic process of length , whose value is drawn from Gaussian (given and parameters):

u <- rnorm(n, mu, sd);

In other words, the underlying follows a random process, which seems reasonable. Given that, is determined as the arithmetic difference, given discrete differential of and desired number of zero crosses for sin curve:

e <- (sin(seq(-zerocross*pi, zerocross*pi, len = n)) + 0.01) / 10
de <- diff(e)
h <- de - u

The following plots illustrates one sample path of , with corresponding , over the multiple periods (parameters n=200; zerocross=1; mu=0; sd=0.05):

These returns look fairly normal, albeit obviously sampled from a distribution which has neither long tails nor memory.

Given underlying and hedge, the correlation density can be generated by sampling :

cor(h, u, method="kendall")

The following plots illustrate the simulated scatter and empirical density for , given 1000 iterations (parameters n=200; zerocross=1; mu=0; sd=0.05):

With a single zero cross, this model recovers the classic one-period optimal hedge result: .

Where this model becomes interesting is when number of zero crosses is increased above 1. Doing so diverges the model from classical single period, conceptually extending time over multiple periods. The following plots illustrate returns when zero cross is 10:

Visual inspection of the cumulative returns plot makes clear something different is afoot. Clearly the underlying and hedge are not behaving as perfect inverses, when there are more zero crosses. There is something deeper at work. To illustrate further, consider the corresponding sampled correlation plots:

Illustrating correlation diverging from -1, peaking near -0.73. Recall this is correlation of underlying and hedge, which is providing asymptotic optimality at a finite number of points in time. Is this an accident, or does it represent a more general principle at work? Consider the following return plots for proxy error with 50 zero crosses:

Now the underlying and hedge look nothing like each other, except their broad directions are consistently inverted. Again, consider the corresponding correlation plots, which illustrate sampled correlation decreasing to density peak near -0.30.

This illustrates perhaps an interesting result demonstrating counterexample to the conventional wisdom of perfect negative correlation for proxy hedging, albeit only in theory.

proxyMultiPeriodCorrelation <- function(i=100, zerocross=20, n=200, mu=0, 
                                        sd=0.05, doPlot=FALSE)
{
  # Monte carlo sampling for visualizing multi-period correlation dynamics.
  #
  # Args:
  #     i: number of sampling iterations
  #     zerocross: number of zero crosses for error
  #     n: number of iterations for error sin curve
  #     mu: mean of Gaussian samples for u
  #     sd: standard deviation of Gaussian samples for u
  #     doPlot: flag to indicate whether to plot returns
  #
  # Returns: vector of sampled correlations
  
  e <- (sin(seq(-zerocross*pi, zerocross*pi, len = n)) + 0.01) / 10
  de <- diff(e)
  
  cors <- sapply(c(1:i), function(i){ 
    u <- rnorm(length(de),mu,sd); 
    h <- de - u;
    pair <- u + h
    c <- cor(h, u, method="kendall");
    
    if (doPlot)
    {
      oldpar <- par(mfrow=c(2,1))
      plot(u, type='l', xlab="Time", ylab="Returns", main="Optimal Proxy Returns");
      lines(h, col=colors[2])
      legend("topleft",legend=c("Underlying", "Hedge"), fill=colors, cex=0.5)
    
      cumH <- cumprod(1+h)-1
      cumU <- cumprod(1+u)-1
        
      plot(cumU,ylim=c(min(cumH,cumU,e),max(cumH,cumU,e)), type='l', ylab="Cumulative Return", xlab="Time", main="Optimal Proxy Cumulative Returns")
      lines(cumH, col=colors[2])
      lines(e, col=colors[3]);
      legend("topleft",legend=c("Underlying", "Hedge", "Error"), fill=colors, cex=0.5)
      par(oldpar)
    }
    
    return (c)
  })
  
  oldpar <- par(mfrow=c(2,1))
  plot(cors, ylab="Correlation", main="Optimal Proxy Hedge Correlation Scatter")
  plot(density(cors), xlab="Correlation", main="Optimal Proxy Hedge Correlation Density")
  par(oldpar)
  
  return (cors)
}

In a world awash with symbolic models, there is ample room for graphical exploratory analysis in finance—as the fine texture of the real world differs from both mathematical formalisms and standard mental models. Indeed, alpha hides in the divergence between model and reality.

Statistician John Tukey is one of the most well-known advocates of exploratory data analysis, captured in his 1977 book of the same title. Tukey nicely captures the essence of exploratory analysis in the opening chapter of his book (p. 1):

Exploratory data analysis is detective work–numerical detective work–or counting detective work–or graphical detective work.

This post illustrates graphical exploratory analysis, using R, specifically for the proxy hedging of previous posts: well-known tech company and QQQ. Although focused on a particular stock, these analysis techniques are applicable to other equities and higher frequencies.

Begin with scatters for daily prices and returns sampled over the previous 5 years, overlaid with fitted OLS and dispersion ellipsoids:

Top left plot illustrates several distinct price regimes. Top right plot illustrates ample returns well outside the dispersion ellipsoids, consistent with previous posts. Bottom left plot illustrates returns split by sign, demonstrating divergence in fit between positive and non-negative returns. Bottom right plot illustrates divergence between returns under and over 0.5 standard deviations, whose fit is quite similar.

Next, lag scatters for CRM:

Illustrating moderately non-spherical returns in the tails at all lags, consistent with the quantiles discussed in previous post. QQQ exhibits similar non-spherical lag returns, although the shape is not consistent with CRM:

To understand return dynamics in more depth, the following plots consider empirical density, absolute cross-correlation, difference ratios, and discrete rate of change:

Top left plot is textbook-style illustration of comparative excess kurtosis, with CRM return tails going out to +/- 10%. Cross correlation in top right plot exemplifies both forward and backward linear dependence for absolute returns, consistent with previous post on autocopulas. Difference ratios in bottom left plot illustrate empirical beta ranges from 1 to several hundred, irrespective whether returns are measured daily, weekly, or monthly—this corroborates difficulty of proxy hedging with linear instruments. These ratios provide the first evidence that late-2010 was perhaps even more anomalous for this pair than 2008, which is quite remarkable. Finally, bottom right plot illustrating return ROC further juxtaposes the comparison, as the ROC of both CRM and QQQ was dramatically higher in 2008 than 2010.

Finally, visualize the rolling proxy variance ratio (not to be confused with the standard statistical variance ratio, explored below) and correlation over the sample period with different durations:

Perhaps most interesting is the top plot which exemplifies that variance ratios (VR) for all roll durations are maximized over the 12 months beginning in August 2010. Contrast those ratios with the nearly flat VR, at all durations, during the financial crisis. The bottom plot provides one dimension of insight into this comparative behavior, illustrating that proxy correlation behaved inversely during these two periods: correlation was maximized during the finance crisis and minimized during 2010/2011.

Another interesting effect illustrated by both plots is temporal scaling behavior: correlation exhibits monotonic smoothing as lag window increases (conceptually similar to low-pass filtering), as exemplified by correlation progressive decreasing with black (daily), red (bi-weekly), blue (monthly), and green (quarterly). In contrast, the roll variance ratio exhibits no such scaling (monotonic or not). For example, early 2007 exhibits 60-day VR exceedingly all period periods. Similarly, the monthly VR frequently exceeds the biweekly VR.

Finally, consider classic unstandardized variance ratios for both CRM (top plot) and QQQ (bottom plot), calculated using the entire 5-year sample period, which illustrates mean-reversion versus trending (due to Lo and Mackinlay (1998)):

Both exhibit similar qualitative behavior: strongly mean reverting for holding periods under 30 days, followed by weaker reverting for longer periods. Variance ratio analysis is worth further consideration, particularly more rich graphical techniques (e.g. Lindemann et al. (2004)) and multivariate methods.

R code to generate the preceding exploratory hedge analysis:

require("tseries")
require("vrtest")
require("fSeries")

colors <- c('black', 'red', 'blue', 'green', 'orange', 'purple', 'yellow', 'brown', 'pink', 'coral', 'cyan', 'darkgreen', 'darkred' ,'darkblue', 'darkgrey');

exploreProxyHedge <- function(p, doMonthlyScatter=TRUE, freq="daily", doQuantilePlots=TRUE)
{
  # Plot various proxy hedge exploratory analysis.
  #
  # Args:
  #   p: matrix of instrument price data, including valid colnames
  #   doMonthlyScatter: flag indicating whether monthly scatter should
  #           be plotted
  #   freq: text string for frequency, used for graphing
  #   doQuantilePlots: flag indicating whether quantiles should be plotted
  #
  # Returns: None
  
  oldpar <- par(mfrow=c(2,2))
  
  # discrete first differences (not logged)
  pROC <- ROC(p, type="discrete", na.pad=FALSE)
  p1ROC <- ROC(p[,1], type="discrete", na.pad=FALSE)
  p2ROC <- ROC(p[,2], type="discrete", na.pad=FALSE)

  # scatter analysis
  plot(coredata(p[,2]), coredata(p[,1]), xlab=colnames(p)[2], ylab=colnames(p)[1], main="Prices")
  plm <- lm(p[,1] ~ p[,2])
  abline(plm, col=colors[2], lty=2)
  mtext(text=paste("slope=",plm$coefficients[2]), side=3, cex=0.75, col=colors[2])
  
  plot(coredata(p1ROC), coredata(p2ROC), xlab=colnames(p)[2], ylab=colnames(p)[1], main="Returns")
  rlm <- lm(p1ROC ~ p2ROC)
  abline(rlm, col=colors[2], lty=2)
    par(xpd=TRUE)
    d <- dataEllipse(as.vector(coredata(p1ROC)),as.vector(coredata(p2ROC)),draw=FALSE)
    lines(d[[1]], col=colors[2], lty=3)
    lines(d[[2]], col=colors[3], lty=3)
    par(xpd=FALSE)
  mtext(text=paste("slope=",rlm$coefficients[2]), side=3, cex=0.75, col=colors[2])
  
  # split positive/negative scatter analysis
  plot(coredata(p2ROC[(coredata(p1ROC)>0)]), coredata(p1ROC[(coredata(p1ROC)>0)]), xlab=colnames(p)[2], ylab=paste(colnames(p)[1], "(Positive Only)"), xlim=c(min(p2ROC),max(p2ROC)), ylim=c(min(p1ROC),max(p1ROC)), main="Split Sign Return Scatter")
  prlm <- lm(coredata(p1ROC[(coredata(p1ROC)>0)]) ~ coredata(p2ROC[(coredata(p1ROC)>0)]))
  abline(prlm, col=colors[1], lty=2)
  points(coredata(p2ROC[(coredata(p1ROC)<0)]), coredata(p1ROC[(coredata(p1ROC)<0)]), xlab=colnames(p)[2], ylab=paste(colnames(p)[1], "(Negative Only)"), col='red')
  nrlm <- lm(coredata(p1ROC[(coredata(p1ROC)<0)]) ~ coredata(p2ROC[(coredata(p1ROC)<0)]))
  abline(nrlm, col=colors[2], lty=2)
  abline(rlm, col=colors[3], lty=2)
  legend("topleft",legend=c("Positive", "Negative", "Both"), fill=colors, cex=0.5)
  
  # split magnitude scatter analysis
  magBound <- sd(p1ROC) / 2
  plot(coredata(p2ROC[(abs(coredata(p2ROC))>=magBound)]), coredata(p1ROC[(abs(coredata(p2ROC))>=magBound)]), xlab=colnames(p)[2], ylab=paste(colnames(p)[1], "(Positive Only)"), xlim=c(min(p2ROC),max(p2ROC)), ylim=c(min(p1ROC),max(p1ROC)), main="Split Magnitude Return Scatter (0.5 SD)")
  orlm <- lm(coredata(p1ROC[(abs(coredata(p2ROC))>=magBound)]) ~ coredata(p2ROC[(abs(coredata(p2ROC))>=magBound)]))
  abline(orlm, lty=2)
  points(coredata(p2ROC[(abs(coredata(p2ROC))<magBound)]), coredata(p1ROC[(abs(coredata(p2ROC))<magBound)]), xlab=colnames(p)[2], ylab=paste(colnames(p)[1], "(Positive Only)"), xlim=c(min(p2ROC),max(p2ROC)), ylim=c(min(p1ROC),max(p1ROC)), main="Split Magnitude Return Scatter", col=colors[2])
  irlm <- lm(coredata(p1ROC[(abs(coredata(p2ROC))<magBound)]) ~ coredata(p2ROC[(abs(coredata(p2ROC))<magBound)]))
  abline(irlm, lty=2, col=colors[2])
  legend("topleft", legend=c("Outer", "Inner (+/- 0.5 SD)"), fill=colors, cex=0.5)

  # monthly scatter analysis
  if (doMonthlyScatter)
  {
    plot(coredata(xts(p)["2011-05"][,2]), coredata(xts(p)["2011-05"][,1]), ylim=c(min(p[,1]),max(p[,1])), xlim=c(min(p[,2]),max(p[,2])), col=colors[1], xlab=colnames(p)[2], ylab=colnames(p)[1], main="Prices by Month")
    points(coredata(xts(p)["2011-06"][,2]), coredata(xts(p)["2011-06"][,1]), col=colors[2])
    points(coredata(xts(p)["2011-07"][,2]), coredata(xts(p)["2011-07"][,1]), col=colors[3])
    points(coredata(xts(p)["2011-08"][,2]), coredata(xts(p)["2011-08"][,1]), col=colors[4])
    points(coredata(xts(p)["2011-09"][,2]), coredata(xts(p)["2011-09"][,1]), col=colors[5])
    points(coredata(xts(p)["2011-10"][,2]), coredata(xts(p)["2011-10"][,1]), col=colors[6])
    legend("topleft", legend=c("05", "06", "07", "08", "09", "10"), fill=colors, cex=0.5)
    abline(lm(coredata(xts(p)["2011-05"][,1]) ~ coredata(xts(p)["2011-05"][,2])), col=colors[1], lty=2)
    abline(lm(coredata(xts(p)["2011-06"][,1]) ~ coredata(xts(p)["2011-06"][,2])), col=colors[2], lty=2)
    abline(lm(coredata(xts(p)["2011-07"][,1]) ~ coredata(xts(p)["2011-07"][,2])), col=colors[3], lty=2)
    abline(lm(coredata(xts(p)["2011-08"][,1]) ~ coredata(xts(p)["2011-08"][,2])), col=colors[4], lty=2)
    abline(lm(coredata(xts(p)["2011-09"][,1]) ~ coredata(xts(p)["2011-09"][,2])), col=colors[5], lty=2)
  }
  
  # quantile plots
  if (doQuantilePlots)
  {
    qqplot(coredata(p2ROC), coredata(p1ROC), xlab=paste(colnames(p)[2], "Returns Quantiles"), ylab=paste(colnames(p)[1], "Returns Quantiles"), main="Empirical Returns QQ-Plot")
    abline(0,1,lty=2)
    grid(20)
    par(xpd=TRUE)
    d <- dataEllipse(as.vector(coredata(p1ROC)),as.vector(coredata(p2ROC)),draw=FALSE)
    lines(d[[1]], col=colors[2], lty=3)
    lines(d[[2]], col=colors[3], lty=3)
    par(xpd=FALSE)
  }
  
  # Price lag dependence
  lag.plot(p1ROC, 9, do.lines=FALSE, main=paste(colnames(p)[1], "Returns Lag Auto Dependence"))
  
  lag.plot(p2ROC, 9, do.lines=FALSE, main=paste(colnames(p)[2], "Returns Lag Auto Dependence"))
  
  par(mfrow=c(2,2))
  
  # return distributions
  p1Density <- density(p1ROC)
  p2Density <- density(p2ROC)
  
  plot(p1Density, ylim=c(0, max(p1Density$y, p2Density$y)), main="Return Distribution With Median")
  lines(p2Density, col=colors[2])
  abline(v=median(p1ROC), lty=2)
  abline(v=median(p2ROC), col=colors[2], lty=2)
  legend("topleft", legend=colnames(p), fill=colors, cex=0.5)
  
  # cross correlation
  ccf(data.frame(abs(coredata(p1ROC))), data.frame(abs(coredata(p2ROC))), main="Absolute Return Cross Correlation")
  
  # diff ratio analysis (exclude periods with zero return QQQ)
  p1NoZeros <- diff(p[(p[,2] != 0),])
  p5NoZeros <- diff(p, lag=5)
  p5NoZeros <- p5NoZeros[(p5NoZeros[,2] != 0),]
  p22NoZeros <- diff(p, lag=22)
  p22NoZeros <- p22NoZeros[(p22NoZeros[,2] != 0),]
  
  diff1Ratio <- p1NoZeros[,1] / p1NoZeros[,2]
  diff5Ratio <- p5NoZeros[,1] / p5NoZeros[,2]
  diff22Ratio <- p22NoZeros[,1] / p22NoZeros[,2]
  
  plot(diff22Ratio, main="Difference Ratios", ylab="Diff Ratio", xlab="")
  lines(diff5Ratio, col=colors[2])
  lines(diff1Ratio, col=colors[3])            
  legend("topright", legend=c("lag-22", "lag-5", "lag-1"), fill=colors, cex=0.5)

  # ROC analysis
  plot(xts(p1ROC), ylab="ROC", xlab="", main="Return Rate of Change")
  lines(xts(p2ROC), col=colors[2])
  legend("topleft", legend=colnames(p), fill=colors, cex=0.5)

  # variance ratio analysis
  vratio <- sd(p[,1]) / sd(p[,2])
  cat(paste("variance ratio:", round(vratio,2)),"\n")
  
  vRatio5 <- rollingVarianceRatio(p,5)
  vRatio10 <- rollingVarianceRatio(p,10)
  
  np <- nrow(p)
  vRatio22 <- c()
  vRatio60 <- c()
  if (np > 22)
  {
    vRatio22 <- rollingVarianceRatio(p,22)
    if (np > 60)
    {
      vRatio60 <- rollingVarianceRatio(p,60)
    }
  }
  
  par(mfrow=c(2,1))
  plot(vRatio5, type='l', xlab="", ylab="VR", main="Rolling Variance Ratio", ylim=c(min(vRatio5, vRatio10,vRatio22,vRatio60), max(vRatio5, vRatio10,vRatio22,vRatio60)))
  lines(vRatio10, col=colors[2])
  
  if (np > 22)
  {
    lines(vRatio22, col=colors[3])
    if(np > 60)
    {
      lines(vRatio60, col=colors[4])
    }
  }
  legend("topleft", legend=c("5", "10", "22", "60"), fill=colors, cex=0.5)

  
  # correlation analysis
  vCorr5 <- rollingCorrelation(pROC, 5)
  vCorr10 <- rollingCorrelation(pROC, 10)
  
  vCorr22 <- c()
  vCorr60 <- c()
  if (np > 22)
  {
    vCorr22 <- rollingCorrelation(pROC, 22)
    if (np > 60)
    {
      vCorr60 <- rollingCorrelation(pROC, 60)
    }
  }
  
  plot(vCorr5, type='l', xlab="", ylab="Correlation", main="Rolling Correlation", ylim=c(min(vCorr5, vCorr10,vCorr22,vCorr60), max(vCorr5, vCorr10,vCorr22,vCorr60)))
  lines(vCorr10, col=colors[2])
  
  if (np > 22)
  {
    lines(vCorr22, col=colors[3])
    if (np > 60)
    {
      lines(vCorr60, col=colors[4])
    }
  }
  legend("topleft", legend=c("5", "10", "22", "60"), fill=colors, cex=0.5)

  # classic variance ratios (unstandardized)
  VR.plot(p1ROC,60)
  VR.plot(p2ROC,60)
      
  par(oldpar)
}

rollingVarianceRatio <- function(p, winLen)
{
  # Calculate rolling variance ratio with a given window length
  #
  # Args:
  #   p: matrix of instrument price data, including valid colnames
  #   winLen: length of window over which to calculate variance
  #
  # Returns: xts of rolling variance ratio
  
  return (xts(sapply(c(1:(nrow(p)-winLen)), function(i) { sd(p[i:(i+winLen),1]) / sd(p[i:(i+winLen),2]) }), order.by=index(p[((winLen+1):nrow(p)),])))
}
    
rollingCorrelation <- function(p, winLen)
{
  # Calculate rolling correlation with a given window length
  #
  # Args:
  #   p: matrix of instrument price data, including valid colnames
  #   winLen: length of window over which to calculate correlation
  #
  # Returns: xts of rolling correlation

  return (xts(sapply(c(1:(nrow(p)-winLen)), function(i) { cor(p[i:(i+winLen)], method="kendall")[2,1] }), order.by=index(p[((winLen+1):nrow(p)),])))
}