Skip to content

P-Q Convergence

September 21, 2011

Numerous smart people are foreshadowing a sea change in quantitative finance. This change has big alpha potential for the mathematically inclined, and will result in a much higher technical bar for those trying to learn algorithmic trading. And, pity those buy and hold investors.

Different folks are converging on this change in different colloquial ways. Derman has an upcoming book criticizing financial over-modeling. Taleb has been writing books on imprudent mathematical assumptions for many years. Haug has debunked Black-Scholes in several papers. Bouchaud and Potters have spent a decade popularizing econophysics by questioning classic dogma. Meucci wrote a book and has been teaching it for several years. Sornette is inventing quant macro by tackling crash modeling and prediction. Even big bank folks are jumping in, such as recent papers by Petrelli et al.

Despite different words, all boil down to the same fundamental change: convergence of the \mathbb{P} and \mathbb{Q} worlds. The financial crisis punctured the pristine mathematical world of \mathbb{Q} risk-neutrality, laying seeds for bi-directional synergy with the real world \mathbb{P}.

For those unfamiliar, \mathbb{P} and \mathbb{Q} are shorthand for two divergent finance traditions (see Meucci 2011). \mathbb{P} is the buy-side world of portfolio management, including retail, prop, and most of the fund world (as well as much of pension and insurance). \mathbb{Q} is the sell-side world of derivatives, best exemplified by exotics (and structured products, to a lesser extent). Historically, the two worlds could not be more different (e.g. compare mathbabe with Falkenstein).

To grossly and unfairly characterize their historical traditions: \mathbb{Q} is former physicists working on mathematically beautiful PDEs and stochastic calculus (hence similarity to statistical mechanics and related fields), driven by traders looking to book P&L and offload risk; \mathbb{P} are portfolio managers building investment models by applying fairly elementary statistics and optimization primarily from the Markowitz / Black-Litterman tradition. Notably absent from both worlds are descendants from old exchange locals who live in the absence of mathematics: spread, execution efficiency, and technology (including pure-play HFT).

The sea change is these two worlds are beginning to increasing blur, along with it the migration of alpha.

The first blurring was statistical arbitrage. Arguably the big quant funds are famous precisely because they recognized this \mathbb{P}\mathbb{Q} convergence and built corresponding large-scale modeling and execution expertise. In the past few years, this convergence is beginning to accelerate: take a few minutes to skim Wang 2009 and Petrelli 2010. Quantivity has never read more overtly confrontational academic papers (by big bank folks, no less), nor more use of italicized text. Yet, Quantivity nodded in affirmation while reading, having traded both stat and vol arb for years.

Most importantly, this blurring is popping up in retail quant projects. This is no theoretical debate reserved for academia.

Take two representative examples faced by millions of retail investors: proxy hedging and optimal derivative liquidation. Both have “solutions” from their respective literatures—yet they are ridiculously unsatisfactory in real life. Proxy hedging hails from \mathbb{P}, exhibiting near complete ignorance of \mathbb{Q} in its treatment of basis risk (while, of course, \mathbb{Q} assuming away residual risk). Optimal liquidation sounds ideal for \mathbb{Q}, but defies risk-neutrality because the holder faces an incomplete market. Practical solutions to both depend upon a concrete convergence of \mathbb{P} and \mathbb{Q}.

Generalizing beyond these two examples, the following are emerging characteristics of this converged \mathbb{P}\mathbb{Q} world:

  • Real life in \mathbb{Q}: risk-neutral fantasy disappears, uncovering a messy nest of \mathbb{P} problems
  • \mathbb{P} formalization: \mathbb{P} is being increasingly mathematically formalized, borrowing models from both \mathbb{Q} and theory from diverse areas of probability and statistics
  • Interdisciplinary: models which are large-dimensional, formalized, and calibrated require deep interdisciplinary knowledge spanning \mathbb{P}, \mathbb{Q}, and large-scale machine learning
  • Empirical: use of empirical statistics and monte carlo methods, rather than closed-form distributions and proofs, is pervasive to modeling the messy real world
  • Leadership: statisticians and computer scientists are accelerating \mathbb{P} (with mathematicians in tow), just as physicists did for \mathbb{Q}

This convergence represents potential for a beautiful renaissance of quantitative finance, opening the door to reconcile long-standing technical contradictions into solutions which can be traded in practice (without deluding ourselves on risk). How many strategies from this converged world have real scalability is a question yet to be answered.

Looking ahead, alpha will increasingly go to those who understand and trade consistent with this convergence. For those who do not, this will quickly evolve into a structure arbitrage.

7 Comments leave one →
  1. Cadogan permalink
    September 21, 2011 5:34 am

    I recently wrote a research paper on portfolio alpha from a P perspective, i.e. linear asset pricing via regression, and ended up with a Q model!! The paper is featured at What that means is that in the world of high frequency trading there is no P-Q difference because P converges to Q (i.e. P —-> Q). Most importantly, the model predicts that price reversal strategies will predominate. The math is fairly sophisticated but it does show how standard active portfolio management practices produce Q like models of SDEs.

    • quantivity permalink*
      September 21, 2011 11:36 am

      @Cadogan: I agree; here is a physics analogy, curious whether you agree: P is gravitation and Q is quantum mechanics; the two are converging into a unified theory via microstructure. Although this has been primarily applied in HFT thus far, there is no reason it cannot be applied to lower frequency. Only hurdle is P folks need to buy lots more computers and get accustomed to large-scale computation and machine learning (excepting the big quant funds, who presumably already do this).

  2. September 21, 2011 5:36 am

    What can I say? I *love* to disagree with numerous smart people, including the author of this blog. I’d like to give a view that is almost diametrically opposed to yours, which I would summarize as “it was either P or Q and it’s increasingly becoming P and Q”. I would argue that “it’s been P and Q for the past 25 years and it’s increasingly becoming neither P nor Q”. In other terms, I think it’s a false dichotomy. To wit:

    1. Quantitative portfolio managers have been using factor models since the late 70s (P world). Starting with Hansen and Singleton (1982) and Hansen and Richard, these models have been revolutionised by the Stochastic Discount Factor (SDF), which can be interpeted as the Radon-Nykodim derivative of Q with respect to P. Even though they don’t realize it (because they don’t read the papers), their P is full of Q. References are “Asset Pricing” by COchrane and “Empirical Asset Pricing” by Singleton.

    2. All option pricing econometrics is also informed by linking P to Q. E.g., read the survey by Garcia et al. in the Handbook of FInancial Econometrics (ed. Y Ait-Sahlia). The two have effectively already merged.

    3. As timescales shorten and microstructure comes to the fore, both P and Q lose meaning. There is no equilibrium price, no law of one price, no absence of arbitrage. Statistical arbitrage used concepts of an asset covariance matrix, mean-variance optimization, etc. This is slowly going away. It’s a neither-P-nor-Q world, but there is no unifying theory behind the approach.

    4. Except for Meucci, I don’t see any other author framing the problem in terms of P vs Q. They are either discounting and/or philosphizing about mathematical modeling in finance (in effect a “neither P nor Q” proposition) or making up phoenomenological models (yet again “neither P nor Q”). Regarding the latter, Econophysics has been promising revolutions for the past 15 years. I remember them doing their pitch to my Enron Research boss 11 years ago (who was a physicist). Even Enron didn’t buy into it. Since then, they have delivered preciously epsilon. If that’s potential for a beautiful Renaissance, good luck with that.

    Last point about “And, pity those buy and hold investors.” Since “hold” is a relative concept, I’ll read it as “pity those investors who think that P and Q are the 16th and 17th letters of the alphabet”. The “P and Q” part of the proposition already affected them, if at all. The “neither P nor Q” is decoupled from them by orders of magnitude in timescales, sizes and by roles (the buy and hold crowd clearly doesn’t belong in the sell side). They are simply complementary. Crowding in the latter space will reduce their profitability (it’s already happening). A world full of quantitative investors is one in which a fundamental investor can be formidably profitable. At the margin, those buy and hold investors will *benefit* from the neither “P nor Q” crowd. That partly explains why some of the most successful funds in recent years have been fundamentally driven: Paulson, SAC fundamental, Bridgewater, Harbinger, Citadel equities, Appaloosa, Icahn, and quantitative investors have shrinked badly in the past 3 years.

    • quantivity permalink*
      September 21, 2011 11:50 am

      @Gappy: thanks for your thoughtful comment, as always.

      To clarify: by “neither P nor Q”, are you implying the models from both worlds are being abandoned or are you implying the importance of fundamental is back on the rise (or something else)? Given this perspective, curious how you would reply to Petrelli et al.

      Can you elaborate on your supposition there is no unifying theory? Is modeling up from the ticks not sufficient for both, with microstructure models providing the mathematical bridge between P and Q (see my above reply to Cadogan)? As you observe, that is greatly preferable to trying to tinker with small sample covariance matrices.

      While I admittedly do not work in the industry (hence defer to your expertise, being at Citadel), my personal experience is there are still quite a few “pure P” and “pure Q” folks out there in the industry.

      Perhaps other readers can chime in on this question.

      • September 21, 2011 7:36 pm

        I was referring primarily to models of both worlds being abandoned. If you look at the microstructure literature, it’s mostly phenomenological in nature, with ad hoc results that explain stylized facts. For the existence of P and Q you need just two axioms: free portfolio formation and law and one price. For Q>>0 a.e. you need no-arbitrage; from which everything follows. At the order level there is no P or Q to speak of. For that sake, there is no canonical model to refer to.

        Regarding my “supposition” (in what sense of the word?) that there is no unifying theory, I’d say on the contrary that *there is* a unifying theory of P and Q (reference: Cochrane, or the first chapter of Duffie, in super-condensed form). That’s the “P and Q” part. On the other side, I don’t think microstructure research provides any foundation, at least for now.
        Furthermore, bridging the gap between microstructure and classical finance will be even more challenging than working out the semiclassical approximation between quantum and Newtonian Physics. More challenging, because microstructure is in its very early stages.

        Regarding Petrelli et al., it is a long paper and haven’t read it yet. At first sight, it seem yet another paper on dynamic hedging though.

  3. September 21, 2011 5:33 pm

    Thanks for the thoughtful post. I wasn’t familiar with the Meucci article, but I would just add a very basic observation: namely, I have never encountered a “pure” approach to anything in the industry.

    As an example, setting these modelling differences aside, there are certainly clear economic schools of thought that are in some sense diametrically opposed, such as around market efficiency. Academics (in principle) tend to view these theories as “pure”: markets are either behavioral or efficient. Yet even industry specialists who fall into one of these schools (such as economists trained at U. Chicago) tend to not fully believe the theory. In practice, even academics rarely believe in pure theory.

    When it comes to P (derivatives modelling, stochastic processing, etc.) vs. Q (portfolio management, statistics), I agree that this is a distinction worth noting (i.e. it is meaningful and does exist) but it is also not black or white. It is a fact that Financial Engineering programs (and the Wilmott clan) are almost entirely focused on derivatives pricing. People engaged in these programs will be beaten over the head with Ito’s lemma. It is a whole world built on top of black-scholes. And it is furthermore true that portfolio management is an entirely different area of expertise based mostly on optimization problems. But in reality, people in both areas are using all available approaches. It would be foolish to ignore a useful method if it would lead to a stronger result. People are not intent on finding pure answers: they want what works.

    If I was going to argue for “what’s next”, I would be more inclined to expect that something new will come along. There’s no reason to think that the existing derivatives pricing and portfolio management will all of a sudden converge. It is more likely that new and unexpected things will arise. Why shouldn’t they? The existing methods are flawed in so many well-understood ways.

    And I would agree with Gappy’s observation regarding successful managers using fundamental information. I might even push it a little farther. The criticisms of Taleb (and I expect from Derman’s new book) seem to be arguing for more human judgement and human risk management, not less of it.


  1. You Don’t Have Alpha « Quantivity

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: