ED vs. LOGICAL FALLACIES, PART 16: SURVIVORSHIP BIAS

This isn't a proper or formal logical fallacy. I don't care. It's a flawed pattern of logic and it's important.

Do you ever wonder why every commercial you see on TV for a mutual fund is able to claim(accurately) that their Super Spazz Fund is one of the top 10 or 20 percent of all mutual funds in terms of performance? And it's not simply that they only advertise the winners. Do a little research online and you'll find that almost every fund from a major company carries with it some sort of performance superlative. Top ten, top quarter, top something. That's rather odd, don't you think?

In a statistical sense, this is called survivorship bias – you only see the funds that didn't fail. T. Rowe Price or Fidelity or Strong want you to think that they are responsible stewards of their investors' money. When they have a fund that performs like shit, they either close it or merge it into a successful fund that will hide its losses. So the reason that every fund you hear about claims to be one of the top 20% of all mutual funds is simple – the other 80% have failed and likely no longer exist. Not all industries have the benefit of this kind of sleight-of-hand. When US News and World Report creates its annual list of college rankings, academia cannot fluff its image by disbanding the Thomas Cooleys and Arizona States. I recommend Elton, Gruber, and Blake's excellent "Survivorship Bias and Mutual Fund Performance" for a more detailed look on how this skews the face the industry shows the public.

Lest we pile all of our scorn on the financial sector, survivorship bias is pervasive throughout much of the research and many of the statistics you see on a daily basis. The pharmaceutical industry is a noteworthy offender. Studies of the effectiveness of certain classes of drugs omit those that have been pulled from the market or failed to win approval. Trials of individual drugs judge effectiveness based on participants who complete the study, noting only in the fine print how many dropped out (possibly, of course, because the drug was having no effect).

And lastly, what amateur logician or statistician is not driven to fits of laughter by the car commercials that talk about sterling grades in "owner satisfaction" surveys? Call me crazy, but I think that a lot of people who are dissatisfied with a car voluntarily remove themselves from the "owners" category by selling it. An owner satisfaction survey is a survey of people who liked the car enough to keep it.

Be Sociable, Share!

2 Responses to “ED vs. LOGICAL FALLACIES, PART 16: SURVIVORSHIP BIAS”

  1. Mike Says:

    Survivorship bias is a big deal, and you are right on with how it skews statistics with mutual funds.

    If you want to see the mutual fund industry look even worse (and can handle some finance theory), I believe this is still the big word on the subject:

    http://persistence.behaviouralfinance.net/Carh97.pdf

  2. Mike Says:

    And why not. This paper RULES; even if you don't follow the IV, the idea of what they are actually testing against CEO pay is quite clever:

    http://www.economics.harvard.edu/faculty/mullainathan/files/ceopay.pdf

    (I remember a huge fight breaking out over class as to whether or not CEOs should get paid for 'luck' – ummm, no.)