We are beginning to talk about probability in a course on research design / methodology, and I introduced the topic with a few basic examples. One of them is the infamous Monty Hall Problem. The MHP is like an optical illusion – no matter how many times I tell you that these lines are of identical length, your brain will keep telling you that the bottom line is longer. It is the kind of paradox with a solution that appears to be absurd even after it has been demonstrated to be correct. You simply cannot wrap your mind around it.

The MHP is a probability game named after the host of the 1970s/early 1980s game show Let's Make a Deal. The most popular part of the show was a game that has since become the subject of dozens of academic papers in math, statistics, and logic. Here is the basic setup:

The host confronts the contestant with three doors. Behind two doors there are goats (which the contestant did not actually win, but were intended as a gag "prize") while the third door hides a brand new car. The three prizes are distrubuted behind the doors randomly and Monty Hall knows what is behind each door. The contestant chooses a door, which remains closed. Hall must then open one of the other two doors to reveal a goat. He cannot open the door hiding the car (if the contestant has not chosen it). He then asks the contestant if they would like to stick with their original choice or switch to the other available mystery door.

An example is clearer; You choose Door 1. Hall opens Door 2 to reveal a goat. He then offers you the option of sticking with Door 1 or changing your choice to Door 3.

To 99.9% of humanity, this problem has an easy solution. Since Hall will open one of the two goat doors, the two unopened doors contain one goat and one car. There is no advantage to switching, since the odds of the chosen door containing the car appear to be 50-50. Going back to the example, if Door 2 is opened to reveal a goat, there's a 50% chance that Door 1 contains the car and a 50% chance that Door 3 contains it, right?

Switching, in fact, is always in the player's interest. When Parade magazine ran this problem in 1990 they received more than 10,000 critical letters, including several hundred from people with PhDs in math and science. All patiently explained that the odds are 50-50 and switching is not beneficial. The well educated letter writers were all wrong.

The probability of winning by staying with the original choice is 1/3, not 1/2. And the probability of winning by switching is 2/3. Most people, no matter how long they stare at this and work through the scenarios, cannot accept that. It took me days to absorb it when I first encountered the problem. It just does not make sense. Yet it's true. Since there are only two different objects behind the three doors – two goats and one car – there are only three possible combinations: car-goat-goat, goat-car-goat, and goat-goat-car. Perhaps the only way to understand why this creates a 2/3 probability of success by switching is to see the three scenarios spelled out. The yellow represents closed doors, the white door is the one opened by Hall to reveal a goat, and the arrow represents the door originally chosen by the player (which is always Door 1, for simplicity's sake).

And now that I've explained it and you've seen the irrefutable evidence, I bet your brain still doesn't comprehend why switching increases your odds. Mine doesn't. I know what the solution is but it will take me a few more years to understand why.

To paraphrase 50 Cent, I love this problem like a fat kid loves cake.

Coca-Cola Co. Chairman and Chief Executive Officer Muhtar Kent said the idea of a federal tax on soft drinks, under consideration by the U.S. Congress and President Barack Obama, is “outrageous.”

“I have never seen it work where a government tells people what to eat and what to drink,” Kent said today, responding to an audience question at the Rotary Club of Atlanta. “If it worked, the Soviet Union would still be around."

If there is a lamer or more transparently juvenile way to defend a right-wing argument than by invoking the Soviet Union – which serves as a combination Red Herring, Straw Man, and false dilemma – I don't know what it is. It has surpassed Reductio ad Hitlerum as the favored fallacy of irrelevance given our new "socialist" President and a nation full of people who don't know what socialism is but are quite sure it's going to rape their daughters. But there are some very good reasons why Mr. Kent should tread more lightly than most in bashing centralized planning.

Our government doesn't tell people what to drink – although "suggestions" abound in the form of dietary guidelines and public health campaigns – but it sure as hell has spent the last century telling them what to grow. Namely corn. Metric shitloads of corn. The Federal government pays people to grow corn. It pays people not to grow corn. It pays people to think about growing corn. This is neither debatable nor controversial, having entered the mainstream of political knowledge with the ethanol debate and popular edutainment like the film Food, Inc. or books like Fast Food Nation, Animal Vegetable Miracle, or The Omnivore's Dilemma. Government policy has rewarded farmers (OK, agribuisness) for growing heathen quantities of corn without regard for the market or consequences and has thus engineered the flooding of the planet with mountains of cheap corn.

No person or entity has benefited more from this legislatively-mandated glut of cheap corn than the soft drink industry dominated by Mr. Kent's company. The pro-corn policy (Cornservative policy? Yes, I like that better.) has created an entire industry of chemists and biologists dedicated to finding some way to use all this shit that Congress pays farmers to grow. One of their earliest successes, aided heavily by the legislatively mandated ban on Cuban sugarcane, was corn-based liquid sweeteners. Expensive sugar would have produced expensive soda but cheap corn syrup produces oceans of cheap soda. And to call corn sweeteners "cheap" is an understatement. The market is so saturated that we could feasibly fill Lake Baikal with Karo for about $50.

OK, not quite that cheap. But the point is that the government has spent a century "telling people what to drink." It has been telling them to drink Coke by going to great lengths at public expense to ensure that a two-liter soda is half the price of a half-gallon of orange juice (which, if from concentrate, has corn in it anyway). Maybe Comrade Kent should thank the Politburo for raiding the public till to help his Collective Beverage Farm push its 200 calorie cans of corn syrup on the lard-assed Proletariat with no risk of failure. Raise your Sprite to the Motherland!

The need for health reform is as plain as the headline on the front page of Tuesday's editions of The Buffalo News: "Salaried employees suffer loss of health care, reduced pensions due to Delphi bankruptcy." The system is falling apart. The wailing from the political right notwithstanding, without reform, we will have rationing.

But those on the left who continue to deny that controlling costs must be the first order of business need to read the same story. Today it's Delphi Corp. in Lockport, a company that is under severe economic strain. But as the costs of health care continue to soar — reaching 20 percent of gross domestic product by 2017 — more companies will find themselves under severe economic strain. Employer-based health care will become increasingly less available.

It's only a matter of time until we are besieged with warnings about capping malpractice claims.

Costs are not causing the problem, they are the end result of it. Drawing upon my experience in the world of medical collections – briefly, lest I start having flashbacks – the average hospital writes off tens of millions of dollars in services annually. That is to say lots of people are receiving services for which they don't pay. We know that emergency rooms are required by Federal law to treat any patient who presents himself regardless of ability to pay. For people without health insurance and with meager incomes, this becomes the sole source of medical care. Hospitals attempt to compensate for services they essentially provide for free by breaking it off in the ass of every patient with half-decent insurance. So the next guy through the door with a Blue Cross PPO gets charged $75 for an ibuprofen in an effort to make up the difference. Hospitals then pay insurers a discounted bulk rate for claims – thousands at a time – according an arcane formula that ends up being slightly more complicated than a group of astronauts doing their taxes in Latin. In short, they charge you $75 to get a negotiated payment of $60 from your insurance to start compensating for the fact that they just did a $15,000 operation to pull the steering column of a 1994 Toyota Tercel out of the sternum of an uninsured teenager.

The costs can't be controlled prior to extending coverage to everyone. The presence of 50 million uninsured people is in fact contributing mightily to the disconnect between the costs of medical care and reality. Bask in the rich irony as the Glenn Beck fan in the next cubicle wails loudly about how he isn't going to pay for anyone else's health insurance. We already pay for the uninsured. It's just a big, disorganized trainwreck, clearly superior to a government-administered program which would achieve the exact same end result.

Economists are useful people to have around in the academic world. They have great quantitative skills and, in my experience, a keen eye for research design. Unfortunately most of them think they are social scientists. They aren't. They are mathematicians practicing a hard science. They should not do political science, biology, or sociology any more than practitioners of those disciplines should do economics. But economists have the misfortune of seeing the entire world in rational choice terms and a field which encourages them to make the most immoderate, speculative conclusions possible based on their findings.

Prof. Sacerdote, for example, could have written a paper in which he said "Here is a very interesting disparity I have uncovered. I hope this encourages others to study this issue and find out what's going on here." Nah. As economists like to do, he just solves the dilemma himself at the end of the paper: it's genetic. People who make more money are smarter than people who make less, and that intelligence is passed on to their children. See how much easier that was compared to doing all that messy "research" and using "logic"?

Similarly, from the are-you-fucking-kidding file we have this entry on Harvard economist Greg Mankiw's blog. This man is a very good economist. He is famous. He has achieved much in his academic and professional career. Now read this:

The NY Times Economix blog offers us the above graph, showing that kids from higher income families get higher average SAT scores.

Of course! But so what? This fact tells us nothing about the causal impact of income on test scores. (Economix does not advance a causal interpretation, but nor does it warn readers against it.)

This graph is a good example of omitted variable bias, a statistical issue discussed in Chapter 2 of my favorite textbook. The key omitted variable here is parents' IQ. Smart parents make more money and pass those good genes on to their offspring.

Suppose we were to graph average SAT scores by the number of bathrooms a student has in his or her family home. That curve would also likely slope upward. (After all, people with more money buy larger homes with more bathrooms.) But it would be a mistake to conclude that installing an extra toilet raises yours kids' SAT scores.

Is this a joke? He opens by recognizing that the data offer no evidence whatsoever about causality and closes with a warning about making spurious and unwarranted conclusions about causality when correlation is present. Which is cute, because between those two statements he grabs his ankles, reaches deep within his ass, and pulls out the uncited, unwarranted, and baseless conclusion that the real causal mechanism here is genetics. A goddamn college freshman could look at the relationship between income and SAT scores and conclude, "Hmm. Well, parents with more money can afford expensive schools and SAT prep courses." That is just about the most basic example of cause and effect one could imagine.

If you took your car to a mechanic because the alternator was shot – you can open the hood and plainly see that the alternator is burnt out and not functioning – a reasonable mechanic would say "Hey, you need a new alternator." If you took the same car to an economist, he or (rarely) she would say "The problem is obvious. Your parents have low IQs, and thus you have inherited genes which make you too dumb to pick out a car that will be free of mechanical problems." If that analogy seems ridiculous, that is exactly what Mankiw has done here. He has ignored the overwhelmingly obvious explanation and substituted his own imagined causal mechanism.

Economists may not suck at causal inferences universally, but the ones that do it well and judiciously are well-hidden. Their field provides perverse incentives to draw attention to one's work by stating the biggest, most shocking conclusion that can be wrung from the data, however tenuous the evidence. Either that or they just suck at playing social scientist. Causality is determined by hypothesis testing and supported with evidence. When hypotheses can't be thoroughly tested, statements about causality should at least adhere to basic logic. Showing a correlation and then going off half-assed on one's own preferred explanation of How the World Works is not science. And it is particularly unwelcome when that preferred explanation happens to be 19th Century eugenics and a handful of social Darwinism.

THE DOW-JONES INDUSTRIAL AVERAGE since the passage of the stimulus bill. Looks like a vote of “no confidence” to me.

In the five months since he decided that the DJIA was a reliable indicator of Presidential and/or policy performance the benchmark index has risen by 37%. 37%! So tomorrow Glenn is going to write about how the Market God has bestowed upon our President the most confident of votes.

Noting that two things happen at the same time and assuming that causality exists between them is called spurious correlation. It is what stupid people do when they are getting ready to make really bad predictions – not just the bad "I think this is the Cubs' year" kind, the epic-bad JFK "I think it's OK to leave the top off the convertible today" kind – or lose a lot of money in the stock market. The fact that both the stimulus and the Dow Jones Average are related to the same central theme (our macro-economy) creates the reassuring but incorrect sense that correlation does in fact imply causation. False. Anything can be correlated with market movement and causality is always – always – a bad assumption. The market is like secret recipe hobo stew: there are so goddamn many ingredients, many of which the average person does not care to know about, that no one can say with certainty what causes what.

To wit: let's say I shaved my balls on Friday. Note that the market responded positively, only to once again decline as a vote of no confidence when the team of international observers (led by Jimmy Carter, former Canadian PM Brian Mulrooney, and the Dalai Lama) detected the growth of stubble.

The evidence is beyond dispute. My balls move markets.

Back in January the USA Today noted that the stock market performs significantly better – nearly 25% – in years in which the Steelers win the Super Bowl. And that America-hating prick Larry Fitzgerald almost ruined it all.

The market is not rational regardless of how many business school professors pitched tents over its majesty as an arbiter of every political, social, and economic process. Remember, the market once told us that DrKoop.com was worth $45/share. It decided that theGlobe.com was worth $97/share (3 months later: a dime). It is not a parliament which issues meaningful votes of confidence and no confidence. Only a long view of market trends can provide useful (and retrospective, mind you) insights. Trying to use its day-to-day and week-to-week fluctuations as evidence of one's preferred version of current events is logical to the same extent that cavemen banging on drums to make the sun rise (and it worked! every day!) made sense.

Attributing the characteristics of a group to its individual members lies at the root of a vast portion of the bad logic in this world (and certainly a majority of the bad social science). In the most basic sensee, aggregation destroys data. And whether in life, politics, or academia, we often want but cannot have that data. We don't know what each member of an audience thinks of a performance; all we know is that the crowd applauded at the end. We want to know what kind of people voted for Obama, but we know only that 54% of voters did. We want to know if liberals or conservatives are experiencing more home mortgage foreclosures, but the only information we have are general foreclosure statistics.

A basic fallacy of aggregation assumes the actions or decisions of an individual based on a group. Mike was at the concert and the crowd cheered loudly, so Mike must have enjoyed it. White people like Arrested Development, so Ed likes it. These may be good guesses, but playing the odds is not the same as being logical. Knowing that Jim Inhofe is a Senator and that the Senate is about to confirm Sotomayor, would we conclude that Mitch McConnell voted to confirm her? Yeah, not really.

A second kind of fallacy is particularly prominent in sociological, political, and economic research because of the preponderance of pooled data (election results, jury verdicts, unemployment rates, the Gross National Product, etc.). A famous sociologist termed it the "Ecological Fallacy" in 1950. An ecological fallacy correlates pieces of aggregate data without evidence that a relationship exists among the data. Here is a basic example.

In the 2008 Election, residents in a particular state were asked to vote on Proposition 1. In City A, Prop 1 got 5% Yes votes. In City B, the Yes votes were 40%. Fine. But we also note that 5% of the population in City A is Latino – as are 40% of the residents of City B. Ergo we conclude, seemingly quite logically, that Latinos voted for Prop 1 and non-Latinos didn't. The percentage of Yes votes and Latinos is equal in both cities. Lacking detailed data about who voted for what and why, commentators often make leaps of faith along these lines. Here's the problem. What if Prop 1 is, "Should public services throughout the state, including schooling, be conducted solely in English?" City A has few Latinos, the ethnic group most likely to want or need services offered in a different language. Because there are few people who are likely to be native Spanish speakers in City A, voters there neither think nor care much about English-only laws. But in City B, the large Latino population makes the issue highly contentious and polarizes non-Latino voters. So City B is 40% Latino, but the 40% voting for Prop 1 are the white people who feel threatened by the multilingual environment in which they live. The fact that the Yes votes and percentage of Latinos in each city are equal does not imply a direct correlation. The true relationship among the data, in this hypothetical, is entirely different than the numbers would suggest.

The foreclosure maps which are popping up in newspapers and around the interwebs are just too tempting for many people. Note the county-level foreclosure rate, throw in some election data, and start making conclusions about partisan balance and imprudent lending. Were it that simple, I and most of my colleagues would be out of a job and contributing mightily to the foreclosure landscape. The data lost in aggregation are often of great interest but no amount of rationalization can recreate them from the pale substitute of numerous data points smashed together in one big, indistinct pile.

The logic is so flawless it's stunning.

"I can eliminate $100 billion of wasteful and earmark spending immediately–35 billion in big spending bills in the last two years, and another 65 billion that has already been made a permanent part of the budget." — Sen. McCain, All Things Considered, April 23, 2008

How does Gramps McCain's mysteriously-derived estimate of earmark spending compare to calculations by organizations that bother with "facts" and "numbers"?

John McCain: earmarks = $100 billion
Anti-tax hawks Citizens Against Government Waste: FY 2008 earmarks = $17.2 billion
Anti-tax hawks Taxpayers for Common Sense: FY 2008 earmarks = $18.3 billion

McCain magically (and without explanation, even when pressed, of where the $100 billion figure comes from) quintuples the amount of "wasteful spending" calculated by people who follow such matters for a living. More importantly, the premise of his argument is downright ridiculous when we look at earmarks in context. Yes, yes, I know. Republicans hate context. Context has a pronounced liberal bias. Consider the following.

FY 2008 earmark spending: ~$18,000,000,000 (#1 state: Alaska)**
Annual cost, tax-deductible home mortgage interest: $65,000,000,000
Bailout of sinking insurance giant AIG: $85,000,000,000
FY 2008 Iraq War spending: ~$155,000,000,000
FY 2008 cost of Bush tax cuts: $189,100,000,000
FY 2008 cost of Bush tax cuts with AMT "relief": $247,400,000,000
FY 2008 interest on the Federal debt: $261,000,000,000
FY 2008 Dept. of Defense budget (excluding Iraq): $481,000,000,000
FY 2008 Social Security: $608,000,000,000
Financial "rescue" plan: $700,000,000,000
FY 2008 Federal Budget: ~$3,000,000,000,000
National Debt as of 9/30/08: ~$10,000,000,000,000

FY 2008 Earmarks as a percentage of the Federal budget: 0.6%

That's right, folks! Vote for John McCain and in the fantasy world in which he can eliminate all Congressional earmarks (using the line-item veto….which was just declared unconsitutional by Scalia and Thomas ten years ago) you can maybe, kinda, potentially (but probably not) save six-tenths of a percent of the annual Federal outlay of your hard-earned dollars! Yes, truly those wasteful earmarks for things like cancer research and military aid to Israel (which McCain hates, right? Let's ask Joe Lieberman) are the cause of our economic difficulties.

Let's go one step further, critically examining the common right-wing bitching point about "unnecessary" Federal spending of the non-earmarked kind.

FY 2008 budget, Environmental Protection Agency: $7.2 billion
FY 2008 Deparment of Labor budget: $10.6 billion
FY 2008 Department of Interior budget: $10.6 billion
FY 2008 Treasury Department budget: $12.1 billion
Iraq War in FY 2008: $12.9 billion per month
FY 2008 Bush tax cuts to top 1% of income earners alone: $79.5 billion

For the mathematically challenged – and polling indicates that about 45% of you are – the annual budgets of your least favorite Cabinet agencies, if completely eliminated, would not fund the Iraq War for 30 days. To put it another way, the combined budget of Labor and Interior ($21.2 billion) is 4.4% of the Department of Defense budget or 3.5% of the cost of Social Security for one year. This year we spent four times as much giving tax cuts to the ultra-wealthy (top 1% = incomes greater than $275,000) as we did on the Departments of Labor and Interior combined.

The point, and one that will surely be lost on McCain, is not that earmarks or the Department of Labor are productive uses of our financial resources. The point is that eliminating them would not make the slightest goddamn difference when considered in the context of our Federal budget. It is a spit in the ocean. A grain of sand in the Sahara. A snowflake on a glacier. "Wasteful" or not, earmarks and the budgets of agencies most loathed by conservatives represent piss-ant sums compared to the items that perennially dominate the budget. Uncle Sam saving the money spent on earmarks would be about as meaningful as a millionaire with $100 million in debt finding a quarter in his couch. McCain's earmark crusade fits snugly into the narrative about those shiftless tax-and-spend libruls. But like so much of the conservative canon, it makes sense only as long as one remains stunningly ignorant of the facts.

**Is life hilarious or what? I could not make this shit up if I tried.

Like emotionally-healthy former President Richard Nixon, Bill O'Reilly has an "enemies list" of sorts. He calls it the "Coward List." It contains more than two dozen names across the political spectrum, including Dick Cheney, "the heads of oil companies," Howard Dean, "NPR" and Jane Fonda. This list of people – and entities, I suppose – share one thing in common. They all refuse Bill's generous invitations to appear on The Factor. They refuse to do this because they are afraid, of course, to expose themselves to the superior intellect and rhetorical powers of Bill O'Reilly.

Early in this presidential primary season, nearly every Democratic candidate refused to appear on a Fox News-sponsored debate. Their motive, of course, is that they were too afraid to expose their indefensible, hysterical politics to the light of truth and fairness that is a Chris Wallace/John Gibson moderated debate.

There is a subset of people in the world with very high opinions of themselves, opinions as high as their rhetorical skills are shitty, who interpret your refusal to have anything to do with them as an endorsement of their beliefs. You don't want to debate Bill O'Reilly because, goddammit, you just know he's right. Your refusal is enough evidence to prove that you are wrong. If you're right, why wouldn't you go on the show? Makes sense to me!

Like all non causa fallacies, this statement could potentially be true. Maybe Jane Fonda really is afraid to be proven wrong and she knows BillO will do it. Yeah, that's one possibility. I guess. Or maybe she doesn't want to debate him for the same reason that she doesn't want to debate the homeless guy who drops his pants for nickels at the bus station.

Declining to debate a person or group probably has a lot more to do with your maturity level and tolerance for stupidity than the validity of your argument. Maybe you don't like arguing with people who aren't intelligent enough to realize when they are proven wrong. Maybe you don't enjoy people who refuse to admit being wrong no matter how obvious you make it. Maybe you're not out to change the mind of every retard you meet on the internet or in a bar. Maybe you can detect situations in which another person simply wants to yell at you rather than have a real discussion. These are all valid reasons, all of which are more plausible than fear.

When someone makes him- or herself a rhetorical pariah, refusing to play the game makes perfect sense. What makes no sense at all is drawing a conclusions about an argument based on how willing people are to argue with their drunk, ranting uncle or the guy talking to himself on the bus.