Raise your hand if you love statistics and want to do a survey

by

‘I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s?’ That was Hal Varian last year in the McKinsey Quarterly.

It’s been hard to miss this kind of talk lately. A few weeks ago The Economist chimed in with a ‘special report’ on ‘the data-centred economy’.

Unfortunately, one prominent growth sector seems to involve hucksters selling meaningless information to anyone dumb, cynical or rich enough to buy. This can be done in several ways, but a few are especially common:

  • Claiming to detect an effect where the coefficient isn’t statistically significant: the observed result could have occurred by chance rather than genuine association (for example, evolutionary psychologist Satoshi Kanazawa purporting to show that ‘beautiful parents have more daughters’ while tall people and engineers have more sons).
  • Failing to control for potential confounds, and thus finding spurious correlations. As previously discussed here, this one is beloved of blowhard journalists who, for example, tie crime figures to ethnicity (look how Sudanese offenders are over-represented! look how Indians are disproportionately victims!) without adjusting for relevant variables like age, gender, occupation and locality.
  • Selection bias, where the sample drawn is unrepresentative of the target population, with some people more likely to be included than others. If you see market research in the popular media, it’s almost guaranteed to have this flaw. Respondents to online surveys usually self-select. They may choose to participate because they have some axe to grind, while people without strong opinions – or older people who don’t use the web – may not bother taking part. 

It’s always been easy to sell equivocal results to a credulous media, which is eager to hype and sensationalise quantitative findings if they involve sex, gender or race. But, increasingly, public and private-sector managers seem willing to pay for noise presented as signal. Consider this week’s report in Melbourne newspaper The Age. Apparently ‘research done for the government’ shows residents of the inner-western suburbs are worried about a proposed freeway tunnel. The research referred to seems to be a local community survey hosted online at SurveyMonkey. Fill it out, if you like; it takes only 15 minutes of your time, and is almost completely worthless. And – hey! – your tax dollars, by way of Sweeney Research, have already paid for it.

Advertisements

Tags: , ,

2 Responses to “Raise your hand if you love statistics and want to do a survey”

  1. christinachurl Says:

    What grinds my gears is when out-of-date or discredited statistics (and “scientific studies”) are commonly cited as reliable sources of data, like that old gem “It’s been medically proven that eating chocolate doesn’t cause acne.” Also, those crappy studies they quote in the MX that say things like “Men are more likely to chew gum and cheat on their wives while at work, whereas women tend to think about getting married while shopping for weight loss drugs, according to a UK study conducted by researchers at Zoo Weekly, of its regular readers.”

    Spew.

  2. Nick Says:

    Yes, that’s annoying, but it’s really just entertainment. We all love to generalise from a few observations, especially about sex.

    What’s original, it seems to me, is bureacrats paying loads of money for data they know to be worthless.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: