Eenglish-tests.com

C2 Text 28 · Article · 768 words

Big Data and the Correlation Trap

A long-form feature on why enormous data sets do not explain themselves, from a failed flu tracker to spurious correlations, hidden third factors and arrows that point the wrong way.

In 2008 a much-quoted magazine essay announced 'the end of theory'. The argument was bold and, at the time, intoxicating: with enough data, we would no longer need to understand why things happen. Numbers would speak for themselves. Correlation would be enough. Scientists who laboured over hypotheses and controlled experiments were, by implication, craftsmen in an age of factories: admirable, perhaps, but obsolete.

The same year, a search engine launched a service that seemed to prove the point. By tracking how often people searched for terms related to influenza, it claimed it could estimate flu levels faster than official health agencies, which relied on reports from doctors. For a few seasons it performed impressively. Then, in the winter of 2012 to 2013, it badly overestimated the number of cases, at one point suggesting roughly twice as many as doctors actually recorded. Its algorithms had latched on to search terms that rose and fell with the season without having much to do with flu itself, and changes to the search engine's own features had shifted people's behaviour in ways nobody had modelled. The service was quietly retired.

'It's a cautionary tale, but not a scandal,' says Dr Priya Nandakumar, a statistician at a university in Edinburgh, who has spent fifteen years teaching students to distrust their own spreadsheets. 'The people involved were clever. The trouble is that size creates a particular kind of blindness. When you have millions of rows, everything looks robust. But a huge sample doesn't protect you from bias. It just lets you be wrong with great precision.'

The textbook example of the correlation trap is almost quaint. Across a summer, sales of ice cream rise and fall in step with the number of drownings. Nobody concludes that ice cream causes drowning; both are driven by a third factor, hot weather, which sends people both to the freezer and to the sea. Statisticians call such a hidden driver a confounding variable. The difficulty with big data is not that the principle has changed but that the confounders are rarely so obvious. In a data set with thousands of variables, the hidden third factor may be something nobody thought to measure, or something recorded under a dull name in column 4,812.

There is a second, subtler problem. If you test enough pairs of variables, some will appear to be connected by pure chance. By the conventional standard used in many sciences, roughly one comparison in twenty between genuinely unrelated things will look 'significant'. Run a few hundred thousand comparisons, as modern software makes effortless, and you are guaranteed thousands of spurious discoveries, the statistical equivalent of seeing faces in clouds. 'The computer doesn't get tired and it doesn't get suspicious,' Nandakumar says. 'It will cheerfully hand you a correlation between cheese consumption and the number of people who become lawyers, and if you want to believe it, it will even draw you a lovely chart.'

Owen Pryce, who leads a team of data scientists at a retail company in Leeds, sees the temptation from the inside. 'Commercially, you often don't care about cause,' he admits. 'If customers who buy garden furniture also tend to buy barbecue charcoal, I'll put the adverts together whether or not I can explain it.' The risk, he says, comes when a pattern that works for prediction is mistaken for a lever that can be pulled. 'We once noticed that customers who used our app were far more loyal. Someone suggested paying people to download it. But the app didn't create loyalty. Loyal customers downloaded the app.' The arrow of cause, in other words, had been read backwards.

None of this means that large data sets are useless; far from it. They have transformed weather forecasting, genetics and astronomy, and they are superb at generating questions. The mistake is to treat them as a substitute for the slower work of finding out why. Increasingly, researchers are borrowing tools designed precisely for that purpose: natural experiments, in which some accident of history or policy separates otherwise similar groups; careful causal models that force analysts to state their assumptions openly; and, where it is possible and ethical, the old-fashioned randomised trial, still the gold standard for telling cause from coincidence.

Perhaps the real lesson of the past fifteen years is that theory never ended; it simply went underground. Every choice about which data to collect, which variables to include and which patterns to take seriously already embodies assumptions about how the world works. 'Data never speak for themselves,' Nandakumar says. 'Somebody is always doing the talking. The question is whether they're willing to admit it.'

Words in colour are explained in the glossary below.

Glossary

WordMeaning
intoxicating/ɪnˈtɒksɪkeɪtɪŋ/very exciting, in a way that can stop you thinking clearly
laboured/ˈleɪbəd/worked very hard, often slowly and with difficulty
obsolete/ˈɒbsəliːt/no longer used or needed because something newer exists
latched on to/ˌlætʃt ˈɒn tuː/connected itself firmly to something and would not let go
retired/rɪˈtaɪəd/(of a product or service) stopped and taken out of use
cautionary tale/ˌkɔːʃənəri ˈteɪl/a story that warns people about a danger or mistake
robust/rəʊˈbʌst/strong and reliable; not easily shown to be wrong
quaint/kweɪnt/charming because it is old-fashioned or simple
confounding/kənˈfaʊndɪŋ/(in statistics) hidden and affecting two other things, so that they seem linked
significant/sɪɡˈnɪfɪkənt/(in statistics) unlikely to be the result of chance alone
spurious/ˈspjʊəriəs/false, although it seems real
lever/ˈliːvə/something you can use to make a change happen
natural experiments/ˌnætʃərəl ɪkˈsperɪmənts/real-life situations in which events, not scientists, divide people into groups that can be compared
gold standard/ˌɡəʊld ˈstændəd/the best and most trusted method, used to judge others
underground/ˈʌndəɡraʊnd/hidden, out of sight, but still active
embodies/ɪmˈbɒdiz/contains or expresses an idea

Big Data and the Correlation Trap: tests

Found this text useful? Share it with a friend or your class.