Editorial: Don’t fear the tweeter: your data trail is doing good
Your online traces are helping fuel a revolution in the understanding of human behaviour – one that’s revealing the mathematical laws of our lives
EVERY move you make, every twitter feed you update, somebody is watching you. You may not think twice about it, but if you use a social networking site, a cellphone or the internet regularly, you are leaving behind a clear digital trail that describes your behaviour, travel patterns, likes and dislikes, divulges who your friends are and reveals your mood and your opinions. In short, it tells the world an awful lot about you.
Advertisement
Now, as any researcher will tell you, good data is gold dust. Its absence leaves theories in the realm of speculation, and worse, poor data can lead you down blind alleys. Physics was the first science to be transformed by accurate information, first with telescopes that revealed the heavens and culminating in massive modern-day experiments like the Large Hadron Collider at CERN, near Geneva, Switzerland. Biology was next, with genome sequencing throwing up so much of the stuff that genetics has turned partly into an information science.
Now the study of human behaviour is heading the same way. Social scientists have long had to rely on crude questionnaires or interviews to gather data to test their theories; methods marred by reporting bias and small survey sizes. For decades, the field has been looked down upon by some as a poor cousin to the hard sciences. The digital age is changing all that – practically overnight, the study of human behaviour and social interactions has switched, from having virtually no hard data to drowning in the stuff. As a result, an entirely different approach to social science has emerged, and studies based on it are appearing with increasing frequency. The impact has been remarkable.
“The data revolution is here for social science,” says Albert-László Barabási of Northeastern University in Boston. “For the first time, scientists have a chance to study what humans do in real time and in an objective way. It’s going to fundamentally change all fields of science that deal with humans.”
“The data revolution is here for social science. For the first time we can study what humans do in real time”
It is becoming possible to tackle fundamental problems previous generations have thought largely untouchable. As with every other data-rich science, Barabási and his ilk ultimately hope to discover mathematical laws that describe human behaviour, and which could be used to predict what people will do.
Sociologists have been hunting for such laws about human interactions and social networks for decades, says Duncan Watts of Yahoo Research in New York, “but the far-reaching implications of their theories have been effectively impossible to test. The measurement technology simply didn’t exist”. That’s changing.
Watts was among the first to realise the potential of the digital trail we leave behind. In 2006, with his colleague , now at Princeton University, he designed a web-based experiment to examine how much social influence determines the popularity of music. When a new song goes straight to number 1, it’s hard to know if its success has come from the song’s inherent appeal, or instead from the herd-like behaviour of many people buying songs they think are already popular. The music industry has had little success in predicting which songs will do well and which won’t, suggesting that a lot might be down to chance.
To examine what made some songs more successful than others, Watts and Salganik created a project called Music Lab. It featured a website where more than 14,000 people listened to any of 48 songs by relatively unknown bands, rated them, and downloaded them if they wanted. These options provided a measure of quality (the average rating given) and popularity (the number or downloads). Crucially, the duo were also able to control whether listeners could see how many times other people had downloaded any particular song, or instead had to rely solely on their own judgement. In this way, they could effectively compare outcomes with the power of social influence turned on or off. They also grouped the socially influenced participants into eight independent “worlds” so that they could explore how the outcomes – the popularity rankings of the various tunes, based on downloads – might change if the tape of history could be rewound and run again.
The results strongly support the idea that human influence has a huge effect in making some songs more popular than others. This factor also makes it much harder to predict what will happen, and which songs will do well. The worlds in which social influence was operating had much higher inequality – with popular songs going up and unpopular songs going down to an even greater extent than in the worlds lacking social influence. With social influence turned on, song popularities fluctuated wildly between one world to the next. So, like it or not, it seems like many of us follow the herd.
Watts and Salganik concluded that so-called experts fail to predict successes not because they are incompetent or misinformed, but because social influences multiply chance effects into lasting differences. Accidents determine the song at the top of the chart as much as true quality.
Though quality does count, Salganik points out. The songs rated as the best rarely did poorly, and those rated as the worst rarely did well, but any other result was possible.
These kinds of experiments are making routine the types of experimental studies which were once thought impossible, says Salganik. “With the vast increase in computing power and the almost limitless pool of participants now available via the internet, we can conduct laboratory-style experiments involving thousands, or even millions, of participants,” he says.
Indeed, Jukka-Pekka Onnela and Felix Reed-Tsochas at the University of Oxford’s Saïd Business School are now using Facebook and its 400-million-plus users as a living laboratory to examine how ideas and behaviours spread through human groups.
Watts and Salganik showed that when it comes to music preference, we behave like sheep. Social scientists have long wondered whether other social transformations – including everything from the popularity of a politician to a change in behaviour to mitigate climate change – arise through independent, individual choice, as many people simultaneously come to similar decisions, or instead through influence, as people copy others’ behaviours.
Onnela and Reed-Tsochas realised that analogous changes take place in Facebook, on which people share their profiles with their online friends. Facebook users can also choose to install applications – software components that personalise their Facebook page. If one person adopts an app, their friends are automatically notified, and they can also see the apps their friends are using. Facebook users also have access to a list of popular applications, akin to a best-sellers list.
So far, so high-street bookstore, but there’s one huge difference: the data stored on Facebook makes it possible to analyse the growth in popularity of individual applications in unprecedented detail. Onnela and Reed-Tsochas analysed the popularity of several thousand applications in 2007, shortly after those apps were introduced, and then studied how other users adopted them over time. They looked to see if the sequence of adoptions for each app followed an essentially random pattern – indicating that each “adoption event” was independent of other previous adoptions – or whether previous adoptions by a participant’s friends influenced the likelihood of their subsequent adoption of an app.
Explosive growth
The results showed that both independent thinking and copying behaviour play a role, reinforcing conclusions reached by conventional survey methods. However, the study also indicated there are two very different processes in action. On the one hand, their analysis shows, at first, when a new app appears it is adopted by users independently of their friends’ opinion. But if the popularity of an app crosses a threshold, its very popularity then seems to draw many people to adopt it, and its growth can become explosive. Just as Watts and Salganik found in their Music Lab experiment.
“We found very distinct regimes in which individual or collective behaviour dominates. The change from one to the other is a sharp on/off process,” says Reed-Tsochas. They don’t yet know whether tipping points of this kind might influence real-world processes beyond the web, such as shifts in political opinions or the popularity of books. “It’s certainly possible,” says Reed-Tsochas, “but we’ll need to wait for equally good data in those areas to find out.”
Some say the raw information for analysis of real-world behaviour is already there in the burgeoning online social networks, and have even shown how it can be used to predict social outcomes. For example, one of the most popular techniques for predicting anything from presidential elections to the box-office success of new movies is by using artificial markets. The () enables movie fans to buy and sell virtual shares in celebrities and in forthcoming or recently released films. This virtual market, which operates with a virtual currency called Hollywood dollars, incorporates the views of millions of people into a stock rating for each film, reflecting the aggregate view on its popularity, or likely popularity. “This is currently the gold standard in the industry for predicting likely box office receipts,” says Bernardo Huberman at Hewlett Packard Laboratories in Palo Alto, California.
Huberman and his colleague Sitarum Asur wondered if it might be possible to do even better by exploiting the enormous volume of opinion expressed through social media such as Facebook and Twitter. Opinions voiced in these media, they reasoned, should have strong predictive power because they actually play a role in determining which films do well. “What gets discussed through these media often ends up setting trends,” says Huberman.
In an attempt to mine these opinions, they studied the chatter on the microblogging site Twitter. They started from the supposition that movies that get talked about a lot – that generate a lot of buzz – should end up being more popular. To measure the buzz for each film, they looked at the rate at which it generated tweets immediately following its release. They used this as a predictor of the ultimate film sales.
The results show that the rate at which movie tweets were generated can provide accurate predictions of box-office revenue, more accurate even than the Hollywood Stock Exchange. In the end, predicting successful movies may only be of interest to film companies and investors. But Asur and Huberman reckon this is just the beginning, and that their technique should be able to predict social outcomes of many kinds. “When properly tapped, social media express a collective wisdom which can yield an extremely powerful and accurate indicator of future outcomes,” says Asur.
Huberman says such analyses could soon help predict many other events, such as election outcomes, or quickly gauge public reactions to major events, just as long as we have evidence reflecting peoples’ views on the relevant issue. “Twitter and texting in general were influential in the election of Barack Obama and some businesses are already analysing these kinds of data to assess the likely success of their products,” says Huberman.
The ocean of digital information about us isn’t limited to opinions. Though it’s still controversial, and difficult to get hold of, some teams are accessing much more personal details. For example, Barabási and his colleagues at Northeastern University used cellphone data to analyse human movements – how we move about over hours, days, weeks and months by walking, driving and public transport in all its forms. Detailed data on the scale now available never existed before cellphones became commonplace. Now millions of people carry a de facto tracking device with them all day that automatically logs their every move.
You’re so predictable
The dataset the team used covered the movements of about 50,000 people over three months. Surprisingly, the team found that, despite our myriad individual differences and diverse daily routines, the overall statistics of our movements follow a mathematical pattern – and we’re far more predictable than you might think. What’s more, they found that analysis of past data on movements can be used to predict where an individual will be – to within 1 kilometre of a cellphone tower – even during the more variable parts of their day, with an accuracy of over 90 per cent (Science, vol 327, p 1018). “We found the same high level of predictability across all users,” says Barabási. That’s perhaps not so surprising as for most of the day, most of our movements are pretty routine, moving from home to work and back, however, this ability to predict your location holds true even for those people who move around more than just the typical home-work-home commute.
This study builds on earlier work in which Barabási and colleagues used cellphone data to explore the patterns of human movements (Nature, vol 453, p 779). There they found that individuals generally travel lots of relatively small distances, but occasionally take long excursions that move us to very different territory. The precise details of the statistics of such movements follow a mathematical pattern – known as a Levy flight – which turns out to be closely linked to the ways animals such as deer, bumble bees and birds forage for food. Mathematically speaking, our movements turn out to be strikingly like those of other organisms. So we’re not so special, at least in this regard.
“There are a lot of details that make us different,” says Barabási, who has found convincing evidence that most of our actions are driven by laws, patterns and mechanisms that rival the reproducibility and predictive power of those encountered in the natural sciences.
It’s the discovery of underlying patterns of this kind that has excited so many scientists. Given the undeniable complexity of individual human beings, it’s not as if social science is going to become like physics, grounded in eternal and general laws, but access to data on human events makes it possible to identify the patterns that do exist and these can be useful for demystifying the social world.
However, as with some developments in physics and biology, the social data explosion also brings with it new risks, says Barabási. “Anyone involved in this kind of research increasingly faces a dilemma – how do we avoid contributing to the creation of a surveillance state?”
Such worries are, perhaps, another sign that social science is finally coming of age. Just as the discovery of nuclear fission raised moral dilemmas for physicists, and genetic modification is now doing for biologists, so the ability to predict human behaviour is presenting new quandaries for social scientists. As ever, with great power, comes great responsibility.
