A nice idea, but crazy. It’ll never work. That’s how many people felt when Jimmy Wales and Larry Sanger decided to create an encyclopedia that anyone could edit.

Those naysayers couldn’t have been more wrong. Wikipedia now contains 25 million entries in 285 languages, and almost half a billion people consult it every month. “It’s an astonishing contribution to our culture,” says David Weinberger at Harvard University, who studies the effect of technology on ideas and knowledge. Wales and Sanger’s utopian vision – to gather the world’s knowledge in one place and make it available to everyone on the planet – hasn’t seemed crazy for years.

But perhaps it should. Over the past few years, Wikipedia supporters have begun to warn that the encyclopedia’s future is under threat. Great swathes of human knowledge remain absent from Wikipedia, yet a cabal of editors is accused of resisting attempts to broaden the encyclopedia’s content. What’s more, new editors are not arriving at the rate they once did.

Wikipedia’s guardians are embarking on a slew of initiatives to tackle its failings. Yet some people are now questioning whether the encyclopedia’s workings could be fundamentally flawed. “I think the problems are irrevocable,” says Heather Ford at the Oxford Internet Institute in the UK. And if they are right, what happens next to Wikipedia could affect more than just the fate of the website: Google and Microsoft, for example, are hatching plans that are intertwined with those of the encyclopedia. Some believe it is time for bold changes, but to work they will have to involve a serious rethink of the way knowledge is compiled. Fixing the world’s greatest encyclopedia is not going to be easy.

Explore the data in our interactive graphic: ““

It seems a long time since Wikipedia encountered its first potential stumbling block: ensuring accuracy. Around a decade ago, many doubted the site could ever rival a traditional encyclopedia. I had a personal role in challenging some of these perceptions. Together with a team of reporters at Nature, I compared the . The exercise produced a surprising endorsement for Wikipedia. Almost all of the errors identified were minor, and although our fact-checkers found an average of about four generally trivial errors per Wikipedia entry, they identified about three in the Encyclopaedia Britannica versions.

Since then, editors have boosted Wikipedia’s reliability by insisting that articles contain better citations, and ending the ability of unregistered editors to create new entries. Yet these measures created fresh controversies – which were brought into focus by the story of a fictional superhero in Kenya.

In 2010, a group of Kenyan Wikipedia users tried to create an entry for Makmende, a hero with a headband and 1980s styling, who is widely known in Kenya. His moniker initially emerged as slang, thanks to the Clint Eastwood line “Go ahead, make my day”. The term was used to mock; someone attempting an overambitious task would be asked if they thought they were Makmende. Later, a local band made a video featuring Makmende and jokes were phoned-in to Kenyan radio stations. Yet because, among other things, editors said the topic lacked notability.

This was odd, because Wikipedia is not exactly highbrow. A similarly frivolous meme, , has been the subject of a Wikipedia entry since 2006. The Makmende entry was eventually allowed, but only after the controversy over its creation attracted attention outside Kenya. To many observers, it seemed that the article had been rejected not because the topic was insignificant, but because it meant nothing to the editors who do most of the work on the encyclopedia.

The most active editors (), and this means the supposedly global project is skewed towards Western interests. According to a 2011 study by Mark Graham at the University of Oxford and colleagues, the snowy wastes of Antarctica have more articles dedicated to them than all but one of the countries in Africa. In fact, many African nations have fewer articles than the fictional realm of Middle Earth. These regions, notes Graham, are “virtual terra incognita”.

Then there is the gender issue. Around 90 per cent of Wikipedia editors are men, and it shows. In 2011, Shyong Lam of the University of Minnesota, Minneapolis, and colleagues measured the length of around 6000 Wikipedia articles about movies. This is a good proxy for quality, since longer articles tend to be more thorough. . Relatively threadbare coverage of When Harry Met Sally is not a big issue, but Lam believes the problem is a wider one. Female editors tend to work on topics like the arts and philosophy, but their lower participation may be making these articles shorter than others.

These biases are serious, but they do not seem insurmountable. If Wikipedia needs to become more diverse, why not recruit editors from the missing demographics? That brings us to a third complaint made about the encyclopedia: that it is unwelcoming to newcomers. Many new editors find themselves deterred by the thicket of rules that govern the encyclopedia, a problem made worse by the brusque attitude of some experienced editors. Newbies often find that their contributions are removed – a change known as a “revert” – with no explanation.

These problems are one reason why the number of active editors on the English Wikipedia has been slowly declining since 2007, as has the rate at which editors are promoted to “admin” status, which brings greater power to police the encyclopedia.

This trend raises the possibility that Wikipedia’s position as a leading source of knowledge might not be as secure as it appears. It would be difficult for another resource to supplant Wikipedia, but not impossible. Sites that allow users to ask and answer questions are one potential rival. In South Korea, for example, a question-and-answer service hosted by local search engine Naver is more popular than Wikipedia, says Andrew Lih at the University of Southern California in Los Angeles. And , a similar service, has attracted millions of dollars from investors since it launched in 2010.

Yet making Wikipedia more welcoming is more complicated than it may seem, says Samuel Klein, an active editor since 2004 who is now a trustee of the Wikimedia Foundation, a San Francisco-based organisation that supports the running of the encyclopedia. Klein points out that the English Wikipedia is under constant attack from spammers and vandals, and sometimes well-meaning newcomers get caught in the crossfire.

There are tentative signs that Wikipedia can address some of these problems. In February 2012, for instance, editors launched , an area of Wikipedia where newbies could ask questions that might, among a more experienced crowd of Wikipedians, seem stupid. Around 2000 questions were asked in the first year and those who visit the Teahouse go on to make nearly three times the number of edits as other newcomers, say the editors behind the project.

And in July, the site will launch a system that dispenses with the need to learn any . “It won’t solve all issues for new editors,” says Erik Möller, deputy director of the Wikimedia Foundation, “but it will solve the one issue we know that all editors are affected by.”

As for encouraging editors in developing nations, the foundation began in 2010 to try to persuade university lecturers in Brazil and Egypt to assign students tasks that involve creating or updating the encyclopedia. Möller also highlights an initiative called that provides free access to Wikipedia for smartphone users in less-developed countries. So far, the scheme covers 12 nations, including Kenya and Thailand, and around 20 more countries are set to join.

Will these efforts be enough to create a truly global encyclopedia? Some Wikipedians think not. The reason, they say, is that no amount of new editors will be able to plug Wikipedia’s gaps until the rules that govern the encyclopedia are changed. Right from its early days, for example, contributors have been barred from adding their own theories. The rules also state that articles should cite sources such as an academic journal or newspaper.

For subjects that have not been well studied, or countries with limited online news sources, these can be impossible standards to meet. “There is no way the publishing industry in India or South Africa is going to do a 400-year catch-up,” says Achal Prabhala, a Bangalore-based writer and an adviser to the Wikimedia Foundation.

So in 2011, Prabhala began promoting . As test cases, Prabhala and other Wikipedians from India and South Africa targeted a handful of local customs, including a religious ritual and a children’s game, that were not documented in Wikipedia. They interviewed participants, transcribed the conversations and cross-checked the accounts. The Wikipedia entries they created met with resistance almost immediately. “There was a bit of ganging up,” he says. “The bulk of the articles were removed.”

It was a scuffle that few outside Wikipedia noticed, but it captures what is perhaps the encyclopedia’s biggest dilemma. Almost everyone involved agrees that large chunks of the planet are poorly represented. Yet allowing editors to collect knowledge would “open up a can of worms”, says Weinberger. What happens, for example, when a disagreement occurs?

To assuage doubters, Prabhala has proposed a set of rules that would govern research by Wikipedians, from declaring their methods to emphasising discrepancies in their accounts. Yet that has not been enough. “I’m a huge fan of the idea,” says Lih. “But the project got stuck in the mud.” Prabhala hasn’t given up on the project, and says that his ideas have attracted interest from editors at Portuguese Wikipedia and elsewhere. But, he adds, there is little chance of them being adopted on the English version any time soon.

Ultimately, if Wikipedia’s coverage fails to expand, or the pace of updates slows, it is not only visitors to the encyclopedia’s website that will feel the effects. Over the past few years, Google and other search engines have begun to cull information from Wikipedia and present it to users alongside search results.

This service is the first manifestation of grander plans behind the scenes to assemble “knowledge graphs”: vast stores of data on hundreds of millions of people, places and other entities, as well as contextual information that describes how the entities are linked. The aim is to create a service that can answer your queries directly, or anticipate your needs, rather than giving you a hyperlink. A significant fraction of the companies’ knowledge graphs is taken directly from Wikipedia. So if the encyclopedia stutters, so will Google and Microsoft’s efforts.

Wikipedia was the place where the radical rethinking of the encyclopedia began. Yet its future may now be threatened by a strain of conservatism and parochialism that its early supporters frowned on in traditional publishing. If this were to lead to Wikipedia’s eventual decline as a cultural force, it would be an odd fate for a project once dubbed too crazy to work.

This article appeared in print under the headline “Wiki-opoly”

ONLINE social networks have helped spur protest movements and put politicians in office. But the power of these networks is built on easy access to platforms like Facebook and Twitter. Now researchers at Microsoft are about to test the power of social technologies in a tougher setting: India, where just a tenth of the population has internet access.

Competitors in the will leverage the power of mobile phones and old-fashioned, face-to-face contact to solve a puzzle posed by a team based at in Bangalore. At stake is a prize of 100,000 rupees, or almost US$2,000, about twice the country’s average annual income. The challenge will show us how people in developing countries use technology to collaborate across long distances and in time-critical situations.

The competition, beginning on 1 February, revolves around a fictional event and a set of five clues – the who, what, where, when and why – that describe the event. To obtain a clue, competitors have to call a number provided by Microsoft and hang up after a single ring. This practice, which tells the recipient they have missed a call, is commonly used in India as a means of prompting someone to ring back.

People in India try not to waste money on calls, says , a computer scientist at the University of California, Santa Cruz, who is working on the experiment. “That’s why the culture of the missed call is so important.”

A missed call to the Microsoft number triggers a text message to the caller’s mobile phone that contains one of the five clues. But the system will only send one message to each phone number. To collect the five clues needed to win, competitors will have to band together and share messages. The key to the competition is that some of the five clues will be sent out very rarely, so each team will need access to many phones to make enough calls to receive all the clues. Vaish estimates that a team will need to make about 500 calls to get the full set. The answers must be sent by text to Microsoft.

The researchers are leaving it up to competitors to decide how to collaborate. When a similar experiment was run in 2009 in the US, the winning team organised their efforts around a website and email. Neither service is widely available in India, so competitors may rely on social networks that revolve around work or places of study, says Vaish. He points out that people who interact with many individuals, such as rickshaw drivers, could do well.

“People who interact with many individuals, such as rickshaw drivers, could do well”

The challenge is also an attempt to see what motivates people to organise themselves. In the US-based experiment, the winning team offered to share the reward to encourage strangers to contribute. Vaish says it will be interesting to see whether a similar method prevails in India, or if entrants band together using existing social ties.

Language is another factor. India has several hundred languages, including about 20 that are spoken by at least a million people. Each text will be sent in English and in the major language served by the local mobile phone provider, and the Microsoft team will pay close attention to the impact that linguistic barriers have on team formation. Some of the techniques the competitors use could be deployed by city governments that want to involve many people in a search, perhaps for a fugitive or a missing person. “This will demonstrate the power of crowd,” says Vaish.

This article appeared in print under the headline “A social whodunnit”

IT’S one of the luxuries of the corporate elite: a personal assistant who takes your and turns it into a simple to-do list. Many people would love such help, if only they could afford it. But what if the cost was less than $2 a day? That’s the idea behind crowdsourced workers to manage email overload.

GmailValet, developed by and colleagues at Stanford University in California, works by connecting a Gmail account with , a crowd-labour web platform that draws upon a relatively skilled workforce. Users can deal with privacy fears by deploying filters that limit the access given to oDesk workers. All emails from family members can be excluded from the system, for example.

Once the workers are in, they examine new emails and, if appropriate, extract a task from the message, which appears in a to-do list that sits alongside the inbox on the GmailValet website, such as reminding the user to respond to a meeting request, for example. Users are encouraged to provide feedback on the tasks, so that the assistants can better understand their needs.

In initial tests, the assistants were paid the California minimum wage of $8 per hour. The researchers suggest that a single assistant could monitor dozens of inboxes simultaneously, though. If that proves to be the case, the service could end up costing each user as little as $1.80 per day.

The tests also revealed that users benefited from the to-do lists: the task-completion rate for those who worked with assistants was nearly 60 per cent, compared with less than 30 per cent for control participants, who had to create their own task lists. One user described the appearance of the tasks as “like magic”.

“This is an important step forward in enabling the crowd to work on private and sensitive information,” says Aniket Kittur, a computer scientist at Carnegie Mellon University in Pittsburgh, Pennsylvania. “It opens the door for the crowd helping us with our personal lives in ways we wouldn’t have imagined even a few years ago.”

“It opens the door for the crowd to help us in ways we wouldn’t have imagined even a few years ago”

The work will be presented next month at the in San Antonio, Texas. The system is available to try out for free at .

Editorial: “Unmasking the cyber saboteurs, whoever they are

EVERY year a computer worm emerges to stalk the internet, each one seemingly bigger and badder than the last (see diagram). Although they seem to come from nowhere, every new bit of malware has a history. Sussing out the family resemblances could generate a faster response to future threats.

“Our vision is to have a database of the world’s malware, which people can use to share insights,” says Josh Saxe from Invincea labs in Fairfax, Virginia. His firm’s scheme is based on a novel method for classifying malware, the programs hackers use to steal passwords, send spam and carry out other nefarious activities.

“Our vision is to have a database of the world’s malware, which focuses on how it behaves”

Malware is produced at such an astonishing rate that security experts already have automated systems for classifying new strains. But many of their plans are based on analyses of malware code, which hackers can often disguise. The new approach focuses instead on the behaviour of the malware itself.

Saxe and colleagues tested their ideas on just over 100,000 malware samples collected between February 2011 and June 2012. The team ran each piece of malware and logged the communication between the software and the machine it was running on. This communication is made up of “calls”, such as requests to read the contents of a particular file. Individual strains often produce tens of thousands of such calls.

After watching the behaviour of many strains, Saxe and colleagues were able to break the communication data into blocks containing specific sequences of calls that occurred repeatedly across different samples. These blocks are a consequence of malware authors reusing code from older strains. The team used these blocks to classify the strains and to group them into what Saxe calls “malware families”. He presented his work last month at the in Lake Tahoe, Nevada.

Analysts will be able to use this catalogue of malware families to share information about new threats. Such a tool could be extremely useful, because much of the collaboration that takes place in the security community is ad hoc. Groups of researchers sometimes band together to tackle new threats, but detailed analyses of emerging strains tends to happen in parallel and independently.

The Invincea system offers an alternative. Analysts can attach notes to the blocks of sequence calls and, if a new strain produces the same block, those notes pop up. This should allow others to get to grips with the strain more rapidly.

Analysts should also be able to visualise groups of malware families, which could help determine the ancestors and authors of new strains, says Gunter Ollmann, chief technology officer at IOActive in Seattle. But he cautions that such sharing may be hampered by technical differences between analysts’ set-ups.

The Invincea project is one of several funded under the Cyber Genome Program, run by the US Defense Advanced Research Projects Agency. The results, many of which have not been made public, will be used to secure computer networks run by the Department of Defense.

“We’re trying to allow people to make more intelligent choices about what to analyse to avoid repetition,” says Saxe. “We’re trying to multiply the effectiveness of analysts.”

WHERE do new words come from? On Twitter at least, they often begin life in cities with large African American populations before spreading more widely, according to a study of the language used on the social network.

at the Georgia Institute of Technology in Atlanta and colleagues examined 30 million tweets sent from US locations between December 2009 and May 2011. Several new terms spread during this period, including “bruh”, an alternative spelling of “bro” or “brother”, which first arose in a few south-east cities before eventually hopping to parts of California. Residents of Cleveland, Ohio, were the first to use “ctfu”, an abbreviation of “cracking the fuck up”, usage that has since spread into Pennsylvania ().

After collecting the data, the team built a mathematical model that captures the large-scale flow of new words between cities. The model revealed that cities with big African American populations tend to lead the way in linguistic innovation. The team is still working on a more detailed analysis and says it is too early to say which cities are the most influential.

Researchers have tracked the diffusion of words like “cool” and “uptight” from black communities to mainstream use in the past. “We have thousands of examples,” says Eisenstein. Their data cannot shed light on why the flow is in this direction, but he notes that language is just one cultural area in which traditions have spread outwards from African American communities.

The team also found that cities that are economically and ethnically similar – rather than geographically close to one another – are more likely to share new words. “Their results indicate that birds of a feather tweet together,” says , a linguist at the University of Groningen in the Netherlands.

“Cities that are ethnically and economically similar are more likely to share new words on Twitter”

Eisenstein says he is looking into whether neologisms now spread more rapidly because of Twitter and other social networks. He is also interested in exploring whether social media is accelerating the evolution of language more generally, something that could be done by analysing everything from blog posts to Facebook entries. It’s not like the old days, he says, when the spread of a word relied on people travelling to new areas.