
From writing encyclopedia entries to designing furniture, a new kind of crowdsourcing could change the way we work
WHEN it appeared on Mechanical Turk, an outsourcing website run by Amazon, Boris Smus’s request looked unremarkable. He wanted people to search the web and gather a few facts about New York City. Workers snapped up the tasks and answers rolled in within minutes. Quick to complete and requiring no specialist skills, it was typical of work available on the site, which usually pays a few cents per task.
There was, however, something unusual about the job. It was not posted by Smus, a researcher at Carnegie Mellon University in Pittsburgh, Pennsylvania, but by software he had created. The same software also posted editing and writing jobs. As the results came in, it farmed the text back out for further checking and editing. The result was an encyclopedia entry on New York City, produced by humans but managed by machine.
Advertisement
This idea – a human assembly line overseen by a silicon supervisor – may change the way we work. Farming out minor tasks such as image-labelling to the crowd is now commonplace, and it looks like far more complex operations are on the way.
For example, over the past few years a handful of companies have built their businesses by providing translation and transcription services that use crowd labour. Blog posts, press releases and even furniture designs could soon be created in the same way. If they are successful, many jobs normally associated with skilled workers could be transferred to a cheaper, quicker – and untrained – online workforce.
“We’re looking for a way of structuring work so it can be done by the crowd,” says computer scientist Aniket Kittur, who leads the research team at Carnegie Mellon. “There are whole domains we haven’t even tapped.”
The idea certainly seems to work well for transcription and translation. CastingWords, based in Seattle, uses software to manage a trained online workforce tasked with transcribing audio interviews and podcasts. The software splits the audio into 5-minute chunks and distributes each to a worker. Completed transcriptions are automatically routed to other workers for quality checks. The software then compiles the results and gives the transcription to one person for a final check before it is returned to the customer, all without the intervention of a human manager.
“The software returns the results to the customer without the intervention of a human manager”
CloudCrowd, a San Francisco-based company, says it raised $5 million in funding last year to use the same model to perform translation services. CEO Alex Edelstein says he plans to launch a service for writing press releases and blog posts later this year. “The whole company is based on the premise that work can be done on virtual assembly lines,” he says.
Kittur and his colleagues are trying to develop those assembly lines. Their encyclopedia project started with requests for workers to create a list of topics that an entry for New York City should cover, such as attractions and history. Then another set of workers looked for facts relating to a single topic drawn from this list. In a third range of tasks, workers turned the sets of facts into individual paragraphs. The paragraphs were then combined to produce the final entry.
Kittur’s team asked another group of workers to rate the result against an entry written by a single worker, as well as one taken from the Simple English Wikipedia, a version of Wikipedia that uses simpler grammar and fewer words than the full version of the online encyclopedia. They rated it more highly than the individual version and on a par with the Wikipedia entry. The team will present their work in May at the Conference on Human Factors in Computing in Vancouver, Canada.
Jeffrey Nickerson of the Stevens Institute of Technology in Hoboken, New Jersey, is trying to break down complex tasks of a different kind. He asked Mechanical Turk workers to come up with designs for a children’s chair. Other workers rated the drawings, and those most highly rated went into a second round in which workers attempted to combine the best elements from pairs of the drawings. Nickerson repeated this process of rating and recombination, which he calls a “human genetic algorithm”, twice more. He too will present his work at the Vancouver conference in May.
From a practical standpoint, these complex crowd-sourced processes are still in their infancy. Researchers are interested in trying to design graphics and write advertising copy in this way, but it’s still not clear which complex tasks can be broken down into ones that can be performed by an unskilled labour force. And if the method continues to gain in popularity, questions of whether Mechanical Turk workers are paid fairly – and whether they should be protected by labour laws – will have to be answered.
The results from Nickerson’s experiment are not good enough to be made into chairs, says John Nastasi, an architect at Stevens who worked with Nickerson on the project. But some of the designs, like a “whimsical” chair with feet that looked like human feet, show that the approach has potential, he says.
Jim Giles and MacGregor Campbell, reporters in èƵ‘s San Francisco bureau, are currently running an experiment in crowdsourced journalism. Find out more at