Wikipedia is one of the greatest knowledge resources ever assembled, containing crowdsourced contributions from millions of humans worldwide – and it faces a growing threat from artificial intelligence developers.
The non-profit Wikimedia Foundation, which operates Wikipedia, says since January 2024 it has seen a in network traffic requesting image and video downloads from its catalogue. That surge mostly comes from automated data scraper programs, which developers use to collect training data for their AI models. This unprecedented increase in internet traffic means Wikimedia must burn through more money to serve up Wikipedia pages and other content from its rented data centres.
“There’s been various reports about other content sites that are suffering in the same way, but when someone as visible and as critically important as Wikimedia goes public with such a statement, people are paying attention,” says at King’s College London. “The issue they are discussing is very, very worrying, and I’m speaking as a scientist who has been working in AI and in responsible AI [research] for more than 15 years.”
Advertisement
The foundation says 65 per cent of its most expensive internet traffic comes from data scraping bots. They often request less popular articles, and these queries must travel all the way to a central data centre, instead of being able to use the caches of more popular articles stored in local data centres.
“This high usage is also causing constant disruption for our Site Reliability team, who has to block overwhelming traffic from such crawlers before it causes issues for our readers,” the Wikimedia Foundation wrote in its blog post.
, director of product at the Wikimedia Foundation, told żěè¶ĚĘÓƵ the organisation is “looking into ways to manage bot traffic” and is also asking commercial users such as AI developers “to directly support the sustainability of the Wikimedia projects”.
“One thing that is often overlooked for public content or openly licensed content is that the content is free to access but the infrastructure is not free to run,” says Simperl. “The infrastructure costs a lot of money, and those resources and the people required to run it need to come from somewhere.”
The Wikimedia Foundation has published for identifying the developers behind data scraping bots, with the goal of reducing automated traffic from scrapers by 30 per cent in terms of bandwidth.
Wikimedia also faces direct competition from AI chatbots that can respond to questions on various topics – even if the AI responses aren’t always factual. Although Wikimedia websites have not seen drops in traffic directly attributable to recent AI developments, Müller expressed concern about how AI services that “use Wikimedia content to provide quick machine-generated summaries and responses” do not typically provide proper citations and “block pathways for people to access the original sources of information”.
have even found signs that AI-generated content may be creeping into Wikipedia entries. But the Wikimedia Foundation is not necessarily opposed to this technology. It already uses AI tools to help human editors detect vandalism of Wikipedia sites, predict article quality, measure article readability and suggest edits.
