The rapid progress in artificial intelligence in recent months is partly due to training on vast data sets of text and images, scraped for free from the internet. Although automated web scraping by search engines has been accepted by website owners for decades, the economic shift being brought about by AI has triggered a rethink.
At a basic level, search engines offer an exchange to website owners: let us scrape to compile the information and serve useful results, and we will send traffic your way in return. But scraping for AI changes this equation, hoovering up text or images while providing nothing for website owners, who have to pay the server costs of being scraped.
As a result, a war on scraping has begun. Twitter owner Elon Musk has for using the platform鈥檚 content to train AI models, while Reddit has suggested companies for doing the same. Getty Images is 聽for allegedly using 12 million of its images to train its own image-generating AI, while record label Universal Music Group has told streaming platforms Spotify and Apple to to train AI models.
Advertisement
Even small websites are taking a stand. The cycling forum has a little over 68,000 users and its owner David Kitchen says he recently took the decision to ban access to anything that looked like a scraping bot, as they grew to account for an estimated 30 per cent of traffic 鈥� a boom he puts down to the recent explosion of AI.
鈥淵ou end up paying more money for infrastructure,鈥� he says. 鈥淲hen they derive value and you鈥檙e not getting value back, you think: 鈥榃hy am I paying my costs to do things for you?鈥� So I鈥檝e literally blocked every single non-human, non-browser-based activity.鈥澛�
Website owners have long had a way to defend against the bots, by placing a file called robots.txt on their server setting out what they can access, but Neil Brown at points out that this is easily ignored. 鈥淩obots.txt is a bit like sticking a note on your front door saying 鈥榯he door鈥檚 open, please don鈥檛 burgle me鈥�. Nothing more than that,鈥� he says. 鈥淭here鈥檚 no law that says 鈥楾hou must follow robots.txt鈥�.鈥�
, co-founder of the website , has experienced the limitations of this approach. After bots hit his website with thousands of requests in a short period, he , , and requested that it only scrape sites that had given explicit permission. In messages on GitHub, Beaumont, who works for Google, said that to do so would be 鈥渦nethical鈥� as it would deprive the world of better AI models. 鈥淚f you don鈥檛 wish for people to view images from your website, the best way is to turn it off,鈥� he said. Beaumont didn鈥檛 respond to a request for interview.
So what alternatives do website owners have? It isn鈥檛 clear that scraping for AI data is actually illegal in any country, but if you wanted to sue for damages, there would be numerous hurdles to overcome, says Brown. First, a website owner would have to find out who was scraping them, which could involve going to court to force an internet service provider to reveal details of its customers. Then, the parties and web servers involved may all be in different countries with different laws and procedures.
Finally, the biggest names in AI are some of the largest companies in the world, making any legal battle an unfair fight. 鈥淵ou have some massive companies with huge war chests who clearly have a vested interest in being able to do this kind of scraping,鈥� says Brown. 鈥淵ou might have a claim, but with the best will in the world, you鈥檙e going up against some of the best-funded legal teams in the world.鈥�
While companies may not face significant challenges over harvesting copyrighted material in order to train AI, Europe鈥檚 strong data protection laws may at least limit what personal information is collected. Italian regulators recently banned ChatGPT temporarily because of data protection concerns and the UK鈥檚 Information Commissioner鈥檚 Office has . But this would still leave the vast majority of online content up for grabs.
Ultimately, Eden believes that bots scraping content for AI models breaks the social contract of the internet, under which tools like search engines would point users back to the original source. 鈥淵ou can鈥檛 build something on stolen content and then resell it without compensating the original owners,鈥� he says. 鈥淚t becomes a purely exploitative practice.鈥�
