Over the past several months, developers have reported increasingly aggressive traffic from bots operated by AI companies, which are scouring the web for training data. Unlike traditional search engine crawlers, many of these bots ignore protocols like robots.txt, disguise their identities, and hammer servers with tens of thousands of requests. What’s being described by some as “scraping” is, in reality, overwhelming some FOSS infrastructure in ways that feel like a constant, slow-motion DDoS attack.

A Growing Crisis

In March 2025, Ars Technica reported that AI crawler traffic has reached a crisis point. Cloudflare, which protects a large swath of the internet, revealed that these bots now generate over 50 billion requests every day—nearly 1% of all global web traffic. This kind of volume isn’t just an annoyance. For small projects hosted on limited infrastructure, it’s unsustainable.

Developer Xe Iaso was one of the first to sound the alarm. After their Git server became unstable due to Amazon’s AI crawler traffic, they built Anubis, a proof-of-work challenge system that requires visitors to prove they’re not bots. When GNOME developers implemented Anubis, it immediately blocked 97% of incoming traffic—highlighting the staggering volume of unauthorized bot activity.

Others are turning to more aggressive defenses. One anonymous developer released Nepenthes, a “malicious” tarpit that traps bots in endless loops of junk data, poisoning their training sets. Cloudflare followed suit with a more measured solution: the AI Labyrinth, designed to confuse scrapers without affecting legitimate users.

The Ethics of AI Scraping

At the heart of this issue lies a bigger question: What does ethical AI scraping look like?

AI companies often argue that public data is fair game. But just because something is technically accessible doesn’t mean it’s freely available for mass harvesting—especially not at the expense of the people and communities who built and maintain it.

Ethical scraping should be based on a few simple principles:

1. Respect established boundaries: Obey robots.txt, user-agent rules, and rate limits.

2. Seek consent: Just because data is public doesn’t mean it’s free for the taking. Consent matters.

3. Give credit and compensation: Open-source developers rarely get paid for their work. If you’re using their content to train a for-profit model, contribute something back.

4. Don’t harm the infrastructure: Crawlers should never disrupt the very platforms they’re mining.

If these standards aren’t followed, the damage could be long-lasting. Already, some FOSS communities have been forced to geo-block entire countries, throttle traffic, or close off public resources—steps that run counter to the spirit of open source.

Toward a Sustainable Solution

As pressure builds, some are calling for legal and policy solutions. One commentator even proposed a “CAN-BOTS Act,” likening today’s bot problem to the email spam crisis of the early 2000s that led to the CAN-SPAM Act. While partly tongue-in-cheek, the idea captures a growing sentiment: self-regulation isn’t enough.

In the meantime, projects like ai.robots.txt are helping developers implement stronger crawler protections. Others are experimenting with proof-of-work systems, honeypots, and clever traffic filtering. But these are band-aids. Without a coordinated response from AI developers, regulators, and the FOSS community, the foundation of the internet may begin to fracture.

Conclusion: FOSS Built the Internet—Don’t Let AI Break It

The FOSS ecosystem gave us Linux, Firefox, Python, PostgreSQL, and thousands of tools that power nearly every app and website we use today. These resources are free—not because they’re worthless, but because a community of people chose to make them so.

That community is now being quietly exploited at scale.

If we want AI to be ethical, sustainable, and truly innovative, it must start by respecting the commons it feeds on. That means asking before taking, giving back where possible, and never forgetting the human effort behind the code.

The internet was built on openness. But if AI keeps scraping without care, the doors may start to close.