Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries

ronjor · Mar 26, 2025

Benj Edwards – Mar 25, 2025

AI bots hungry for data are taking down FOSS sites by accident, but humans are fighting back.
Click to expand...

Software developer Xe Iaso reached a breaking point earlier this year when aggressive AI crawler traffic from Amazon overwhelmed their Git repository service, repeatedly causing instability and downtime
Click to expand...

.

Despite configuring standard defensive measures—adjusting robots.txt, blocking known crawler user-agents, and filtering suspicious traffic—Iaso found that AI crawlers continued evading all attempts to stop them, spoofing user-agents and cycling through residential IP addresses as proxies.
Click to expand...

reasonablePrivacy · Apr 4, 2025

Wikipedia faces increased infrastructure costs due to AI crawlers as well

AI bots strain Wikimedia as bandwidth surges 50%
https://arstechnica.com/information-technology/2025/04/ai-bots-strain-wikimedia-as-bandwidth-surges-50/

Log in or Sign up

Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries

ronjor Global Moderator

reasonablePrivacy Registered Member

Log in or Sign up

Open Source devs say AI crawlers dominate traffic, forcing blocks on entire countries

ronjor Global Moderator

reasonablePrivacy Registered Member

Useful Searches