Feed the bots

The article discusses the prevalence of bots, particularly scrapers used by AI companies to train language models. These bots are aggressive, ignoring traditional blocking methods and consuming server resources by requesting old and obscure pages. Traditional tactics like IP bans and rate limits are ineffective due to the bots' ability to switch IPs and addresses. The author explores various strategies to combat the bots, including serving them dynamically generated content, but finds that even sending them "gzip bombs" or 404 errors doesn't deter their activity. Ultimately, the author concludes that feeding the bots nonsensical content is the most cost-effective approach to managing their impact on server resources.