Differences

This shows you the differences between two versions of the page.

--- folkzone:networking:bot_traffic [2026/09/23 14:52] – created - external edit 127.0.0.1
+++ folkzone:networking:bot_traffic [2026/10/23 16:02] (current) – external edit 127.0.0.1
@@ Line 8: / Line 8: @@
   * **51% of all web traffic** is now bots
   * Good bots (search crawlers, etc.): 14%
-  * **Bad bots: 37%** — six consecutive years of growth
+  * **Bad bots: 37%**: six consecutive years of growth
-Bad bots: data scraping, fraud, credential stuffing, server overwhelming. Growth driven by genAI tools making bot deployment faster, cheaper, and accessible to people with minimal technical skill.
+Bad bots: data scraping, fraud, credential stuffing, overloading servers. Growth driven by genAI tools making bot deployment faster, cheaper, and accessible to people with minimal technical skill.
 ===== AI Crawlers =====
@@ Line 16: / Line 16: @@
 AI training crawlers operate at an extractive crawl-to-referral ratio:
-  * Anthropic's Claude crawler peaked at approximately **500,000:1** — 500,000 pages crawled per ~1 visitor sent back
+  * Anthropic's Claude crawler peaked at approximately **500,000:1**: 500,000 pages crawled per ~1 visitor sent back
   * **13.26% of AI bot requests actively ignored robots.txt** directives in Q2 2025
@@ Line 23: / Line 23: @@
 ===== Why robots.txt Is Not Enough =====
-The worst offenders do not act in good faith to honour ''robots.txt''. Maintaining a blocklist is an uphill arms race — new crawlers appear faster than blocks can be added. For independent publishers, this is not a viable solution.
+The worst offenders do not act in good faith to honour ''robots.txt''. A blocklist cannot keep up: new crawlers appear faster than blocks can be added. For independent publishers, this is not a viable solution.
 ===== Real Mitigations =====
-  * **DDoS protection / CDN** — absorbs volumetric bot traffic before it reaches your origin server. See [[folkzone:networking:deflect|Deflect]] (for independent/civil society sites) and [[folkzone:networking:cdn_consolidation|CDN options]]
+  * **DDoS protection / CDN**: absorbs volumetric bot traffic before it reaches your origin server. See [[folkzone:networking:deflect|Deflect]] (for independent/civil society sites) and [[folkzone:networking:cdn_consolidation|CDN options]]
-  * **Rate limiting** — limit requests per IP per time window at the reverse proxy layer (Caddy, nginx)
+  * **Rate limiting**: limit requests per IP per time window at the reverse proxy layer (Caddy, nginx)
-  * **Honeypot paths** — serve fake endpoints that only bots visit; block IPs that hit them
+  * **Honeypot paths**: serve fake endpoints that only bots visit; block IPs that hit them
-  * **Netlify bandwidth** — if on a free tier, bot traffic can exhaust your bandwidth allocation before human visitors are served. 106.8 GB in a single month is not unusual for a site with any visibility
+  * **Netlify bandwidth**: if on a free tier, bot traffic can exhaust your bandwidth allocation before human visitors are served. 106.8 GB in a single month is not unusual for a site with any visibility
 ===== Intentional Apathy =====
-One position: don't block AI crawlers. The [[indieweb:good_web|Good Web]] is built on openness; attempting to restrict crawlers selectively is an unwinnable arms race, and legitimate archiving bots (Wayback Machine, search engines) use the same mechanisms. See [[indieweb:principles|IndieWeb Principles]] on the balance between openness and protection.
+One position: don't block AI crawlers. The [[indieweb:good_web|Good Web]] is built on openness; selectively restricting crawlers cannot keep pace, and legitimate archiving bots (Wayback Machine, search engines) use the same mechanisms. See [[indieweb:principles|IndieWeb Principles]] on the balance between openness and protection.
 ===== See Also =====
@@ Line 41: / Line 41: @@
   * [[folkzone:networking:deflect|Deflect]]
   * [[folkzone:networking:cdn_consolidation|CDN Consolidation]]
-  * [[folkzone:services:cloudflared|Cloudflare Tunnel]]
+  * [[terminal:tools:ssh_via_cloudflare|Cloudflare Tunnel]]
-  * [[folkzone:services:caddy|Caddy — Rate Limiting]]
+  * [[folkzone:services:caddy|Caddy: Rate Limiting]]
   * [[start|Return to wiki home]]
   * [[folkzone:start|Return to folkzone]]