What robots.txt can and cannot control for AI crawlers

robots.txt is only a crawling permission declaration, not access control; it can only constrain which paths compliant crawlers may fetch, and cannot control non-compliant crawlers, content that has already leaked, or acquisition methods that do not go through crawlers. Following RFC 9309, the article explains rules such as User-agent, Allow, Disallow, and the 500 KiB read limit, and provides robots.txt syntax for blocking GPTBot, CCBot, ClaudeBot, and Bytespider by UA, as well as methods for using nginx to apply differentiated rate limiting by UA, using logs to count request volumes per crawler, and using curl to verify whether rate limiting takes effect. The author emphasizes that robots.txt expresses intent, server-side rate limiting controls costs, logs reveal the facts, and licensing agreements address legal issues—each of the four handles a distinct part.