{"id":238,"date":"2026-10-23T22:54:31","date_gmt":"2026-10-23T14:54:31","guid":{"rendered":"https:\/\/glossary.clawoxy.com\/?p=238"},"modified":"2026-10-23T22:54:31","modified_gmt":"2026-10-23T14:54:31","slug":"robots-txt","status":"publish","type":"post","link":"https:\/\/www.clawoxy.com\/glossary\/robots-txt\/","title":{"rendered":"Robots.txt"},"content":{"rendered":"\n

robots.txt<\/code> is a plain-text file at a site’s root that publishes crawler instructions, such as which paths a user-agent group may or may not fetch. A crawler typically checks a URL like https:\/\/example.com\/robots.txt<\/code> before requesting pages on that host.<\/p>\n\n\n\n

Reading the file correctly<\/h2>\n\n\n\n

Rules apply to user-agent groups and URL paths according to the Robots Exclusion Protocol. A Disallow<\/code> line is not a password-protected boundary; the URL may still be accessible to a browser or another client. Conversely, a path omitted from the file is not automatically authorized for every use. Review the site’s terms and access policy as well.<\/p>\n\n\n\n

For site owners, keep the file syntactically valid and test how intended crawlers interpret the rules. Do not list private URLs as a way to hide them; use authentication for restricted content.<\/p>\n\n\n\n

Directives such as Crawl-delay<\/code> are not standardized by RFC 9309, so crawler support differs. Consult the documentation for the crawler you use.<\/p>\n\n\n\n

Sources<\/h2>\n\n\n\n