1. How to Check and Validate Robots.txt Files Online
Checking a robots.txt file requires validating three core elements: 1. **Directive Syntax:** Ensuring every record has valid `User-agent`, `Allow`, `Disallow`, or `Sitemap` declarations with correct colon delimiters. 2. **Path Matching Logic:** Simulating whether target URLs (e.g., `/blog/post-1/`) are permitted or blocked based on RFC 9309 prefix and wildcard rules (`*` and `$`). 3. **Crawler Group Scope:** Confirming that specific crawlers (like Googlebot or GPTBot) have explicitly configured directive blocks, as specific user-agents do NOT inherit disallow rules from `User-agent: *`.
2. The Most Common Robots.txt Errors Caught by Validators
- **Accidental Full Disallow:** Using `Disallow: /` instead of `Disallow:` blocks crawlers from the entire site. - **Blocking CSS and JavaScript Assets:** Disallowing `/assets/` or `/static/` prevents Googlebot from rendering pages, triggering mobile usability and SEO ranking drops. - **Case Sensitivity Mismatches:** Directives in robots.txt are case-sensitive. `Disallow: /admin/` will not block `/Admin/`. - **Malformed Wildcard Endings:** Placing wildcards without understanding that trailing slashes anchor directories while `$` anchors the exact end of a URL path. - **Missing XML Sitemap Declaration:** Forgetting to declare `Sitemap: https://yourdomain.com/sitemap-index.xml` at the bottom of the file.
3. RFC 9309 Path Precedence & Conflict Resolution
When a URL matches both an `Allow` and a `Disallow` rule within the same user-agent group, RFC 9309 mandates the **longest match** rule: the directive with the greater number of characters wins. If the pattern lengths are identical, `Allow` takes precedence.
4. Testing AI Search Crawlers vs. Model Training Crawlers
Modern robots.txt files must balance SEO visibility with IP protection. Webmasters should distinguish between real-time search crawlers (`ChatGPT-User`, `PerplexityBot`, `Googlebot`) and training dataset scrapers (`GPTBot`, `CCBot`, `Bytespider`). By testing your directives in this validator, you can verify that live search engines can index your site while restricting automated dataset scraping.
01. Privacy and offline availability
Core tools process input locally. Optional AI sends the selected snippet through Cloudflare to Cloudflare AI or the optional Google Gemini fallback after consent; provider retention policies apply. Contact submissions send your details to our email provider. Offline use requires the page and its assets to finish caching; external AI and contact delivery require internet.