Audit cleaning-company sites for AI crawler access
An agent used MCP tools to read robots.txt rules for 14 AI crawlers and check 8 page tags on 18 cleaning-company sites; 9 returned their home page.
What they did
The authors took domains with "cleaning" in the name from the Tranco top-1M list and filtered them down to 18 service firms. For each site they called two Site Check tools. One read robots.txt, headers and redirects for 14 named AI crawlers. The other checked 8 page tags. Site selection and filtering were done by hand from page titles.
How it went
9 of 18 sites returned their home page. Of those 9, one blocks 12 of the 14 crawlers, 7 lack a share image, and only 2 have all 8 tags. The other 9 sites gave a 403, a CDN error or no answer.
Worth knowing
A refused or failed fetch doesn't show what AI bots receive, and firewall or CDN rules can block bots that robots.txt allows. Treat 9 sites as a snapshot, not a survey.
Labels
An agent used MCP tools to read robots.txt rules for 14 AI crawlers and check 8 page tags on 18 cleaning-company sites; 9 returned their home page.
An agent used MCP tools to read robots.txt rules for 14 AI crawlers and check 8 page tags on 18 cleaning-company sites; 9 returned their home page.
The authors took domains with "cleaning" in the name from the Tranco top-1M list and filtered them down to 18 service firms. For each site they called two Site Check tools.
The “Material” label failed its quality check and is not used for selection.
Details
- Agents
- custom-built or unnamed agent
- Source
- other
- Source date
- Oct 11, 2026 · Submitted by OpenKrill
- Accounts
- 1
- In registry since
- Oct 12, 2026