Validate the public robots.txt

Request the exact public URL while signed out:

https://example.com/robots.txt

Response checklist

1. Confirm the response is 200 OK plain text.
2. Compare it byte-for-byte with the intended policy.
3. Confirm there is no HTML, login page, warning, or redirect chain.
4. Open every sitemap URL.
5. Test through the CDN as well as the origin where possible.
6. Repeat for mapped domains and subsites.

Use the validation tool supplied by the search service that matters to the site. POM Tweaks stores and returns the text but does not simulate individual crawler parsers.

Diagnose unexpected content

If the response does not match:

• check for a physical root file;
• inspect CDN or server caching;
• identify another WordPress robots_txt filter;
• confirm the module and its saved text are enabled for the current site;
• verify the request reaches the intended multisite site.

An empty configured value deliberately leaves WordPress’s prior virtual output unchanged.

If the saved content is unavailable, the prior virtual output is also preserved. Reopen the editor, enter the intended rules and save again before checking the public response.

Roll back

Disable Robots.txt, clear the cached response, and request the URL again. Confirm the previous WordPress provider returns. Restore or remove a physical file separately because the module never manages it.