Write crawler rules

The Robots.txt field accepts the complete plain-text response. POM Tweaks does not merge individual lines with the WordPress default, validate crawler syntax, or add a sitemap automatically.

Start from an explicit policy

A minimal public-site example is:

User-agent: *
Disallow:
Sitemap: https://example.com/wp-sitemap.xml

Replace the sitemap URL and rules for the actual site. Do not copy a staging policy into production.

Important limits

• robots.txt is a crawl request, not access control.
• A disallowed URL can still be discovered or indexed from other signals.
• Private content requires WordPress permissions, authentication, and server protection.
• Paths are interpreted by crawlers, so site language and subdirectory structure matter.
• A syntax error can change how a crawler interprets the remaining file.

Save safely

1. Copy the current public response.
2. Identify the owner of every existing rule.
3. Enter the complete replacement text.
4. Use absolute sitemap URLs.
5. Save the module.
6. Purge any CDN or page cache for /robots.txt.
7. Run the public validation.

For a multisite network, configure and test the current site’s hostname and path independently.