Configure robots and crawler access

POM Translate keeps machine-readable root files outside language routing. It does not create or translate robots directives for individual pages.

Root files remain unprefixed

Paths such as these stay at the site root:

• /robots.txt;
• /sitemap_index.xml;
• other sitemap XML and XSL files;
• /llms.txt;
• /favicon.ico.

Adding a language prefix could bypass WordPress or the SEO plugin rewrite responsible for the file. Do not submit /es/sitemap_index.xml or disallow only a prefixed robots path.

Page-level robots directives

Use WordPress or the active SEO plugin to control index, noindex, follow, and sitemap inclusion. POM Translate does not change those directives based on translation status.

Apply the same deliberate policy to:

• unfinished language routes;
• search and author archives;
• paginated archives;
• excluded posts or terms;
• staging sites;
• WooCommerce account, cart, and checkout pages.

Browser-language redirects

Automatic browser detection is a visitor convenience, not a crawler discovery system. Stable language URLs and HTML links should remain accessible without requiring a redirect or cookie.

Test known crawler-like requests without an existing language preference. An explicitly prefixed URL should remain in that language.

Crawler review

1. Fetch /robots.txt and the sitemap index from the root.
2. Confirm both return their expected content type.
3. Fetch representative base and target URLs.
4. Inspect HTTP status, redirect chain, canonical, robots meta, <html lang>, and hreflang.
5. Confirm no private account or checkout routes are exposed contrary to the SEO plugin's policy.
6. Check server and CDN rules for accidental locale blocking.

POM Translate skips normal HTML output translation for XML, sitemap, feed, embed, REST, and other machine-oriented responses. This prevents translated prose processing from corrupting those formats.