Glossary
What is robots.txt?
The file telling crawlers what to fetch, where sitemaps live, and increasingly which AI crawlers may use your content.
In more depth
List sitemaps for all languages; consider explicit rules for GPTBot, ClaudeBot and other AI agents.
Why it matters for your website
Understanding robots.txt helps you make better decisions when taking a WordPress site multilingual — from choosing URL structures and engines to controlling costs. Transxlate handles the mechanics (language URLs, lang attributes, caching, hreflang on Pro) so terms like this become settings, not engineering projects.
Related terms
Translate your WordPress site today
Bring your own Google or DeepL key, pay wholesale rates, own your translations forever.