What is Robots.txt?

The robots.txt file serves as the first point of contact between your website and search engine crawlers. Located at yourdomain.com/robots.txt, it provides directives that control crawler access to specific areas of your site. While crawlers generally respect these rules, robots.txt is more of a suggestion than a security barrier—it won't hide content from determined users.

Key directives include: User-agent: specifies which crawler (Googlebot, *, etc.), Disallow: blocks access to specific paths, Allow: permits access within blocked areas, Sitemap: points to your XML sitemap, and Crawl-delay: suggests crawl speed (not universally supported). Directives apply to user-agents listed above them until the next user-agent line.

Block with robots.txt: Admin and login pages, Internal search results, Staging or development sections, Duplicate parameter URLs, Private user directories, and PDF or resource libraries. Remember: blocking a URL prevents crawling but pages may still appear in search if linked to from other sites.

Avoid these errors: Blocking entire site accidentally, Using robots.txt for security (use authentication instead), Blocking CSS/JS files (hurts mobile rendering), Forgetting trailing slashes (affects path matching), and Conflicting directives without proper order. Test changes before deploying.

Validate robots.txt with: Google Search Console's robots.txt Tester, Bing Webmaster Tools, Third-party validators, and Manual testing with site: searches. After changes, use Search Console's URL Inspection to verify pages are crawlable.