Free SEO tool
Generate your robots.txt in seconds
Create a valid robots.txt file to control what search engines crawl. Choose a mode, add paths to block and your sitemap URL, and copy the result.
Robots.txt Generator
Blocking a URL in robots.txt prevents crawling, not indexing. To remove a page from Google use the noindex tag instead. Upload the file as robots.txt in your domain root.
Frequently asked questions
What is the robots.txt file?
It is a text file in your domain root that tells crawlers (Googlebot, Bingbot, etc.) which parts of your site they may or may not crawl. It is not mandatory, but it helps control crawl budget and hide internal sections.
Does blocking a page in robots.txt remove it from Google?
No. Disallow prevents crawling, but a blocked URL can still appear indexed if it has links. To remove it from the index, use the noindex meta tag (and do not block it in robots.txt, so Google can read the noindex).
Where do I upload robots.txt?
In the domain root, reachable at your-site.com/robots.txt. On WordPress you can edit it with RankMath or Yoast; on other sites upload it via FTP or your hosting panel.
Should I include the sitemap?
Yes, it is best practice. Adding the Sitemap line helps search engines discover all your URLs faster.
Grow with real data
A correct robots.txt is the baseline. AnalySEO connects your Search Console and tells you what to optimize next to grow traffic. AnalySEO