Free SEO tool
Generate your robots.txt in seconds
Create a valid robots.txt file to control what search engines crawl. Choose a mode, add paths to block and your sitemap URL, and copy the result.
Robots.txt Generator
Blocking a URL in robots.txt prevents crawling, not indexing. To remove a page from Google use the noindex tag instead. Upload the file as robots.txt in your domain root.
Frequently asked questions
What is the robots.txt file?
It is a text file in your domain root that tells crawlers which parts of your site they may or may not crawl. It helps control crawl budget and hide internal sections.
Does blocking a page in robots.txt remove it from Google?
No. Disallow prevents crawling, but a blocked URL can still appear indexed if it has links. To remove it from the index, use the noindex meta tag.
Where do I upload robots.txt?
In the domain root, reachable at your-site.com/robots.txt. On WordPress you can edit it with RankMath or Yoast.
Should I include the sitemap?
Yes. Adding the Sitemap line helps search engines discover all your URLs faster.
Grow with real data
A correct robots.txt is the baseline. AnalySEO connects your Search Console and tells you what to optimize next to grow traffic. AnalySEO