Robots.txt Generator
Set your crawl rules and sitemap URL, then copy or download a ready-to-upload robots.txt file for your site's root directory.
User-agent: *
Disallow: /admin/How to use this tool
- 1Set the user-agent, or leave it as * to apply to all crawlers.
- 2Add allow or disallow rules for the paths you want to control.
- 3Add your sitemap URL, then copy or download the file and upload it to your site's root.
Example use cases
Block admin areas
Disallow /admin/ and /login/ so search engines never crawl or index private dashboard pages.
Allow full crawling
Leave rules empty and just add a sitemap reference when you want crawlers to access your entire site.
Frequently asked questions
Where does robots.txt need to live?
It must be uploaded to the root of your domain, for example https://example.com/robots.txt, or crawlers won't find it.
Does disallowing a page remove it from Google?
Not by itself. Disallow only blocks crawling; use a noindex tag or removal request to fully remove a page from search results.
Do I need to include a sitemap line?
It's optional but recommended — it helps search engines discover your sitemap and crawl your site more efficiently.