Robots.txt Generator
Set a user-agent, list the paths you want to allow or block, and optionally add your sitemap URL to generate a ready-to-use robots.txt file.
How to use
- 1
Set the user-agent
Use * to target all crawlers, or name a specific one.
- 2
List allow and disallow paths
Enter one path per line in each box, e.g. /admin or /public.
- 3
Add your sitemap URL (optional)
Helps crawlers find your sitemap automatically.
- 4
Copy the result
Paste the generated text into a file named robots.txt at your site's root.
Examples
- Disallow /admin and /private, with sitemap https://example.com/sitemap.xml
- User-agent * with no disallow rules → crawlers can access the entire site
Frequently asked questions
Where does robots.txt go on my site?
It must be placed at the root of your domain, e.g. https://example.com/robots.txt. Search engine crawlers look for it there specifically — it won't work in a subfolder.
What does 'User-agent: *' mean?
The asterisk is a wildcard that applies the rules to all crawlers. You can target a specific crawler instead, like Googlebot, by entering its name.
Does Disallow guarantee a page won't be indexed?
Not entirely. Disallow tells well-behaved crawlers not to fetch a path, but a page can still be indexed without being crawled if other pages link to it. Use a noindex meta tag for a stronger guarantee.
Should I include my sitemap here?
It's a common and helpful practice — adding a Sitemap line points crawlers directly to your sitemap.xml, helping them discover your pages faster.