Free Robots.txt Generator

Build a valid robots.txt file for your site — control which crawlers can access which paths, block AI scrapers, and point search engines to your sitemap.

One per line, e.g. /admin or /cart

Optional — one per line. Useful to re-allow a path inside a disallowed folder.

How to create a robots.txt file

  1. 1

    List paths to disallow

    Add any folders or URL patterns crawlers shouldn't request, one per line.

  2. 2

    Add exceptions to allow

    Optionally re-allow a specific path inside a disallowed folder.

  3. 3

    Add your sitemap URL

    Paste your sitemap.xml address so crawlers can find it.

  4. 4

    Pick any AI crawlers to block

    Tick the bots you don't want scraping your content for AI training or browsing.

  5. 5

    Generate and upload

    Click "Generate robots.txt", then download and upload the file to your site's root.

Example use cases

1

Blocking admin & checkout paths

Keep crawlers away from login, cart, and account pages that have no reason to appear in search results.

2

Opting out of AI training crawlers

Block GPTBot, Google-Extended, CCBot, and similar bots from scraping your content for AI model training.

3

Pointing crawlers to your sitemap

Help search engines discover every indexable page faster by declaring your sitemap location.

Frequently asked questions

What is a robots.txt file?

A plain-text file placed at the root of your domain (e.g. example.com/robots.txt) that tells search engine and other crawlers which parts of your site they may or may not request.

Does robots.txt keep pages out of Google?

Not reliably. Disallowing a path stops crawling, but a page can still get indexed (e.g. from external links) without its content being shown. To fully block indexing, use a noindex meta tag or header instead — those require the page to be crawlable, so don't disallow a page you're trying to noindex.

Does blocking AI bots here actually stop them?

Only for crawlers that respect robots.txt, which the major ones (GPTBot, Google-Extended, CCBot, ClaudeBot, and similar) publicly commit to. It's a request, not an access control — it can't force compliance from a bot that ignores it.

Where do I upload this file?

Upload it as robots.txt in your site's root directory so it's reachable at https://yourdomain.com/robots.txt — that exact path is where crawlers look for it.

Should I list my sitemap in robots.txt?

It's a common convention that helps crawlers discover it, but it's optional — submitting your sitemap directly in Google Search Console or Bing Webmaster Tools works independently of robots.txt.