Skip to content
RankYouSite
Add Listing

Generators

Robots.txt Generator

Create a clean, valid robots.txt file for your website. Set rules for all crawlers, block private folders, add your sitemap, and decide which search engines and AI bots (GPTBot, ClaudeBot, PerplexityBot and more) may crawl your pages. Everything runs in your browser.

Free · No signup

Rules for all robots (User-agent: *)

Quick add common paths

Disallowed paths

Start with / . Use * as a wildcard and $ to match the end of a URL.

Allowed paths

Exceptions inside a disallowed folder. The longest matching rule wins.

Googlebot ignores Crawl-delay (Google adjusts its crawl rate automatically). Bing and Yandex respect it.

Search engine crawlers

Default follows the * rules above. Allow gives the bot full access (it then ignores the * rules). Block disallows the whole site for that bot.

  • Googlebot

    Google · Google Search crawler

  • Googlebot-Image

    Google · Google Images

  • Bingbot

    Microsoft · Bing Search (also feeds Copilot)

  • Yandex

    Yandex · Yandex Search

  • Baiduspider

    Baidu · Baidu Search

  • DuckDuckBot

    DuckDuckGo · DuckDuckGo Search

  • Applebot

    Apple · Siri and Spotlight search

AI crawlers

Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot…) collect pages to train AI models. Blocking them does not affect your search rankings.

AI search and user bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User…) fetch pages to show and cite them in AI answers. Blocking them can remove your site from ChatGPT, Claude or Perplexity answers and the traffic they send.

  • GPTBotTraining

    OpenAI · Collects training data

  • OAI-SearchBotAI search

    OpenAI · ChatGPT search index

  • ChatGPT-UserUser fetch

    OpenAI · Fetches pages a user asks about

  • ClaudeBotTraining

    Anthropic · Collects training data

  • Claude-SearchBotAI search

    Anthropic · Claude search index

  • Claude-UserUser fetch

    Anthropic · Fetches pages a user asks about

  • PerplexityBotAI search

    Perplexity · Perplexity search index

  • Perplexity-UserUser fetch

    Perplexity · Fetches pages a user asks about

  • Google-ExtendedTraining

    Google · Gemini training and grounding (not Search)

  • Applebot-ExtendedTraining

    Apple · Apple AI model training

  • CCBotTraining

    Common Crawl · Open web dataset used for AI training

  • BytespiderTraining

    ByteDance · AI training data

  • AmazonbotTraining

    Amazon · Alexa answers and AI training

  • meta-externalagentTraining

    Meta · Meta AI training data

Sitemap URLs

Full URLs, e.g. https://example.com/sitemap.xml. They are listed at the end of the file.

Your robots.txt

# robots.txt
# Generated with the RankYouSite free Robots.txt Generator
# Upload to the root of your domain, e.g. https://example.com/robots.txt

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/

Sitemap: https://example.com/sitemap.xml
  • Save the file as robots.txt at the root of your domain (e.g. https://example.com/robots.txt). Crawlers ignore it anywhere else, and each subdomain needs its own.
  • robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other sites link to it. To keep a page out of results, use a noindex meta tag and leave it crawlable.

How it works

  1. 1

    Set the default rules

    Choose Allow all, Disallow all, or Custom, then add the folders you want kept out of crawlers’ reach. Quick-add chips cover WordPress admin, cart, checkout and search pages.

  2. 2

    Pick rules per crawler

    Leave each bot on Default, or allow or block it individually. Use a preset to block AI training bots while keeping AI search bots that can cite and link to you.

  3. 3

    Copy or download

    Add your sitemap URL, check the warnings, then copy the file or download robots.txt and upload it to the root of your domain.

What a robots.txt file does

robots.txt is a plain text file at https://yourdomain.com/robots.txt that tells crawlers which parts of a site they may request. Each block starts with one or more User-agent lines followed by Allow and Disallow rules. A crawler obeys only the most specific group that names it, falling back to the User-agent: * group otherwise.

It is a set of instructions, not a lock. Reputable crawlers follow it, but anyone can still open a blocked URL in a browser, so never rely on robots.txt to hide private or sensitive content. Use passwords or server rules for that.

Crawling is not the same as indexing

Disallowing a URL stops Google from fetching it, but if other pages link to it, Google can still list the bare URL in results without a description. To remove a page from search, add <meta name="robots" content="noindex"> and keep the page crawlable so Google can see that tag.

Also avoid blocking CSS, JavaScript or image folders that your pages need to render. Google renders pages like a browser, and blocked resources can make your content look broken or thin.

AI training bots vs AI search bots

AI companies run different crawlers for different jobs. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect content to train models. Blocking them keeps your content out of future training sets and has no effect on Google or Bing rankings.

Search and user-triggered crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot fetch pages so AI assistants can quote and link to them in answers. For most businesses these bring visitors, so the “Block AI training, allow AI search” preset is a sensible middle ground.

Wildcards, precedence and common mistakes

Google and Bing support * (any characters) and $ (end of URL). Disallow: /*?sort= blocks every URL with a sort parameter, and Disallow: /*.pdf$ blocks PDFs. When an Allow and a Disallow rule both match, the longer (more specific) rule wins, which is why Allow: /wp-admin/admin-ajax.php works inside a blocked /wp-admin/.

The costliest mistake is leaving Disallow: / live after a site launch, which blocks the entire site. Paths are case-sensitive, must start with a slash, and each subdomain (such as shop.example.com) needs its own file.

Frequently asked questions

Where do I upload robots.txt?

At the root of your domain, so it loads at https://yourdomain.com/robots.txt. Crawlers do not look for it in subfolders. On WordPress you can upload it by FTP or edit it with an SEO plugin; on most hosts it goes in the public_html folder.

Will blocking a page in robots.txt remove it from Google?

Not reliably. robots.txt stops crawling, not indexing, so a blocked URL that is linked from elsewhere can still appear in results. Use a noindex meta tag (and keep the page crawlable) or remove the page to take it out of Google.

Should I block AI bots like GPTBot and ClaudeBot?

It depends on your goals. Blocking training bots keeps your content out of model training without hurting search rankings. Blocking AI search bots such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can stop AI assistants from citing and linking to you, which usually costs traffic.

Does blocking Google-Extended affect my Google rankings?

No. Google-Extended is a control token for Gemini training and grounding only. Google Search crawls with Googlebot, so blocking Google-Extended does not change how you rank or appear in Google Search.

Does Google respect Crawl-delay?

No. Googlebot ignores the Crawl-delay directive and sets its crawl rate automatically based on how your server responds. Bing and Yandex do respect it. If Googlebot is overloading your server, return 503 or 429 responses temporarily.

Do I need a robots.txt file at all?

It is not required. Without one, crawlers assume they can crawl everything. A short file is still useful to point crawlers to your sitemap, keep them out of cart, search and admin pages, and set rules for AI crawlers.

Should I add my sitemap to robots.txt?

Yes. A Sitemap line with the full URL helps every crawler find your sitemap, including ones where you have not submitted it manually. You can list several sitemaps or a sitemap index file.

Is this robots.txt generator free and private?

Yes. It is completely free with no sign-up, and the file is built in your browser. Nothing you enter is sent to our servers.

More free SEO tools

View all free tools