Every website on the open web has a quiet, one-page conversation with the machines that visit it. Long before a search engine ranks your pages, before an AI model decides whether to learn from your writing, and before a link-analysis crawler starts mapping your site, those bots pause at a single plain-text file and ask a simple question: where am I allowed to go? That file is robots.txt, and for a document that is often only a few lines long, it carries a surprising amount of weight. Get it right and crawlers spend their time on the pages that matter, your server breathes easier, and your best content gets discovered faster. Get it wrong and you can accidentally hide an entire site from Google, leak the location of pages you meant to keep quiet, or waste crawl budget on endless dead ends.
The Robots.txt Generator on this page exists to make the right version effortless. Instead of memorizing syntax, second-guessing whether a slash belongs at the start of a path, or copying a snippet from a forum that may be years out of date, you answer a few plain questions and watch a correct, standards-compliant file assemble itself in real time. Under the hood it follows the same rules that Googlebot and Bingbot use, warns you about the mistakes that quietly damage sites, and even lets you test any URL against your rules before you publish. This guide walks through how the tool works, then goes deep on the file itself: what it is, where it came from, how search engines interpret it, how it differs across platforms like Blogger, WordPress and Shopify, how to handle the new wave of AI crawlers, and the handful of best practices that separate a healthy robots.txt from a harmful one.
What a robots.txt file actually is
A robots.txt file is a set of instructions for automated visitors, called crawlers, robots, spiders or user-agents. It lives at the very root of a domain, at an address such as https://www.example.com/robots.txt, and it tells well-behaved crawlers which parts of the site they may request and which they should leave alone. The file is written in a small, deliberately simple text format so that any crawler, in any programming language, can parse it without ambiguity. There is no login, no database, and no special software involved; it is just a text file served over HTTP, and anyone in the world can open it in a browser.
It is important to understand what kind of instruction this is. A robots.txt rule is a request about crawling, which is the act of fetching a URL. It is not a lock, not a firewall, and not a guarantee. The protocol works on the honor system: reputable crawlers such as those run by Google, Microsoft, Apple and the major AI companies read the file and obey it, while malicious scrapers can and do ignore it entirely. This single distinction explains most of the confusion people have about robots.txt, and we will return to it several times, because misunderstanding it is the source of the two most damaging mistakes site owners make.
One robots.txt governs exactly one origin, defined by the combination of protocol, host and port. That means http://example.com, https://example.com and https://blog.example.com are three separate origins, each of which needs its own file at its own root. A robots.txt placed in a subfolder, such as example.com/blog/robots.txt, is simply ignored; crawlers only ever look in the root. The filename itself must be entirely lowercase, because the request is case-sensitive on most servers, so Robots.txt or ROBOTS.TXT will not be found.
How our Robots.txt Generator works
The tool is designed so that a complete beginner can produce a professional file in under a minute, while an experienced SEO can still express complex, multi-crawler rules without hand-editing text. Everything happens instantly in your browser; nothing is uploaded, and the output updates the moment you change anything.
Start with your site and a platform preset
You begin by entering your website URL, which the tool uses to build correct, absolute sitemap links and to sanity-check the protocol you are using. Then you pick a platform preset. Choosing Blogger, WordPress, Shopify or Squarespace instantly loads the rules that platform actually needs, so a Blogger user immediately gets a file that keeps crawlers out of internal search results while pointing them at the right sitemaps, and a WordPress user gets the well-known combination that blocks the admin area while still allowing the single admin file that themes and plugins rely on. If you host your own site or simply want to start from a clean slate, the Custom preset opens everything, and dedicated "allow all" and "block all" presets are there for the two extreme cases.
Build precise crawler rules in groups
Real robots.txt files are organized into groups, where each group names one or more user-agents and then lists the paths they may or may not visit. The generator mirrors this exactly. You can add as many groups as you like, choose a target crawler from a list of more than thirty common ones or type a custom name, and then enter the paths to block under Disallow and the exceptions to permit under Allow, one per line. If you forget the leading slash the tool adds it for you and tells you it did. Each group can also carry a crawl-delay value for the search engines that still honor it. When two groups accidentally target the same crawler, the tool merges them and warns you, because duplicate groups are a classic source of silent misbehavior.
One-click controls for AdSense, AI crawlers and scrapers
Three switches handle the requests that would otherwise take a lot of manual typing. The first explicitly welcomes Google's AdSense fetcher so that ad targeting can read every page. The second blocks the current generation of AI training crawlers in a single move, adding a group that covers the major model-training user-agents. The third blocks the aggressive SEO scraper bots that some site owners prefer to keep off their servers to save bandwidth. Each switch also explains, in plain language, exactly which crawlers it affects, so you are never toggling something you do not understand.
Live validation, statistics and a real path tester
As you work, a validation panel flags problems in real time. It will warn you if a rule would block your entire site, if you are about to stop Google from reading the CSS or JavaScript it needs to render your pages, if a sitemap line is not an absolute URL, or if the protocol of a sitemap does not match your site. A small statistics strip shows the line count, the number of rules, and the exact byte size, which matters because search engines cap how much of the file they will read. Best of all, the built-in tester lets you type any path, pick any crawler, and see instantly whether your current rules would allow or block it, along with the exact rule that decided the outcome. This turns robots.txt from a document you hope is correct into one you can prove is correct before it ever goes live.
Copy, download and paste anywhere
When you are happy, you copy the file to your clipboard or download a ready-to-upload robots.txt, and the tool shows a short, platform-specific note telling you exactly where to put it. Because the whole thing works in six languages and adapts to any screen width, it is equally at home in a full blog post, a documentation page, or a narrow sidebar, and it never interferes with the styling of the page that hosts it.
The anatomy of a robots.txt file
To use the tool well, and to read any site's file with confidence, it helps to understand the grammar. A robots.txt file is a sequence of records, and each record is a single line made of a field name, a colon, and a value. Blank lines separate one group of rules from the next, and anything after a # on a line is a comment that crawlers ignore. The handful of fields you will ever need are summarized below.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the crawler that the following rules apply to. An asterisk means "all crawlers not named elsewhere". | User-agent: Googlebot |
| Disallow | Tells the named crawler not to request URLs that begin with the given path. An empty value means "nothing is blocked". | Disallow: /private/ |
| Allow | Creates an exception to a Disallow, permitting a more specific path inside a blocked folder. Supported by Google and Bing. | Allow: /private/public.html |
| Sitemap | Points crawlers to an XML sitemap. It must be an absolute URL and can appear anywhere in the file, independent of any group. | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Asks a crawler to wait a number of seconds between requests. Ignored by Google, honored by Bing, Yandex and some others. | Crawl-delay: 10 |
Groups and the meaning of the asterisk
A group starts with one or more User-agent lines and continues with the rules that apply to them, until a blank line or the end of the file. The special user-agent * is a catch-all, but it is not a global override. When a crawler reads the file, it looks for the single group whose user-agent best matches its own name; only that group applies to it. If a crawler finds a group named specifically for it, the catch-all group is ignored for that crawler entirely. This is why you will often see a specific group for Googlebot followed by a broader group for everyone else, and why placing a rule in the wrong group is such a common and confusing error.
Paths, prefixes and the two wildcards
Every path in a Disallow or Allow line is matched as a prefix from the start of the URL path. Writing Disallow: /images therefore blocks /images, /images/ and /images-archive alike, because all three begin with that string. To express more precise patterns, the major search engines support two wildcards that, while not part of the original 1994 convention, are now universally understood. The asterisk * matches any sequence of characters, so Disallow: /*.pdf targets every PDF anywhere on the site. The dollar sign $ anchors a rule to the end of the URL, so Disallow: /*.pdf$ blocks files that end in ".pdf" but leaves a URL like /report.pdf?download=1 alone. Combining the two lets you write compact rules for entire categories of URLs, and the generator's tester makes it easy to confirm they behave the way you expect.
How search engines read and obey robots.txt
Knowing the syntax is only half the picture. The other half is understanding how a crawler resolves conflicts, because a single URL can match several rules at once. Google's widely documented behavior, which most other engines broadly follow, is that the most specific rule wins, where specificity is measured by the number of characters in the matched path. When a Disallow and an Allow of equal length both match the same URL, the least restrictive rule wins, meaning Allow takes precedence. This is exactly why the standard WordPress file can block the entire /wp-admin/ folder while a longer, more specific Allow line keeps the single admin-ajax.php file reachable.
| Situation | Google's resolution |
|---|---|
| No robots.txt file, or an empty one | Everything is allowed; the site is crawled normally. |
| File returns a 404 or other 4xx error | Treated as no restrictions; crawling proceeds as if the file did not exist. |
| File returns a 5xx server error | Crawling may be paused temporarily, as the crawler cannot confirm what is allowed. |
| Two rules of different lengths match a URL | The longer, more specific rule is applied. |
| An Allow and a Disallow of equal length match | Allow wins; the URL is crawlable. |
| File larger than 500 KiB | Only the first 500 KiB is parsed; the rest is ignored. |
Two practical details flow from this. First, matching of the user-agent name is case-insensitive, and Google chooses the most specific matching group by substring, so a rule written for "Googlebot" also applies to "Googlebot-Image" unless a more specific group exists. Second, crawlers cache the file, typically for around a day, so a change you publish is not necessarily seen the moment you save it. If you have just fixed a serious mistake, it can take a crawl cycle for the correction to take effect, which is a good reason to get the file right before publishing rather than after.
A short history of the Robots Exclusion Protocol
The idea behind robots.txt is almost as old as the web itself. In early 1994, the web was tiny, servers were fragile, and the first automated crawlers were beginning to explore it. One such crawler repeatedly hammered the server of a Dutch software engineer named Martijn Koster, and out of that frustration came a proposal: a simple, voluntary convention that would let a site tell robots which areas to avoid. Koster floated the idea on a mailing list of web pioneers, they reached a rough consensus, and the Robots Exclusion Protocol was born. There was no standards body, no formal specification, and no enforcement, just a shared agreement that a file named robots.txt at the root of a site would be respected.
Remarkably, that informal handshake held the web together for a quarter of a century. Search engines came and went, the number of pages exploded from thousands to trillions, and yet the humble text file kept doing its job on little more than good faith and mutual interest. Along the way the convention quietly grew: the Allow directive was added so that exceptions could be carved out of blocked areas, wildcards were adopted so patterns could be expressed, and in 2007 the major search engines agreed on the Sitemap directive so that a robots.txt could also announce where a site's XML sitemap lived.
The protocol finally became an official internet standard in September 2022, when the Internet Engineering Task Force published RFC 9309, the "Robots Exclusion Protocol." Co-authored by engineers including Koster himself alongside members of Google's crawling team, the document formalized decades of accepted practice: where the file must live, how big it can be, how errors should be handled, and how rules should be matched. It changed very little about how the file works day to day, because it was largely writing down what everyone already did. But it mattered symbolically, turning a gentlemen's agreement into a documented standard that new crawlers, including the AI crawlers that arrived soon after, could point to as the authoritative reference.
Crawling is not indexing: the biggest misconception
If you take one idea away from this guide, make it this one, because it trips up beginners and professionals alike. Blocking a URL in robots.txt stops a crawler from fetching the page, but it does not reliably stop that page from appearing in search results. When Google discovers a link to a page it is not allowed to crawl, it may still index the URL itself, showing it in results without a description because it was never allowed to read the content. You end up with the worst of both worlds: the page is visible in search, but with an empty, unhelpful listing.
The reason is subtle but logical. A "noindex" instruction, the thing that actually keeps a page out of the index, lives inside the page, either as a meta tag in the HTML or as an HTTP header. If you block the page in robots.txt, the crawler never fetches it, and therefore never sees the noindex instruction, so it cannot obey it. To reliably keep a page out of search, you must allow crawling and add a noindex directive, or protect the page behind a login. Using robots.txt for this purpose does the opposite of what people intend. It is also worth knowing that Google formally stopped honoring an unofficial "noindex" line inside robots.txt itself on the first of September 2019, so that shortcut, which some old tutorials still recommend, no longer works at all.
robots.txt and your crawl budget
For small sites, crawlers can visit every page comfortably, and robots.txt is mostly about tidiness. For large sites, it becomes a genuine performance tool. Search engines allocate each site a rough crawl budget, an informal limit on how many pages they will fetch in a given period, based on the site's size, health and popularity. If that budget is spent on thousands of low-value URLs, the pages you actually care about are crawled less often and updated more slowly in the index.
This is where thoughtful Disallow rules earn their keep. Faceted navigation that generates endless filtered combinations, internal search result pages, printer-friendly duplicates, session identifiers in URLs, and infinite calendar archives are all classic "crawler traps" that can consume budget without adding anything to search. Blocking these patterns steers crawlers toward your real content. The key is precision: you want to close the dead ends without accidentally blocking the templates, images, scripts and stylesheets that search engines need to render and understand your pages. The generator's warnings are tuned to catch exactly that last mistake, because blocking your own assets is one of the easiest ways to hurt rankings while believing you are helping.
Platform by platform: Blogger, WordPress, Shopify and more
Because every publishing platform structures its URLs differently, the ideal robots.txt is not the same everywhere. The table below summarizes how the major platforms handle the file and what a sensible baseline looks like on each, all of which the generator's presets produce for you.
| Platform | How robots.txt is handled | Typical baseline |
|---|---|---|
| Blogger / Blogspot | Auto-generated, but you can enable a custom file under Settings, Crawlers and indexing. | Block internal search at /search, allow the rest, and list the blog's sitemaps. |
| WordPress | A virtual file is served automatically; SEO plugins like Yoast and Rank Math let you edit it, or you can upload a real file via FTP. | Block /wp-admin/ while allowing /wp-admin/admin-ajax.php, and point to the sitemap. |
| Shopify | Auto-generated and, since 2021, editable through a robots.txt.liquid theme template. | The defaults already block /admin, /cart, /checkout and similar; add only what your store needs. |
| Squarespace | Generated automatically and not editable, by design. | Use a generated file as a reference only; you cannot override it. |
| Self-hosted / custom | You are fully in control; upload the file to the web root yourself. | Whatever your architecture requires, plus every sitemap you publish. |
The Blogger case deserves a special mention, since this tool is built and shared with the Blogger community in mind. Blogger sites generate internal search and label pages under /search that duplicate content and add nothing to the index, which is why the platform's own default keeps crawlers out of that path. When you enable a custom robots.txt in Blogger's settings, you can paste the file this tool produces, keeping the sensible search block while making sure both your posts sitemap and your pages sitemap are advertised so nothing gets overlooked.
Controlling AI crawlers and scrapers
The most significant change to robots.txt in years has nothing to do with search at all. Since 2023, a new category of crawler has arrived: bots that gather web pages to train large language models and to answer questions in AI assistants. Many site owners are comfortable being in Google's index but want a say in whether their work is used to train commercial AI, and robots.txt has become the primary place to express that preference. The major AI companies have, to their credit, published named user-agents and committed to respecting the file, so a few lines can opt you out of the ones you choose.
| Crawler | Operated by | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collects pages to help train future models. |
| Google-Extended | Controls use of your content for AI training without affecting Search ranking or crawling. | |
| ClaudeBot / anthropic-ai | Anthropic | Gathers web content for model training. |
| CCBot | Common Crawl | Builds a large public dataset widely used to train models. |
| PerplexityBot | Perplexity | Fetches pages to support an AI answer engine. |
| Bytespider | ByteDance | Crawls content for AI and product development. |
There is an important nuance with Google-Extended. Unlike a normal Disallow for Googlebot, which would remove you from search, Google-Extended is a dedicated control that governs only whether your content feeds Google's generative AI products. Blocking it lets you keep your hard-won search visibility while stepping out of AI training, which is exactly the balance many publishers want. The generator's AI switch bundles these and other current AI user-agents into one clean group, and because new ones appear regularly, it is worth revisiting your file periodically. Keep in mind, too, that the honor system still applies: naming a crawler only works if that crawler chooses to obey, and the least scrupulous scrapers ignore the file no matter what it says.
Security and privacy: what robots.txt can and cannot do
Because the file is publicly readable, it is a poor place to hide anything sensitive. Listing Disallow: /secret-admin-panel/ does not protect that folder; it publishes a signpost to it for anyone curious enough to read your robots.txt, which is the first file many security researchers and attackers check. If a URL must be kept private, the answer is authentication, access controls, or removing it from the public server, never a Disallow line. Think of robots.txt as a set of polite requests posted on a public notice board, not as a lock on a door.
This does not make the file useless for housekeeping. It is perfectly reasonable to keep crawlers out of areas that are simply not meant for search, such as staging paths, cart and checkout flows, or internal tools, as long as you remember that "not meant for search" is different from "confidential." For anything truly private, the robots.txt is the wrong layer of defense, and treating it as a security measure is one of the most consequential misunderstandings in web publishing.
Common mistakes that quietly hurt rankings
Most robots.txt damage is invisible at first. The file looks fine, the site still loads, and only weeks later does someone notice traffic has slipped. These are the errors worth guarding against, all of which the generator is built to prevent or warn about.
- Blocking the whole site by accident. A stray
Disallow: /under the catch-all user-agent tells every crawler to stay out completely, and it is a shockingly common leftover from a staging environment that was never cleaned up before launch. - Blocking CSS and JavaScript. Modern search engines render pages the way a browser does, and if they cannot fetch your styles and scripts they may see a broken layout and rank the page lower. Folders like a theme or assets directory should almost never be disallowed.
- Using robots.txt to remove a page from search. As covered above, this often keeps the page visible with an empty listing. Use noindex on a crawlable page instead.
- Relative or malformed sitemap URLs. The Sitemap directive requires a full, absolute URL beginning with the protocol; a relative path is silently ignored.
- Putting rules in the wrong group. Because a specific user-agent group overrides the catch-all for that crawler, a rule you thought applied to everyone may not apply to Googlebot at all if a Googlebot group exists.
- Forgetting the file is per origin. A single robots.txt cannot cover a bare domain and a subdomain; each needs its own at its own root.
A best-practice checklist
Before you publish, run through this short list. The generator handles most of it automatically, but it is good to understand what "correct" looks like.
- Place the file at the exact root of each origin, named in lowercase as
robots.txt. - Keep it small and readable; you are almost never near the size limit, but clarity prevents mistakes.
- Allow crawlers to reach the CSS, JavaScript and images needed to render your pages.
- Block only genuine dead ends, such as internal search, faceted duplicates and infinite archives.
- Advertise every XML sitemap you maintain, using absolute URLs.
- Never rely on robots.txt to hide confidential content; use real access controls.
- Decide deliberately how you want to treat AI training crawlers, and revisit that choice as new ones appear.
- Test representative URLs against your rules before going live, and check the file again after any major site change.
Add the Robots.txt Generator to Your Own Website
Run a blog, a webmaster resource, an SEO site, or a documentation hub where a robots.txt tool would earn its place? You are welcome to embed the Tooliqo Robots.txt Generator directly. It is fully self-contained, works in all six site languages, adapts to any screen, and will not interfere with your page's own styling. This share code is universal: it works on every platform and on any website, and you can drop it anywhere a snippet fits, inside a post, a page, a sidebar or a footer. Pick whichever of the two methods suits your platform, paste the snippet where you want the tool to appear, and you are done.
Method 1 — Direct iframe (best for Blogger posts and pages)
The most reliable option for Blogger and most content platforms. Drop this into the HTML view of your post exactly where the generator should sit. The allowfullscreen attribute keeps the built-in fullscreen button working inside the frame.
<iframe data-tooliqo src="https://tools.tooliqo.co/Robots-txt-Generator/?lang=en"title="Tooliqo — Robots-txt-Generator"style="width:100%;border:0;height:820px" height="820" loading="lazy" scrolling="no"allowfullscreen allow="fullscreen; clipboard-write"></iframe><script src="https://tools.tooliqo.co/embed.js" async></script>
Method 2 — Lightweight embed script
If your platform allows external scripts, this drop-in version keeps the tool sized automatically and is the tidier choice for full themes. Place the div where you want the generator, and the script does the rest.
<div class="tooliqo-tool" data-tool="Robots-txt-Generator" data-lang="en"data-height="680"></div><script src="https://tools.tooliqo.co/embed.js" async></script>
To load the generator in another language, simply swap the language code — en, fr, es, zh, hi, or ar — in whichever snippet you choose. The Arabic version switches to a right-to-left layout automatically.
Frequently asked questions
Do I actually need a robots.txt file?
Not strictly. If you have no file, crawlers assume everything is allowed and your site is crawled normally. But having a clean, correct file lets you point crawlers at your sitemaps, keep them out of low-value paths, and state your position on AI crawlers, so it is well worth the two minutes it takes to create one.
Where exactly do I upload it?
At the root of your domain, so it resolves at yoursite.com/robots.txt. On Blogger you paste the text into Settings, Crawlers and indexing. On WordPress you can use an SEO plugin's file editor or upload via FTP. On Shopify you edit the robots.txt.liquid template. The tool shows the right instruction for your chosen platform.
Will blocking a page in robots.txt remove it from Google?
No, and this is the single most important thing to understand. Blocking stops crawling, not indexing, and a blocked page can still appear in results with no description. To remove a page from search, allow crawling and add a noindex tag, or require a login.
Can I stop AI companies from training on my content?
You can ask the ones that publish named crawlers and commit to obeying robots.txt, such as GPTBot, Google-Extended, ClaudeBot and CCBot, by disallowing them. The generator does this with a single switch. Bear in mind that compliance is voluntary, so this influences responsible crawlers rather than guaranteeing anything against every scraper.
Why is Crawl-delay ignored sometimes?
Google does not support Crawl-delay at all and manages its crawl rate through its own systems. Bing, Yandex and some other engines do honor it. The tool lets you set it where it is useful and warns you where it is not.
How big can the file be?
Google reads up to 500 KiB and ignores anything beyond that. In practice a good robots.txt is a few lines to a few dozen, so you will never come close to the limit, and the tool shows the exact byte size as you build.
Is the tool free and does it store my data?
It is completely free, runs entirely in your browser, and sends nothing to any server. Your URL and rules never leave the page, which is why it works even offline once loaded.
A robots.txt file is a small thing that quietly shapes how the entire automated web sees your site. It decides where search engines spend their attention, how quickly your best pages are discovered, and, increasingly, whether your words help train the AI systems people now ask for answers. For something so consequential, it deserves to be built with care rather than copied blindly, and that is exactly what this generator is for: a correct, standards-compliant file, validated as you go and testable before you publish, ready to paste wherever you need it. Take a minute, set it up properly once, and you can stop worrying about the conversation your site is having with the machines.