A robots.txt file is simply a plain text file that sits at the root of your website (domain.com/robots.txt). To see if your site has one, type /robots.txt after your domain in your browser.
Robots.txt isn’t something new. It has actually been around since 1994! Its job is to tell bots (crawlers) which parts of your site they can visit and which parts they should stay away from. If you do not have a robots.txt file, bots will treat everything like it is allowed.
Think of robots.txt as a recommendation rather than a rule. There's no guarantee that bots will obey your robots.txt configuration. Aggressive bots commonly ignore robots.txt entirely.
Security Note: Don't use robots.txt to protect private information. Sensitive content needs real security measures, such as password protection or restricted access.
These are the types of things you’ll commonly see in a robots.txt file
User-agent: which bot the rules apply to (* means all bots).
Disallow: paths the bot shouldn't crawl (for example /admin).
Allow: exceptions inside a blocked path, such as opening up one folder inside an area that's otherwise off-limits (for example /resources/guides/ lets a bot get to the guides even if /resources is disallowed)
Sitemap: the full URL to the XML sitemap.
Some additional things you may see
Wildcards: * matches anything, and $ marks the end of a URL (for example, Disallow: /*.pdf$ blocks all PDFs)
Comments: lines starting with #. These are often meant for humans and completely ignored by bots
A robots.txt file is a list of rule groups. Each group includes up to three items. It is more common to just include the User-agent and the Disallow items than having all three items.
The bots it is talking to (User-agent)
What they can access (Allow)
What they should stay away from (Disallow)
User-agent: Googlebot
Disallow: /
User-agent: Googlebot
User-agent: AdsBot-Google
Disallow: /
User-agent: *
Disallow: /
Example 3 is usually a bad idea. It blocks every bot, including the ones that help your site show up in search.
AI has added a new layer to robots.txt. AI companies use many different bots for different types of jobs. Some collect content to train AI models, and others fetch pages to answer people's questions in AI search tools. That means you can block some AI bots without blocking all of them.
If you block every AI bot, your site won't appear in AI search results, so people using those tools may never find you.
It depends on your content. If your content is your product, like paid research, articles or courses, blocking training bots helps protect it. If your site mainly markets your organization, being included in AI training data can help AI tools describe you accurately.
Keep in mind that AI moves fast and these bots change often. If you block specific AI bots, review your file regularly.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /
Robots.txt is small, but it can have a big impact. Blocking the wrong bots can affect how your organization appears in both traditional search and AI search, so think carefully before blocking bots.
IAAP CPWA (Certified Professional in Web Accessibility) and 508 Trusted Tester who's spoken at conferences from South Dakota to international stages. With a Master's in Information Systems and Business Administration, Kaitlyn expertly handles everything from CMS builds to front-end development—all while ensuring universal access. Click Rain's go-to accessibility guru.
Read more »
Monthly digital marketing insights sent straight to your inbox, on us.