SEO & content

Robots.txt, Sitemaps and AI Crawlers

Separate search, training and user-request access. Review robots, sitemaps, noindex and llms.txt through a scoped decision and verification plan.

Prepared with AI assistance. Source links appear in the relevant sections.

Designer working on website interface sketches

Robots.txt tells participating crawlers which addresses they may visit; a sitemap lists addresses you want discovered. For AI services, distinguish search crawling, potential model training and visits requested by a user. Allowing access does not guarantee a recommendation, and a robots rule cannot replace access protection for private information. Start with the purpose of each page before choosing a rule.

If these decisions are unclear, our SEO and organic visibility service can help define the problem and an appropriate review. Changing every bot rule because you want AI visibility is not a useful starting scope.

Agree the decision before editing a file

A business can want its public service pages discovered in search while taking a different position on collection for model training. Those are separate choices. A bot’s familiar name is insufficient reason to allow or restrict it. First explain the business intention in language the content owner and implementer both understand.

List the content groups involved. Public service descriptions, guides and project explanations belong in a different discussion from customer accounts, confidential proposals and application records. A page without a navigation link is not necessarily private. Genuine access protection needs its own review; this guide does not instruct you to change authentication or expose restricted content.

For each public group, record whether the decision concerns discovery, a user-requested visit or a training preference. Name the person responsible for approving that decision. A note saying “needed for GEO” will not help someone maintain the site later. A record explaining the purpose, affected pages and expected behaviour is much more useful.

Evidence to record before a decision
ObservationEvidenceWhat it does not establish
robots.txtCrawler access preferencesGuaranteed exclusion from indexing
noindexIndexing instruction that can be readPassword protection
SitemapA list of canonical indexable URLsA promise to index
llms.txtA proposed information routeRequired AI search optimisation

Separate crawler roles

OpenAI distinguishes OAI-SearchBot for search, GPTBot for potential model-training collection and ChatGPT-User for certain user-requested visits. Search and training preferences are independent. Robots rules may not apply to user-initiated visits in every case. Check the official crawler documentation before selecting the relevant agent.

Anthropic describes ClaudeBot in relation to model-development collection, Claude-SearchBot for search and Claude-User for user-requested retrieval. Restricting one should not be assumed to have the same effect on every role. Use Anthropic’s crawler guidance to check the current distinction.

PerplexityBot supports search discovery and links, rather than foundation-model training. Perplexity-User handles user-requested visits and generally ignores robots rules. The official Perplexity documentation explains the difference. A line in robots.txt should never be described as complete control over every type of access.

These descriptions were checked on 8 November 2026. Revisit the operator’s documentation when implementing a policy later. Names, roles and verification arrangements can change. Moving the year in an article’s title does not establish what an unreleased service will do.

Robots, noindex and a sitemap answer different questions

Robots controls crawling preferences. Noindex communicates an indexing instruction to a system that supports it; that system needs access to read it. A sitemap helps discovery. These are not interchangeable ways to hide a page. Google’s introduction explains that a disallowed address can still appear in results.

Consider a public campaign page that should remain usable by visitors but stay out of organic results. A crawling restriction alone may not express the intended outcome. Conversely, blocking a service page that the business wants discovered can conflict with its visibility objective. State the intended behaviour before deciding which mechanism belongs in the implementation.

For a sitemap review, consider working, preferred addresses. Investigate redirected locations, incorrect language versions and discontinued campaigns rather than retaining them simply because they are already listed. Inclusion is not a promise of indexing or recommendation. People also need useful navigation: our Google and AI content guide explains why a page’s purpose and connections matter alongside technical access.

Define the scope when several addresses are involved

A store, company website and help centre may use different addresses. List the actual domains and subdomains before saying the site has been checked. Record both the starting address and final destination when a visitor is redirected. A correct result for one section should not automatically become a pass for another.

If a guide sits on the company site while its images are hosted elsewhere, include the relevant public resources in the agreed review. State what was excluded. Editing another project's rules or assuming its successful check applies here makes the handover unreliable. Keep the authority and evidence tied to the addresses actually reviewed.

Review sequence for this task
  1. Identify the bot and purpose
  2. Read the relevant policy
  3. Review robots and access together
  4. Check a small sample

Read the existing groups together

A user-agent line identifies the crawler group addressed by the following rules. Review specific groups and the general group before appending a new block. The Robots Exclusion Protocol defines matching groups and paths. Treat the effective rules for the intended crawler as the subject of the check, rather than relying on a quick visual scan of the file.

Imagine a draft that adds “User-agent: OAI-SearchBot” followed only by “Allow: /”. That does not establish that restrictions written elsewhere were reviewed for this group. The implementer must check its effective rules. These two lines are an illustration, not a configuration to copy onto your site. Avoid templates that accidentally open panel or API paths.

Prepare a short set of representative paths: the home page, a service, a guide, a panel route and a thank-you page. Write the expected preference beside each. That makes the intention reviewable by someone who did not author the change. For Google-specific interpretation, consult its robots specification guidance. Do not assume every provider supports every extension in the same way.

An allowed rule does not establish successful retrieval

A page opening in your own browser does not prove that another client receives the same content. Hosting and security layers can return different responses. Keep the public crawling preference separate from evidence about an actual request. The operators’ documentation points to their current verification resources; a request merely carrying a bot name is not sufficient proof of identity.

Do not turn off protections across the site to investigate visibility. Ask the authorised technical team to examine a specific failure using the affected address, time and observed response. Remove credentials, session information and customer data from evidence before sharing it. This guide does not recommend bypassing a challenge or granting a new security privilege.

Repeated heavy requests are also a poor diagnostic method. Start with a small representative sample. If 429 or 503 responses increase, stop and record the condition. A rushed visibility review should not make the experience worse for normal visitors. Without adequate evidence, leave the cause unresolved rather than naming either traffic or security rules as the confirmed fault.

An illustrative policy decision

Suppose a consultancy wants public service and resource pages discovered, while choosing to restrict training-related collection. Begin with three separate decisions: search, training and user-requested access. Do not start by applying one allow or block list to every familiar agent name. This example describes a planning method, not a real customer outcome.

Next, map those decisions to the current operator documentation. Review the affected public pages and existing restricted-path rules. Keep the proposed change small enough to explain. “The bots are allowed, so the company will now be recommended” cannot be the conclusion. The implemented result is an access preference, with visibility still to be observed separately.

The review record should show expected behaviour before and after. If a public guide becomes unavailable, correct the implementation. If a private route’s treatment becomes uncertain, do not mark the work complete. Another person should be able to see which decision was made, why the affected rule was chosen and how the result was checked.

For a similar concern, call to discuss crawler access or send your website and affected public page on WhatsApp. An introductory review does not require your administrator password.

Ask for evidence at handover

A useful report does more than say “robots fixed”. It identifies the decision owner, affected addresses, original problem, purpose of the change and verification outcome. Preserving another person’s concurrent work also matters. Replacing a current file with an old local copy is not a harmless consequence of a small SEO task.

Use this order for the review:

  • Does the public robots address return the intended file?
  • Does the effective preference match the decision for each sampled crawler and path?
  • Do public service and guide addresses still show their correct content?
  • Do indexing instructions, canonical and sitemap support the same page purpose?
  • Did any restricted route or protection change unnecessarily?
  • Is there a way to reverse only this operation’s changes if it causes a fault?

Our questions to ask an SEO agency can also help clarify which checks belong in a proposal.

Separate a pre-publication test from a live check. A locally correct file may not yet be in use on the server. A live file being available does not show that a search system has reprocessed it. The completion record should identify the actual level checked, rather than present expected future behaviour as an observed result.

Also record the boundaries of the work. If only public pages were reviewed, say so. If a security team must examine request logs, that is a separate dependency. Do not report a successful crawler visit simply because your own inspection tool obtained a normal response. Each observation needs a clear subject and source of evidence.

Measure access, visibility and enquiries separately

A permitted request does not establish that a page was selected as a source. Source selection is different from a website visit or a sale. The work described in our Google and AI content guide concerns useful content and page purpose as well as access. Do not compress those separate needs into one technical score.

Use a recorded question and comparable conditions for answer observations. Record referral visits when they genuinely appear in Analytics. A phone or WhatsApp click is not a completed conversation, and an unverified form event should not become either a zero or a growth claim. A few visits are insufficient evidence for a strong commercial conclusion.

An initial engagement can be a bounded review and priority list. Languages, subdomains, content types and publishing arrangements affect implementation effort. There is no reason to edit the same file every month to demonstrate activity. Revisit the relevant decision when a site section, operator behaviour or verified problem changes.

When might llms.txt be useful?

llms.txt is a proposal for giving AI tools a concise route through a website's information. It does not replace robots rules, a sitemap or the visible page. Publishing one does not guarantee that a search system will read it or recommend the business. Read the primary proposal.

Define the intended information task first. For a manufacturer, a short set of verified links can distinguish product specifications, delivery scope and maintenance documentation. Copying the entire website, including private material or adding instructions to recommend the business does not produce a useful information map.

Start with a small list of accurate, public links. Keep the map consistent with the actual pages when information changes. File availability and observed use by a particular tool are separate acceptance checks. Google explains that its AI search features do not require a special AI text file. Read Google's eligibility guidance.

Prepare a useful first request

Share the site address, important public pages and a short example of the concern. “We suspect an access issue on this service page” is more actionable than a general request to appear in ChatGPT. Keep a suspicion labelled as a suspicion; the first task is to establish whether the proposed fault exists.

Use the WeAreMedia contact form to describe that scope. We can review the current behaviour and distinguish technical implementation, content improvements and measurement. The aim is a clear account of what needs changing and how it will be verified, without promising rankings or recommendations controlled by another system.

Related guides

Apply this review to your website

Share your website and priority business goal. We can identify the missing information, technical checks and next step.