LLMS.txt, Robots.txt and Schema: What Does Each Control for AI Visibility?

robots.txt, llms.txt and schema do not solve the same problem:

• robots.txt gives named crawlers instructions about which URLs they may request.
• llms.txt is an emerging Markdown proposal intended to present a curated map of important website resources to agents and language-model tools.
• Schema/structured data describes people, organizations, products, articles and other entities represented in visible page content through machine-readable properties.

None of these mechanisms guarantees indexing, ranking, a brand mention, a citation or a conversion. Choose a tool by identifying the technical problem and checking whether the target platform officially supports the proposed solution.

Separate discovery, crawling, indexing and citation

Many technical GEO mistakes begin by treating different stages as one event:

1. Discovery: A system learns that a URL exists.
2. Access/crawling: A crawler requests the URL and receives a response.
3. Processing: The system interprets HTML, text, links and supported markup.
4. Indexing or preparation: The platform may store or prepare content according to its own policies.
5. Retrieval/grounding: Relevant sources may be retrieved for a user question.
6. Answer and citation: The system may use the content, mention the brand or show a source link.
7. Visit and conversion: A person may follow the link and complete an important action.

robots.txt primarily affects crawler access. Schema can make visible information more explicit during processing. llms.txt aims to provide a curated content map to systems that support it. Later stages depend on platform systems, relevance, quality assessment and the user’s query.

What does robots.txt do?

robots.txt is a public text file served from the root of a host. It declares crawler rules by user agent.

Google’s robots.txt guide describes the file as a way to manage crawler access and request traffic. It also warns that robots.txt should not be used to keep an HTML page out of Google Search. A blocked URL may still appear if Google discovers it through links elsewhere.

Appropriate uses

• Managing crawl load across unimportant or repetitive URL spaces
• Defining policy for a named crawler
• Separating search crawling from model-development or training crawling
• Declaring sitemap locations for supporting crawlers

What robots.txt does not do

• Provide authentication or security
• Protect confidential information
• Prevent indexing in every case
• Force every crawler to comply
• Guarantee that a page will be used or cited in an AI answer

Protect sensitive material with authentication, authorization or appropriate network controls—not a public robots file.

AI crawlers do not all serve the same purpose

“Allow AI bots” is not an actionable policy. Search, model development and user-initiated access may use different user agents.

OpenAI’s official crawler documentation distinguishes:

• OAI-SearchBot: used to surface websites in ChatGPT search features
• GPTBot: associated with content that may be crawled for developing and training generative AI foundation models
• ChatGPT-User: may visit a page in response to a user action and is not used for automatic web crawling

OpenAI states that OAI-SearchBot and GPTBot preferences can be managed independently. A company should therefore answer separate questions:

1. Do we want eligibility for ChatGPT search visibility?
2. What is our policy for model-development or training use?
3. How should user-initiated visits be handled technically and securely?
4. Does the CDN or WAF validate official bots using published IP ranges as well as the user-agent string?

A robots allowance may not be enough if the security layer still blocks the request. Conversely, a user-agent string can be imitated, so it should not be trusted as a basis for privileged access.

What do robots meta and X-Robots-Tag do?

While robots.txt addresses crawler access, a robots meta tag or X-Robots-Tag can provide supported search engines with page- or response-level indexing and presentation directives.

Examples include:

• noindex: do not show the page in search results
• nosnippet: do not show a text snippet
• max-snippet: limit the length of a text snippet
• noimageindex: do not index images on the page

Google’s robots meta documentation explains meta tags for HTML and the X-Robots-Tag HTTP response header for non-HTML resources such as PDFs. It also states that crawlers can only read and follow these controls when they are allowed to access the page.

That order matters. If a page is blocked in robots.txt, a crawler may not see the noindex directive inside it. Blocking crawling and adding noindex at the same time may therefore fail to produce the expected result.

Support and effects differ by platform. Verify the current official documentation for each target system.

What is llms.txt?

llms.txt is a proposal for helping agents find important, preferably clean and Markdown-friendly website resources. A file may contain a project or site name, a short description and grouped links to selected documents.

The current llms.txt v2 proposal presents it as a Markdown format intended to help agents use website information. It may be useful for extensive technical documentation where an agent needs a curated route to an API guide, getting-started page, policy or reference section.

What can safely be said about llms.txt?

• It is an emerging proposal used by some sites and documentation systems.
• It can function as a human- and machine-readable curated resource map.
• It does not directly replace robots.txt, an XML sitemap or structured data.
• Support and actual use must be verified by platform.
• Its maintenance owner and review process should be defined.

OpenAI’s developer documentation currently links to an llms.txt documentation index. That is a real documentation use case; it does not establish llms.txt as a requirement for ranking in ChatGPT Search.

What llms.txt does not guarantee

• Every AI crawler will read it
• Listed pages will be indexed
• The brand will appear in answers
• Its links will be cited
• Traffic, rankings or conversions will increase

Google’s 2026 generative AI optimization guide explicitly says Google Search does not use llms.txt. Google may discover or index many file types, but that does not give the file a special role. For Google Search, maintaining an llms.txt file neither improves nor harms visibility or rankings.

This statement is specific to Google Search. Another agent or documentation tool may adopt the proposal.

How should llms.txt be maintained if you use it?

Before publishing the file, document:

1. Use case: Which agent or user task should it support?
2. Support evidence: Does the target system officially support or use it?
3. Resource selection: Are only current, canonical and genuinely important pages listed?
4. Content consistency: Do descriptions match the visible pages?
5. Language structure: Are localized resources mapped to the correct locale URLs?
6. Privacy: Does the file reveal any path that should not be public?
7. Ownership: Who updates the file after a page move or scope change?
8. Validation: Does it return HTTP 200 with working links and appropriate content?

An outdated file can give agents an outdated map. Even when generated automatically, links and descriptions require human verification.

What do schema and structured data do?

Structured data, often expressed as JSON-LD using the Schema.org vocabulary, defines entities and their properties in a machine-readable form. An article can describe its headline, author, publisher and dates; an organization can describe its name and contact properties where appropriate.

The central rule is that structured data must describe accurate information visible on the page. Do not add services, reviews, prices, authors or outcomes that users cannot see.

Google’s AI features documentation says AI Overviews and AI Mode do not require special Schema.org markup. Existing structured-data practices can continue to support established Search features, but the markup must match visible text.

What schema can do

• Describe visible entities and relationships to supporting systems
• Contribute to technical eligibility for supported search-result features
• Present author, publisher, date and content-type information consistently
• Make organization, product or article information easier to parse

What schema cannot do

• Replace visible content
• Create content quality or first-party evidence
• Guarantee AI visibility through a special “GEO schema”
• Make false organization or service information trustworthy
• Guarantee display in a supported feature

Markup may be syntactically valid while remaining editorially false. Technical validation and factual review are separate checks.

Where do XML sitemaps and canonical tags fit?

XML sitemap

A sitemap helps search engines discover important, new or updated URLs. It is not an alternative to access policy, an agent content map or entity description.

Google’s crawling and indexing FAQ says a sitemap can help Google learn about a site but does not guarantee indexing or ranking.

Canonical

A canonical signal indicates the preferred version among similar or duplicate URLs. It is not a redirect, crawler-access policy or AI-bot control.

For multilingual pages, verify canonical, hreflang and the visible language switch separately. Pointing an English page’s canonical to the Turkish version can weaken the independent eligibility of the localized page.

Technical GEO comparison table

Which mechanism should you use?

“A named bot should not crawl this area”

Verify the official user agent and IP method. Manage robots.txt and the security layer together. Use authentication for confidential content.

“This page should not be indexed”

Use the target search engine’s supported noindex meta or X-Robots-Tag method and make sure the crawler can read it. Do not rely only on a robots.txt block.

“An agent should find our important documentation more easily”

If the target system supports llms.txt or Markdown alternatives, consider a curated map. Without support evidence, prioritise canonical HTML, internal links, clean sitemaps and clear content structure.

“We want to define the organization, author or product more clearly”

Use an appropriate Schema.org type and the target platform’s guidance. Match structured data to visible information and test it.

“We want search engines to discover new URLs”

Use a clean XML sitemap, contextual internal links and working HTTP responses. A sitemap is not an indexing guarantee.

For the broader search foundation, see our guide to effective SEO techniques.

Implementation checklist

1. Is the problem defined in one sentence?
2. Were the target platform and user agent verified in official documentation?
3. Are crawling, indexing, training use and search visibility separate decisions?
4. Was the change tested in a limited scope or test environment?
5. Were critical URLs checked for accidental blocks?
6. Can the crawler read the meta or header directive?
7. Does schema match the visible page?
8. If llms.txt is used, are platform support and maintenance ownership documented?
9. Does the sitemap contain only working, canonical URLs intended for indexing?
10. Are the change date, owner and rollback method recorded?
11. Do server and CDN logs show the expected crawler behaviour?
12. Is every technical change described without a visibility or citation guarantee?

Assign ongoing ownership deliberately. Our comparison of website maintenance and SEO responsibilities helps separate routine file upkeep from visibility analysis.

Common mistakes

Using llms.txt instead of robots.txt

llms.txt is not an access-control standard. Use the target platform’s supported crawler and security controls.

Allowing or blocking every AI bot with one rule

Search, training and user-initiated access may serve different purposes. Define company policy for each.

Trying to create privacy with robots.txt

The file is public, and malicious crawlers may ignore it. Protect private content with authorization.

Adding invisible claims to schema

Markup should describe accurate, visible content—not hidden marketing claims.

Publishing a technical file without an owner

Broken links, moved pages and obsolete service descriptions create an inaccurate map. Every technical file needs a review date.