AI search technical SEO checklist: access, index and measurement

The short answer: there is no special AI file, the base is the same

To appear as a supporting link in Google's AI Overviews and AI Mode, a page has to be indexed in Google Search and eligible to be shown with a snippet. Google's AI features document says there are no additional technical requirements beyond that, and that you do not need new machine-readable files, AI text files or special markup. Most of this checklist is therefore the technical SEO you already know: accessible, indexable, consistent and visible content.

This article is not a model comparison or an "AI ranking" guide. Nobody can guarantee which URL will be selected in which answer; Google's own guide says indexing and serving are not guaranteed. There are two things we can do: remove technical blockers one by one, and measure what can be measured. The sections below separate the request, crawl, index eligibility and selection stages and give a check and a working example for each. Last checked: 25 October 2026. Search Console features and Google's guides can change, so keep this date in mind.

The examples only use synthetic HTTPS addresses such as example.com. I did not measure any real site's ranking, traffic or citation rate. The outputs shown come from small Python examples that run offline.

Request, crawl, index, selection: four separate gates

Treating AI search as one black box sends you looking for the problem in the wrong place. Four things need to be kept apart. A request is a client sending an HTTP request to your URL and you returning a response. Crawling is a crawler reaching that URL according to your robots.txt rules. Index eligibility is the page being indexed and eligible to be shown with a snippet. Selection is the system deciding which links to show as support for a particular query.

In practice this means "AI does not show me" is not a diagnosis by itself. If the page returns a 503 the problem is at the first gate, if robots.txt blocks the crawler it is at the second, and if there is a noindex it is at the third. If all three gates are open and the page is still not selected, there may be no technical fix; content, query and competition come into play. At that point technical SEO has done its job and managing expectations is what is left.

Access: robots.txt, HTTP status and crawler tokens

robots.txt is a file that tells crawlers which URLs they can access, and according to Google's documentation its main purpose is to manage crawl traffic; it is not a way to hide a page from Google. A URL you block can still show up in results without a description if other sites link to it. RFC 9309 makes the same point more bluntly: these rules are not a form of access authorization. You protect private content with a password or authorization, not with robots.txt.

HTTP status is part of this gate too. According to Google's documentation, content from URLs that return 4xx is not used and those URLs are not indexed; 5xx and 429 responses make crawlers slow down temporarily, and persistent server errors can lead to removal from the index. A 200 response is not a guarantee of indexing; it only lets the content be processed. The status code of robots.txt itself matters as well: under RFC 9309 a 4xx response makes the file "unavailable" and the crawler may access resources, while a 5xx response requires assuming a complete disallow. This paragraph describes the RFC behavior only; check Google's own implementation details separately.

The real confusion starts with tokens. "Block AI bots" covers two different things: access for search and citation, and a preference about model training. The table below separates the tokens whose documentation I read.

The robots.txt below shows this separation in one file: search access open, two training tokens closed, and only /private/ closed for everyone else. I produce the output with Python's urllib.robotparser module. That module is not Google's parser; the example is deliberately built from non-overlapping rules and only shows group selection.

01from urllib.robotparser import RobotFileParser02 03RULES = """User-agent: Googlebot04Allow: /05 06User-agent: OAI-SearchBot07Allow: /08 09User-agent: GPTBot10Disallow: /11 12User-agent: Google-Extended13Disallow: /14 15User-agent: *16Disallow: /private/17"""18 19parser = RobotFileParser()20parser.parse(RULES.splitlines())21for agent in ("Googlebot", "OAI-SearchBot", "GPTBot", "Google-Extended", "ExampleBot"):22    for path in ("/blog/", "/private/"):23        verdict = "ALLOW" if parser.can_fetch(agent, "https://example.com" + path) else "BLOCK"24        print(f"{agent:<16}{path:<10}{verdict}")25 

01Googlebot       /blog/    ALLOW02Googlebot       /private/ ALLOW03OAI-SearchBot   /blog/    ALLOW04OAI-SearchBot   /private/ ALLOW05GPTBot          /blog/    BLOCK06GPTBot          /private/ BLOCK07Google-Extended /blog/    BLOCK08Google-Extended /private/ BLOCK09ExampleBot      /blog/    ALLOW10ExampleBot      /private/ BLOCK

Two things stand out in the output. First, GPTBot and Google-Extended are blocked on every path while Googlebot and OAI-SearchBot are open: the training preference and search access are managed separately. Second, even the /private/ path looks allowed for Googlebot. Under RFC 9309 a crawler uses the group that matches its own token and only looks at the * group when no group matches; groups are not merged. A Disallow you add to the * group is not automatically applied to a crawler that has its own group. This is an easy detail to miss in robots.txt files.

To draft rules, the robots.txt generator speeds things up, and it also lets you test paths locally. Read the generated file against your own token list before publishing it. To see which AI bots are allowed once a site's robots.txt, llms.txt and HTML are pasted, there is the AI crawler checker; it is a checking aid and gives no guarantee of indexing or citation.

Index eligibility: noindex, canonical and rendered HTML

After the first three stages comes index eligibility. The diagram below asks five gates in order, and each "No" exit leads to a different class of problem.

According to Google's documentation, noindex tells search not to show the page in results. It has a catch, though: these settings are only read if the crawler can access the page. If you block the page with robots.txt, the noindex inside it is never seen and is ignored. If you really want to keep a page out of the index you need to allow crawling and use noindex; robots.txt does not replace it. For non-HTML files the same rules can be sent with the X-Robots-Tag HTTP header.

01<meta name="robots" content="noindex">02<link rel="canonical" href="https://example.com/ai-search-checklist/">

For canonicals, Google's documentation describes the methods as "signals": a redirect is a strong signal, rel="canonical" is a strong signal, and listing a URL in a sitemap is a weak one; the methods can reinforce each other. The advice is clear: use absolute URLs, give the canonical page a canonical pointing to itself, do not point to different URLs with different techniques for the same page, and do not use robots.txt for canonicalization. The document also does not recommend noindex to prevent canonical selection within a single site.

Search Console's URL Inspection tool shows two fields separately: the canonical you declared and the canonical Google selected. Google takes the first into account, but there is no guarantee it will choose your preference. So instead of saying "I added a canonical, done", compare the two fields. In the same tool, the indexed version is not a live test; the live test only confirms whether Google-InspectionTool can access the page and does not mean the page will be indexed.

The last gate is visible text in the rendered HTML. According to Google's JavaScript documentation, pages with a 200 response go to the rendering queue, indexing uses the rendered HTML, and content that is not visible in the rendered HTML cannot be indexed. In a client-side single-page app it can be hard to return a meaningful status code for error pages; the document suggests redirecting to a URL that returns 404 or adding noindex to the error page to avoid soft 404s.

Visible text, an evidence paragraph and structured data

No special structured data is needed for AI features; Google says so explicitly. Structured data is not required for generative AI, but it is still recommended for general SEO. If you write it, the rule is simple: do not mark up content readers cannot see, and make the markup a true representation of the page content. Passing the Rich Results Test is not a guarantee of appearing either.

That is why a good evidence paragraph matters. You state the claim directly in the body text, with its date and basis; structured data only repeats the same information in a machine-readable form. Google also says there is no need to break content into small pieces or to follow a particular page length, and that you do not need to write differently for AI. Instead of an "AI rewrite", writing a clear, sourced and current paragraph for the reader makes more sense.

The small script below checks this consistency on a synthetic page: is the canonical absolute, does it match the JSON-LD url, is datePublished present in a visible time element, and do the headline and description appear in the visible text. A mismatch was left in on purpose: the JSON-LD description mentions a benchmark table that is not on the page.

01import json02from html.parser import HTMLParser03 04PAGE = """<!doctype html>05<html lang="en"><head>06<title>AI search technical SEO checklist</title>07<meta name="robots" content="max-snippet:-1">08<link rel="canonical" href="https://example.com/ai-search-checklist/">09<script type="application/ld+json">10{"@context": "https://schema.org", "@type": "TechArticle",11 "headline": "AI search technical SEO checklist",12 "url": "https://example.com/ai-search-checklist/",13 "datePublished": "2026-10-30",14 "description": "Includes a benchmark table of AI answer citations."}15</script></head><body><main>16<h1>AI search technical SEO checklist</h1>17<p><time datetime="2026-10-30">30 October 2026</time></p>18<p>A page can be crawled and indexed and still not be selected.</p>19</main></body></html>"""20 21 22class Audit(HTMLParser):23    def __init__(self):24        super().__init__()25        self.in_body = self.in_ld = False26        self.text, self.ld, self.dates = [], [], []27        self.canonical = self.robots = None28 29    def handle_starttag(self, tag, attrs):30        a = dict(attrs)31        if tag == "body":32            self.in_body = True33        elif tag == "link" and a.get("rel") == "canonical":34            self.canonical = a.get("href")35        elif tag == "meta" and a.get("name") == "robots":36            self.robots = a.get("content")37        elif tag == "time":38            self.dates.append(a.get("datetime"))39        elif tag == "script" and a.get("type") == "application/ld+json":40            self.in_ld = True41 42    def handle_endtag(self, tag):43        if tag == "script":44            self.in_ld = False45 46    def handle_data(self, data):47        if self.in_ld:48            self.ld.append(data)49        elif self.in_body:50            self.text.append(data)51 52 53audit = Audit()54audit.feed(PAGE)55data = json.loads("".join(audit.ld))56visible = " ".join(" ".join(audit.text).split())57print("robots_meta", audit.robots)58print("canonical_is_absolute_https", audit.canonical.startswith("https://"))59print("canonical_matches_jsonld_url", audit.canonical == data["url"])60print("date_in_time_element", data["datePublished"] in audit.dates)61print("headline_in_visible_text", data["headline"] in visible)62print("description_in_visible_text", data["description"] in visible)63 

01robots_meta max-snippet:-102canonical_is_absolute_https True03canonical_matches_jsonld_url True04date_in_time_element True05headline_in_visible_text True06description_in_visible_text False

The last line printed False. Do not read this as proof of a policy violation: exact text matching is a heuristic, and summary sentences may be worded differently from the visible text. The script's job is to list candidates for a human to look at. The fix is either to add the sentence to the page visibly or to remove it from the JSON-LD. To produce JSON-LD, the schema generator can be used; comparing its output with the page's visible content is still your job.

Internal links and URL hygiene

A crawler usually follows a link to find a page. According to Google's JavaScript documentation, link discovery needs a elements with an href attribute. So important pages should be reached from within the site, with meaningful link text and a real a href. Using the History API instead of fragment-based client-side routing is recommended in the same document.

URLs themselves can be a source of noise. ?utm_source=... variants that lead to the same content, case differences, or versions with and without a trailing slash create duplicate URLs; the canonical methods in Google's documentation exist precisely to consolidate them. If a URL contains Turkish characters or spaces, make sure the percent-encoding is correct; the URL encode and decode tool helps you try that.

External links are a separate job. The free backlink sources article covers them platform by platform, and it also makes no promise of earned links or ranking. The checklist in this article looks at the site's own pages, not at external links.

Snippet controls, llms.txt and product controls

Google's AI features document lists nosnippet, data-nosnippet, max-snippet and noindex as controls to limit what is shown. According to the robots meta documentation, nosnippet covers all forms of search results including AI Overviews and AI Mode and also prevents the content from being used as direct input for those features. max-snippet limits the same input (a value of 0 is equivalent to nosnippet, -1 leaves the length to Google). data-nosnippet keeps specific parts of span, div and section elements out of the snippet.

All of these are a trade-off: you gain control over what is shown, but you may reduce visibility. Do not decide without observing the effect on your own page; this article does not say which is right.

llms.txt is a separate topic. llmstxt.org describes it as "a proposal": an open draft published by Jeremy Howard on 3 September 2024, built on a Markdown file. The page does not claim it belongs to any standards body and does not say search engines use it. Google, in its AI optimization guide, says you do not need new machine-readable files, AI text files or Markdown, that Google Search ignores them, and that they will neither harm nor help your site's visibility or rankings. So llms.txt is not a lever for Google AI visibility. If another agent or tool deliberately uses the file, read that tool's own documentation; the llms.txt generator helps you produce the file in that case, but it offers no index or citation guarantee.

Search Console also has a property setting called "Search generative AI control". According to its help page, the setting manages whether the site is included in generative AI features in AI Overviews, AI Mode and Discover; the options are Include, Exclude and Inherit from parent. The same page says the setting is not a ranking or inclusion signal for other parts of Search and does not affect AI training. It is a product control, not a robots.txt rule; read the help page before using it. This article does not recommend using it.

Measurement: what you can and cannot see

Measurement is the most misunderstood part. Google's AI features document says traffic from these features is included in overall search traffic and is counted under the "Web" search type in the Search Console Performance report; the document mentions no separate filter. In addition, Search Console has a report called "Generative AI performance report (Search)". According to its help page the report covers AI Overviews and AI Mode and gives impressions only. Its dimensions are Pages, Countries, Dates and Devices, and the search type filter offers "Web: text-based" and "Web: multimodal". The page says that as of 31 August 2026 the report has rolled out to all sites; if there are not enough impressions, the report may not appear.

What the report does not list matters as much as what it does. There are no clicks, CTR or average position; query is not listed among the dimensions; Search Labs experiments are not included and the Performance report's 1,000-row limit applies. The help page says the report includes data from the "Web" search type but does not explain whether the two reports double count, so do not add the numbers of the two reports together.

Google's guide also says no third-party tool has access to Google's internal ranking and AI systems. If a tool gives you an "AI visibility score", do not read it as Google data; be the one who asks how the score is calculated. On the index side URL Inspection is useful but limited: a request for indexing does not guarantee anything either. Do not forget your server logs: a log shows the request and your response, not index or citation status.

The observation plan can stay simple. Before the change, record the target page's Web data in the Performance report with a page filter, and its impressions in the Generative AI performance report if you have it. After changing the page, look again over the same periods and check in the Performance report which queries brought traffic to that page. Do not infer causation between a change and an increase: seasonality, competition and Google's own changes can act in the same period.

Limitations and a working checklist

The limits of this article are explicit. I did not measure any real site; the sources are only Google's and OpenAI's own documentation, RFC 9309 and llmstxt.org. Search Console features and Google's guides change quickly; the article describes the state on 25 October 2026 and does not cover later changes. I did not verify the behavior of AI search products other than ChatGPT, so I did not describe them. No item is a guarantee of ranking, indexing or citation.

• Important URLs return HTTP 200; error pages return 4xx, and there are no soft 404s that show an error with a 200.
• robots.txt does not block content for Googlebot; the training tokens (Google-Extended, GPTBot) are managed with separate decisions.
• noindex appears only on pages you do not want indexed, and those pages are not blocked by robots.txt.
• Every page has an absolute canonical pointing to itself; redirects, the sitemap and the canonical back the same URL.
• Important text and links are visible in the rendered HTML; links are a elements with an href attribute.
• Structured data describes only visible content; passing the Rich Results Test is not treated as a guarantee.
• Snippet controls (nosnippet, max-snippet, data-nosnippet) are used by deliberate decision or not at all.
• For measurement the Performance report (Web), the Generative AI performance report and URL Inspection are read together; the two reports' numbers are not added.
• llms.txt is not presented as a tool for Google visibility.