GUIDE · TECHNICAL ACCESS
Can AI search
read your website?
Before worrying about content quality or AI-specific optimisation, check whether AI search platforms can actually access your pages. A locked door does not need better signage.
THE FIRST CHECK
Is your content
publicly crawlable?
Public service information should be accessible to the systems you want to use it. Login requirements, crawler blocks, indexing directives, server errors and rendering problems can restrict discovery or retrieval. Check the specific engine and URL rather than assuming all AI products follow identical rules. Keep confidential content private.
For Google AI features in Search, a supporting page must be indexed and eligible for a search snippet. ChatGPT also publishes its own OAI-SearchBot controls; it should not be described as simply a Bing interface. Access is a prerequisite to investigate, not a promise of a recommendation.
THE CHECKLIST
Five things to check.
- 01
Robots.txt. Open
yourdomain.com/robots.txtin a browser. Look forDisallow: /orDisallowrules that block your important pages. Some website builders add broad blocks by default. A rule blockingGooglebotlimits Google crawling. A blocked URL may still be listed without its content; a robots.txt block is not a reliable removal instruction. - 02
Noindex tags. Right-click your service page, view the page source, and search for
noindex. A<meta name="robots" content="noindex">tag tells search engines not to include the page in their index. This is sometimes left on accidentally after a staging or development phase. - 03
Login or paywall. For public service pages, verify the important facts are visible without signing in. Private account content should stay private. Paywalled content has platform-specific rules, so do not assume every paywall or registration flow has the same search behaviour.
- 04
JavaScript rendering. Some websites load their content entirely through JavaScript after the initial page loads. Google can render JavaScript, but a broken script or failed resource can hide important text. Compare the initial HTML with the rendered page and, for Google, inspect its rendered HTML in URL Inspection. Disabling JavaScript can identify dependencies; it is not a definitive simulation of every crawler.
- 05
HTTP status codes. Your service pages should return a 200 status code. Investigate unexpected 404 (not found) or 500 (server error) responses and redirect loops. A valid redirect is not inherently a defect: check that it reaches the correct working destination. Use a free tool or check the Network tab in your browser's developer tools.
TRAINING CRAWLERS VS SEARCH CRAWLERS
They are different bots
with different purposes.
Crawler and product controls have different purposes. Identify the exact control before changing it:
- Training crawlers collect web content to train AI models. OpenAI identifies GPTBot as a training crawler. Its search control is separate. Google-Extended is a product token controlling use for training and grounding in certain Gemini and Vertex AI products; it is not simply a second name for Googlebot.
- Search crawlers index web content to serve in search results and AI search features. Examples include Googlebot, OAI-SearchBot (OpenAI's search feature) and PerplexityBot. These controls affect access for their respective services; allowing a crawler does not ensure that an answer will name or cite you.
OpenAI documents independent GPTBot and OAI-SearchBot settings. Google says Google-Extended does not control inclusion in Google Search or its AI features, although it can affect the separate Gemini and Vertex uses it covers. Avoid treating every consumer AI product as the same search system.
Do not weaken your website security to serve training bots. If you have login protection, rate limiting or bot filtering for legitimate security reasons, keep them. Removing security controls to let training crawlers in is not necessary for AI search visibility and can expose your site to abuse.
NO SPECIAL AI FILES NEEDED
Standard practices work.
For Google AI features in Search, Google says no special AI text file, schema or additional optimisation is required. Existing SEO practices remain relevant. That guidance is specific to Google; check other platforms' own documentation for their requirements. Google's AI features documentation.
If a provider claims you need a special "AI-ready" file, proprietary schema markup or a paid plugin to appear in AI answers, verify that claim against the platform's own documentation before paying for it.
Primary references: OpenAI crawler controls and Google crawler and product controls.
What does matter: clear, crawlable pages with accurate business information, consistent entity details (name, address, services) across your website and Google Business Profile, and content that directly answers the questions buyers ask.
WHAT TO DO
Fix access first.
- 01
Run the five checks above on your main service pages.
- 02
If you find robots.txt blocks, noindex tags or login walls on pages that should be public, fix them.
- 03
Use Google Search Console URL Inspection to check the page and distinguish live-test eligibility, a submitted recrawl request and actual indexed status. None guarantees the next stage.
- 04
Then focus on content quality: are your service pages answering the questions buyers actually ask?
Ask for URL-specific findings and the evidence behind an access diagnosis. Where Google account data is needed, agree the access first. See what the AI visibility audit covers ↗
COMMON QUESTIONS
Access and AI search.
Do I need special schema or files for Google AI Search?
Google says its AI features in Search require no special AI file or schema. Apply its standard search requirements and check other platforms separately. Eligibility does not guarantee inclusion.
Should I unblock AI training crawlers to improve visibility?
Do not change controls blindly. OpenAI separates GPTBot training and OAI-SearchBot search settings. Google-Extended covers separate Gemini and Vertex uses and does not control Google Search inclusion. Keep legitimate security protections and verify the exact product control.
RELATED
From a guide to the work.
What the Tendou US subscription does ↗ · Pricing ↗ · Choosing between a tool, agency and managed plan ↗ · AI citations vs referral traffic ↗ · Give a provider access while keeping ownership ↗ · All guides ↗
Published October 18, 2026 by Brayden Tan, Tendou US. This guide is general information, not a forecast. No ranking, AI citation or enquiry volume is guaranteed.