Even an excellent service page can remain hard to discover when automated systems cannot reliably access its public content. The issue may involve the site's technical architecture, HTTP responses, indexing directives or a deliberate restriction on certain crawlers.
This checklist helps you assess whether your publicly available business information is technically accessible. It does not promise that ChatGPT will cite your website or recommend your company.
1. Check the HTTP response
Open your most important service page in an ordinary browser. It should load without a login and generally return a successful 200 OK response. An old or deleted URL needs an appropriate redirect or an actual 404 status. Avoid a fake “not found” message served with a 200 response.
Have the website administrator confirm the server's response. Working in your own browser doesn't guarantee that remote crawlers can access the same content.
2. Inspect the HTML before JavaScript runs
If the key page content only appears after JavaScript executes in the browser, some automatic readers may not see it. Prefer having the main heading, service description, pricing context, actual area served and internal links in the initial server response.
Modern JavaScript is perfectly compatible with this approach. A dynamic user interface and server-delivered SEO content can coexist. Google's JavaScript SEO documentation explains why accessible HTML helps rendering and discovery.
3. Review robots.txt and indexing controls
The robots.txt file at the root of a domain can restrict which URLs automatic crawlers may fetch. A page-level noindex directive tells supporting search engines not to include that page in their index. These are different mechanisms.
Check that public service pages are not accidentally left behind a development-environment Disallow: / rule. Do not remove intentional access restrictions merely to chase visibility. Coordinate changes with the website owner or administrator.
For Google specifically, remember that a crawler blocked by robots.txt cannot read a noindex directive on that page. Blocking crawling is not a privacy measure and is not the same as requiring authentication.
4. Understand OAI-SearchBot and GPTBot
OpenAI's publisher guidance says OAI-SearchBot is relevant to whether public pages can be surfaced in ChatGPT search. GPTBot is a separate crawler concerning content that may be used to train AI models.
Keeping these settings separate matters. Neither bot setting is a magic shortcut to a recommendation. Refer to the provider's current documentation rather than assuming crawler rules will remain unchanged forever.
Official source: OpenAI Publishers and Developers FAQ.
Different agents, different purposes
Agent | Main purpose | Decision it relates to |
|---|---|---|
| Discovering web pages for ChatGPT search features | Do you want public pages eligible to be surfaced in ChatGPT Search? |
| Collecting material potentially used for model training | Do you permit this separate use of your content? |
| Certain web visits initiated by a ChatGPT user | Not a crawler for general ChatGPT Search visibility |
These distinctions follow OpenAI's crawler documentation. Search discovery and training access are independent decisions. Only allow public content that should be discoverable. Do not open customer records or staging environments to bots.
For a public site, one illustrative robots.txt configuration allowing the search bot while blocking the training bot is:
Merge this carefully with your existing rules; don't paste it blindly into production or a private preview. Firewall rules and bot protection may still block access. OpenAI publishes its search crawler's IP ranges and user-agent identifiers. Verify real accessibility through logs or controlled tests.
5. Create a useful sitemap and internal linking structure
An XML sitemap helps search engines discover important URLs, but it does not replace navigational links. Essential services should also be reachable through ordinary, descriptive internal links.
A link labelled “Bookkeeping services for companies” says more than a generic “Read more”. A sitemap should only list valid, public, canonical pages.
6. Set the correct page title and language
Give every important service page a clear H1, distinctive page title and helpful description. Finnish-language pages should use lang="fi". If there is an English-language version, give it a separate fully translated URL and use appropriate hreflang annotations to connect genuine equivalents.
Where a company operates only in specific areas, explain that visibly in the copy. A hidden keyword in source code is not a substitute for a useful, accurate service description.
7. Use structured data for real, visible facts
Structured data can help identify the type of company, service or article. It is not a guarantee of inclusion in AI-generated answers. Do not invent ratings, areas served or services that cannot be verified on the page.
Keep schema.org markup aligned with the actual copy and validate it. Correct information matters more than the number of schema properties.
8. Run an actual discovery check
Once the technical basics are correct, test several plausible customer questions. Leave the company name out if you are measuring category-based discovery. Open the cited sources and determine whether the AI is relying on your own services, reliable external sources or something unrelated.
Technical accessibility is only one part of AI visibility. Answers also vary by question, model, available source material and time.
Technical checklist at a glance
Public service pages return a valid HTTP 200 status.
The core service content is present in readable HTML.
Robots.txt, noindex and firewall controls have been checked.
Current OpenAI OAI-SearchBot rules have been reviewed.
Services, actual areas served and contact details are clear.
Internal links and the sitemap point to important pages.
Language, page metadata and structured data describe real content.
Measurements save actual questions and answers as evidence.
[Check your business's AI visibility for free](/fi/#testi)
Further reading
[AI optimisation for Finnish businesses](/fi/tekoalyoptimointi)
[How to appear in ChatGPT answers](/fi/oppaat/miten-nakya-chatgptssa)
AI Rank editorial
Practical insights into AI search and brand visibility.
Understand your visibility in AI answers.
Track mentions, compare competitors and explore the sources behind the answers.