A client rang recently and said: "I asked ChatGPT for construction companies in my area and we weren't on the list."

A year ago nobody asked me that. Now I hear it regularly, and it's a fair question — more people are starting their search by asking an assistant rather than typing a phrase into a search engine.

That changes one thing fundamentally. In classic search the goal was a high position, because a result was followed by a click. When an assistant answers, there often is no click at all — the person gets a finished answer and stops there. Position stops being the goal. Being cited in the answer becomes the goal.

This doesn't mean classic search work stopped mattering. It means a second layer has appeared, and while some of what you do for Google carries over exactly, some of it doesn't carry over at all.

Below I've split this into three parts: what is verifiable and genuinely works, what is uncertain, and what is being sold to you. Starting with what I think is the most common and most expensive mistake.

The most common mistake: you blocked the wrong bot

This is the heart of it, and most guides skip it.

The companies behind these assistants don't run one crawler. They run several, and each does something different. One collects data for training models. One builds the search index used when the assistant looks something up live. Another visits a page because a user pasted a specific link.

That distinction has direct financial consequences. Through 2023 and 2024 a wave of "block AI, it's stealing content" went through the industry, and a lot of companies added blocks to their `robots.txt` — often wholesale, against anything that sounded like AI. Objecting to models being trained on your work is entirely reasonable. The problem is that some of those companies also cut off the search crawler in the process — precisely the one that decides whether an assistant can cite them today.

Here is how it actually breaks down, according to the providers' own documentation:

OpenAI (docs) separates this into four:

  • `GPTBot` — collects content that may be used to train the models
  • `OAI-SearchBot` — builds the index behind ChatGPT's search features
  • `ChatGPT-User` — visits a page when a user asks it to
  • `OAI-AdsBot` — checks the safety of pages submitted as ads

Anthropic (docs) documents three:

  • `ClaudeBot` — training
  • `Claude-SearchBot` — improving search result quality
  • `Claude-User` — user-initiated visits

Google (docs) uses the `Google-Extended` token, and attaches a detail worth knowing: blocking `Google-Extended` does not affect your inclusion in Google Search and is not used as a ranking signal. In other words you can decline to have your content train Gemini without paying for it in ordinary search positions. Very few decisions in this field are that cost-free.

The practical conclusion is simple: blocking a training crawler and blocking a search crawler are two different decisions. The first is about whether your content feeds a model. The second is about whether you exist to the assistant at all. You can comfortably make the first without making the second.

If you run a business site and you want customers, open your `robots.txt` — it's just `yourdomain.com/robots.txt` — and read what's there. Bot names get added and renamed, so always take the current list from the provider's own documentation rather than from an article written a year ago. Including this one.

What actually works

Everything on this list is checkable and none of it requires taking anyone's word for anything.

The content has to be in the HTML, not painted in by JavaScript. This is the biggest own-goal I see. If a page only renders its content in the browser, then a crawler that doesn't execute JavaScript receives an empty page. Not a worse page — an empty one. The test takes ten seconds: open the page, choose "View page source", and search the code for a sentence you can see on screen. If it isn't there, then as far as some crawlers are concerned your page has no content.

The heading structure has to mean something. One `h1`, then `h2` for sections, `h3` for subsections. Not because it's tidy, but because it's the only hierarchy information a machine gets from prose.

Write in question-then-answer shape. A model quotes fragments. A fragment that can be lifted out and pasted as an answer stands a better chance than a paragraph that needs context from the three before it. Instead of "our experience allows us to take a flexible approach to pricing", write "a brochure site costs 1,000–8,000 PLN. The price depends on the number of pages, the integrations, and whether you need a shop." The second is quotable. The first says nothing.

State the specifics plainly. Prices, city, scope, hours, lead times. An assistant asked "what does this cost" needs a number. A page with no numbers won't get cited in an answer about pricing — it'll be skipped in favour of a competitor who published one. That's the same argument that has always worked on humans; I went into it in how to prepare content for your website.

Structured data. `LocalBusiness`, `FAQPage`, `Article` — schema.org. It's a description of your business in a format a machine doesn't have to infer from prose. It has worked for Google for years and it's exactly the kind of signal that helps here too.

Currency. A model answering a question about 2026 will more readily reach for material that describes 2026. A three-year-old price list hurts twice: it's wrong, and it looks abandoned. I've written separately about what happens to a site nobody looks after.

Notice that this is essentially a list of good practice you needed anyway. There's no new magic here — there is a new reason to finally get it done.

What's uncertain: llms.txt

I have to be straight here, because I've implemented it myself.

`llms.txt` is a proposed convention: a text file at the root that summarises what a site is and where things live — something like `robots.txt`, but for language models. I run one on this site, at szympanstudio.pl/llms.txt.

And here's the part most articles on the subject leave out: no major provider has publicly committed to reading that file in their products. Google has said outright that it doesn't support the idea and isn't planning to. OpenAI's own documentation points to `robots.txt` as the way to control crawlers. Some companies publish an `llms.txt` for their own documentation — but that's a developer-tooling use, not a signal that influences who an assistant cites.

So why do I have one? Because it costs an afternoon, there's no way for it to break anything, and if the convention takes hold I'm ready. I treat it as cheap insurance, not as a tactic.

If someone is selling you `llms.txt` as "the way to rank in ChatGPT", they're selling you something nobody has confirmed yet.

What's being sold to you

A whole category of "AI positioning" services has appeared. You can spot them by a few things:

  • A guaranteed position in the answer. There is no such thing as a position in an assistant's answer. The answer is generated fresh each time and can differ between two people asking the same thing. There's no list to climb.
  • Reports showing "your rank in ChatGPT". Since the answer varies, such a report is a single sample presented as a measurement.
  • Copy written "for AI". A model cites what is specific and credible. There's no separate style that raises your odds of being quoted, beyond writing clearly — which is what works on people anyway.

A healthy rule: if a service promises control over something the provider exposes in no interface at all, it isn't a service, it's a story.

What to do this week

Concretely, in order:

  1. Open `yourdomain.com/robots.txt` and check you aren't blocking the search crawlers. If you are, make it a deliberate choice — training is one decision, search is another.
  2. Open your homepage source and confirm the visible text is genuinely in the HTML.
  3. Rewrite your offer as specific sentences containing numbers: prices, timelines, scope, location.
  4. Check whether you have structured data — Google publishes a free rich results test.
  5. Update whatever is out of date. Start with the price list and with "recent projects".
  6. Ask a few assistants about your industry and your town. See who they cite, and what that page has that yours doesn't. It's the cheapest audit available to you.

Point six matters more than it looks. You stop guessing and start looking at a specific answer with specific sources.

In short

The shift is real and it's worth acting on now, while most of your local competition still isn't. But how you act on it matters.

Almost everything that genuinely helps is something that should have been done regardless: clean HTML with the content actually in it, a sensible structure, specifics instead of generalities, current information, and no accidental block on the crawlers you care about. Dull. Checkable. Works on humans too.

The rest — promise-files, guaranteed positions, reports built on one sample — is a new label on an old product. I collected the most common mistakes on business websites a while back, and half of them are the same things that now decide whether you show up in an AI answer.

Want to know how your site looks from this angle — whether its content is visible to crawlers at all, and whether you've accidentally blocked something? Get in touch and I'll check it and tell you plainly what needs fixing and what already works.