Building a site for AI, not just for Google

The question used to be simple: "how do I rank on Google?" Today language models have joined the search engines — ChatGPT, Claude, Perplexity. They crawl sites too, they take content too, and more and more often they are where a person gets their answer without ever opening your site.

I recently went through this on my own blog, and I want to share what actually works in "optimizing for AI" — and what is still just noise.

The main thing: AI crawlers don't run JavaScript

If you remember one fact from this article, make it this one.

Googlebot can render JavaScript — it loads the page, waits for the scripts, and sees the final result. AI crawlers can't. GPTBot, ClaudeBot and PerplexityBot read only the HTML the server returned immediately. They may download your scripts as text, but they never execute them.

The conclusion is simple and unpleasant for a lot of modern sites: if your content only appears after JavaScript runs, it doesn't exist for language models. Not "indexes poorly" — literally isn't there.

So item number one isn't structured data or clever files, it's server-side rendering. SSR, static generation, plain PHP returning ready HTML — anything, as long as the text is in the page source. Easy to check:

curl -s https://example.com/page | grep "a phrase from your text"

If that finds nothing, no model will ever see your text.

What actually helps

Structured data (JSON-LD)

This is markup that tells a machine the meaning of the page outright: who the author is, what kind of material this is, when it was published, what it's about. The model shouldn't have to guess from your CSS classes — you say it directly.

A minimal set for a blog is Person (who you are), WebSite (what the site is) and BlogPosting on each article:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Person",
      "@id": "https://example.com/#person",
      "name": "Your Name",
      "jobTitle": "Developer",
      "knowsAbout": ["Backend", "AI", "Fintech"],
      "sameAs": ["https://github.com/…", "https://t.me/…"]
    },
    {
      "@type": "BlogPosting",
      "headline": "Article title",
      "datePublished": "2026-09-05T10:00:00+05:00",
      "author": { "@id": "https://example.com/#person" }
    }
  ]
}
</script>

Note the @id and the author reference: this makes the article's author and the person one entity, not two similar-looking strings. For a machine that difference is fundamental.

knowsAbout and sameAs are worth filling in separately — they answer "what does he actually know" and "where else does he exist online."

The boring things that never went away

Meaningful HTML. One h1, real h2s, article, time with a datetime attribute.
meta description — a short, honest summary of the page.
canonical and hreflang if the site is multilingual; otherwise your versions compete with each other.
Machine-readable dates — 2026-09-05T10:00:00+05:00, not "September 5th".
RSS and sitemap.xml — old, boring, still-working ways to say "here is all my content."
robots.txt, where you deliberately decide who gets in.

Text that is easy to quote

A model rarely retells a whole article — it pulls out a fragment. So texts win where each section stands on its own: a clear subheading, a direct answer in the first paragraph, no "as we mentioned above." Which, conveniently, also helps human readers.

What about llms.txt

There's a lot of hype around this file, so let me be blunt.

llms.txt is a text file at the site root with a short description of the project and a list of its materials, aimed specifically at language models. The idea is good: hand the machine a ready summary instead of making it wander through markup.

But as of 2026 it's worth being clear-eyed:

it is not a standard — neither W3C nor IETF backs it; it's a community convention;
neither OpenAI, Google nor Anthropic has publicly committed to reading it in production;
across observations of hundreds of millions of AI-bot visits, requests to llms.txt itself are a fraction of a percent;
adoption sits around 10% of sites, mostly technical ones.

It is alive in a different niche, though: developer tooling reads it — IDE agents like Cursor and Claude Code, MCP servers, built-in assistants. For documentation and technical projects it already pays off.

My take: shipping an llms.txt is worth it — it's a few lines and can be generated automatically. But don't expect traffic from ChatGPT because of it. It's a bet on the future, not a working channel today.

Should you let AI bots in at all

A separate question, and everyone answers it for themselves.

Via robots.txt you can block GPTBot, ClaudeBot, Google-Extended, CCBot — then your content won't feed training. But you also give up the chance of a model citing you in an answer.

For a personal blog written to be read, letting everyone in makes sense. For commercial content people pay for, the decision may be the opposite. What matters is making it deliberately rather than by default.

How to check yourself

A quick checklist you can run in five minutes:

Does curl on your page show the article text? If not, start there — nothing else matters yet.
Is there an application/ld+json block, and is it valid?
Are the article's author and your About page linked through a shared @id?
Are canonical, hreflang and meta description in place?
Are sitemap.xml and RSS reachable?
Who do you allow in robots.txt — did you decide that, or did it just happen?

The takeaway

A layer of myths and paid methodologies has already grown around "AI optimization." In practice it comes down to things a good site does anyway: render text on the server, mark up meaning explicitly, write clearly, and don't hide content behind scripts.

The good news is that this is one of those rare cases where working for the machine and working for the human are the same job.