The terms that decide whether your website gets found — defined once, in plain English, without the padding. Each one takes about thirty seconds to read.
An AI crawler is an automated bot — like GPTBot, ClaudeBot or PerplexityBot — that reads public web pages so an AI assistant can describe or recommend them later. It is separate from Googlebot: allowing Google does not allow AI crawlers, and many websites block them without realising it.
updated 2026-08-17GPTBot is OpenAI's web crawler — the bot that reads public pages so ChatGPT can describe them. If GPTBot receives a 403 on your site, the block is usually in your CDN or firewall rather than your robots.txt, which is why editing robots.txt so often changes nothing.
updated 2026-08-17AEO, or Answer Engine Optimization, is the practice of making a website more likely to be named in an AI assistant's answer. Where SEO competes for a place in a list of ten links, AEO competes to be quoted inside a single written answer — where there is no second page.
updated 2026-08-17llms.txt is a proposed file, placed at your site's root, that describes in plain markdown what your website is and where its key pages live. The honest position in 2026: no major AI provider reads it in production yet, so treat it as a cheap nice-to-have rather than a way to get found.
updated 2026-08-17A website audit is a structured check of everything that decides whether a website works for real visitors: whether it renders properly on phones, loads quickly, can be found in search and AI answers, meets accessibility basics, and avoids public security mistakes. It examines what your site shows the outside world.