SEO for AI Search: llms.txt, Structured Data and Crawler Rules
People now find websites through AI assistants as well as search engines. The good news is that most of what helps one helps the other: crawlable pages, clear structure and trustworthy content. This post covers the practical steps I used when building my own portfolio, and I am clear about which parts are proven and which are still experimental.
Key takeaways
- Allow the crawlers you want, and say so explicitly in
robots.txt. - Server-render your content so every crawler can read it without running JavaScript.
- Use JSON-LD to describe who you are and what each page is.
llms.txtis a proposed convention and no guarantee, so treat it as a cheap extra and not a strategy.
1. Make the content easy to read
Crawlers read HTML. If the important text only appears after client-side rendering, some of them will not see it. Server-render or statically generate pages, use one h1 per page, keep a logical heading order, write descriptive link text and give every image meaningful alt text.
2. Control crawlers in robots.txt
Search and AI products use different user agents, and some separate search from training. Common ones include:
GooglebotandBingbotfor searchGPTBotandOAI-SearchBotfrom OpenAIClaudeBotfrom AnthropicPerplexityBotfrom PerplexityApplebot-ExtendedandGoogle-Extended, which are robots.txt tokens that control whether your content is used for certain AI features, not separate crawlers
Check each vendor's documentation for the current list, because names change. In Next.js you can generate the file from code:
export default function robots() {
const bots = ["GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot"];
return {
rules: [
{ userAgent: "*", allow: "/" },
...bots.map((userAgent) => ({ userAgent, allow: "/" })),
],
sitemap: "https://example.com/sitemap.xml",
};
}If you do not want your content used by a particular crawler, disallow it. This is your decision, and the file is a request that well-behaved crawlers honour.
3. Add structured data
JSON-LD tells machines exactly what a page is. For a personal site or portfolio the useful types are:
- Person with
name,jobTitle,sameAslinks to your profiles andknowsAbout - WebSite linking to the Person
- BlogPosting for articles, with headline, dates, author and image
- BreadcrumbList for navigation context
- FAQPage when a page genuinely has questions and answers
Google limits FAQ rich results to a small set of sites, but the markup is still a clean, machine-readable statement of your content. Only mark up what is actually visible on the page, and validate it with the Rich Results Test.
4. Publish a sitemap and canonical URLs
A sitemap lists the pages you want indexed. Each page should declare a canonical URL, a unique title and a unique description. Keep titles around 60 characters and descriptions around 155 so they are not truncated.
5. Consider llms.txt
llms.txt is a proposal for a plain Markdown file at the root of your site that gives language models a concise, curated summary of your most important pages. It is cheap to add. It is not a standard that every AI product reads, so do not expect it to change your rankings or citations on its own. If you add one, keep it accurate and generate it from the same data as your site so it never drifts.
6. Build real authority
Technical SEO gets you indexed and understood. It does not make you the best answer. For that you need:
- specific, experience-based content that other pages do not already cover
- a clear author identity with consistent profiles that match your structured data
- links from other reputable sites
- fast pages, since Core Web Vitals are a ranking signal and a usability one
What I did on this site
This portfolio ships a generated robots.txt, a sitemap, canonical URLs on every page, Person, WebSite, FAQPage and BlogPosting structured data, a generated social image and an llms.txt built from the same data files as the pages. If you want help with technical SEO for a Next.js site, get in touch or read about my web application work.
- SEO
- AI search
- structured data
- llms.txt
- robots.txt