Welcoming AI Agents to Your Website
Technical cures for AI invisibility
AI agents such as ChatGPT, Claude, Gemini, and Perplexity are quickly becoming the way people find information and become aware of businesses. But most websites were built before AI agents existed, and many are essentially invisible to them.
The good news is that fixing this does not require a redesign. It requires focused technical updates that can be implemented for any website.
Key ideas:
- Why AI agents evaluate your site differently than Google
- The files and permissions that can influence how AI agents interact with your site
- Making your content actually readable by crawlers that do not render JavaScript
- Structured data and discovery signals that can get your site cited and recommended
Why AI agents evaluate your site differently than Google Search
Google has spent years teaching its crawler to behave like a browser. It renders JavaScript, executes page logic, and indexes what a human would actually see. AI agents like GPTBot (ChatGPT), ClaudeBot (Anthropic), PerplexityBot, and Gemini take a fundamentally different approach. Most request the raw HTML your server delivers and stop there. They do not run your JavaScript. They do not wait for your page to load. They read what is in the source, and if your content is not there, it does not exist to them.
This distinction matters more every month. ChatGPT crossed 900 million weekly active users in early 2026, up from 400 million just a year prior.1 In 2026, people are choosing to use AI over traditional search 29% of the time for a quick answer or fact, 32% of the time for recommendations, and 40% of the time for step by step instructions.2 If your site is not readable to these agents, it is being skipped in a channel that is growing fast.
The files and permissions that can influence how AI agents interact with your site
Your website communicates with crawlers through a set of small but critical files. Most site owners know robots.txt as the file that tells Google what to index. In 2026, that same file needs to address a growing list of AI-specific crawlers, each with its own user-agent identifier — GPTBot, ClaudeBot, PerplexityBot, GoogleOther, and more. Without explicit entries, you have no control over what these agents access or how they use it.
Beyond robots.txt, a new file called llms.txt has emerged with the intention to serve as the AI-era equivalent of a welcome mat. The file is hosted on a website's root directory and acts as a curated map for AI bots and Large Language Models (LLMs). Where robots.txt sets boundaries, llms.txt provides guidance by pointing AI agents to your best, most authoritative content and clarifying how it may be used.
It is important to note, however, that the llms.txt file is a proposed standard. It has not been officially adopted by major AI companies, and the benefits of including it are unclear. In general, our position is to include it because: various assessment tools check for this file and its absence can impact scoring, there's an element of "future proofing" while standards are developing, it doesn't hurt the site, and the level of effort to set it up is low.
A complete setup also includes a traditional sitemap that AI agents can follow, response headers that aid discovery, and for more sophisticated implementations, a tool index or server tool card that describes what your site can do and not just what it contains.
Making your content actually readable by crawlers that do not render JavaScript
Modern websites are built to look great in a browser. The problem is that AI crawlers are not browsers. Any content your site loads through JavaScript — such as product descriptions, article bodies, pricing tables, or navigation menus — is effectively invisible to a crawler that only reads raw HTML. The solution is prerendering, which converts the JavaScript application into flat, text-heavy HTML on the server before serving it. Once the text becomes readable, AI bots have a much fuller picture of the website's information.
The fix does not usually warrant a rebuild, but some restructuring may be needed. The goal is to ensure that the most important content on each page, such as the main body text, headings, and key facts, exists in the HTML the server sends, before any JavaScript runs. This is what server-side rendering (SSR) and static site generation (SSG) accomplish. It is also worth checking the ratio of actual text to page markup. A page that is 90% code and 10% readable content will score poorly with AI agents that are trying to understand what the page is about. Semantic HTML structure, like using the right tags for headings, sections, and main content, makes it easier for agents to identify what matters on each page.
Structured data and discovery signals that can get your site cited and recommended
Structured data is the vocabulary AI agents use to understand what your content means, not just what it says. Some recommend using JSON-LD markup based on schema.org to tell crawlers that a block of text refers to an organization, a product, a how-to guide, or a frequently asked question so that the agent's crawler doesn't have to guess. While there is some debate about the degree to which AI crawlers respect specific schema.org markup, in general, improving the quality of the knowledge the crawler is gathering through structured data increases its semantic meaning and makes it easier for AI agents to consume. Research suggests that pages with valid structured data appeared more often in AI-generated summaries, and that pages with clean semantic structure paired with schema markup earned higher AI citation rates than poorly structured pages.
Beyond schema, there are several other discovery signals worth implementing. A well-formatted XML sitemap, that is kept current and free of dead links, gives crawlers a reliable map of everything on your site. Open Graph and Twitter/X Card meta tags, while originally built for social sharing, are also read by AI agents to understand page context. For businesses that want to go further, publishing a Markdown version of key content pages gives text-only crawlers a clean, low-friction format to work with. None of these changes are dramatic. Together, however, they shift your site from one that AI agents can technically access to one they can actually understand and confidently cite.
The web is not broken for AI agents, but most websites were not designed with them in mind. That gap is closing quickly, and the businesses that close it intentionally will have an advantage as AI-driven discovery continues to grow. The encouraging part is that none of this requires a redesign, a new platform, or a large budget. It requires attention to a layer of your website that most people have never looked at.
Since AI agents read your raw HTML and not your rendered page… here's a recap of what you'll want to assess technically in order to roll out the website welcome mat for AI agents.
- robots.txt — updated with AI-specific crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, GoogleOther)
- AI crawler permissions — explicit access rules for each major agent
- AI content usage preferences — clarify what agents may and may not do with your content
- llms.txt — AI guidance file pointing agents to your best content
- XML sitemap — current, complete, and free of dead links
- Discovery headers — response headers that help AI agents identify and index your content
- JavaScript rendering — critical content present in server-delivered HTML, not dependent on JS to load
- Content-to-markup ratio — meaningful text relative to page code on all pages
- Semantic page structure — proper heading hierarchy, section tags, and main content identification
- Structured data (JSON-LD) — schema.org markup for key content types (Organization, Article, etc.)
- Open Graph / meta tags — page context signals readable by AI agents
- Markdown content delivery — clean text format available for AI crawlers on key pages
You have already done the hard work of building a website and creating content. These technical steps are how you make sure that work gets seen by the humans who visit your site and the AI agents that are increasingly deciding what information sources to recommend.
See how your site measures up
Knowing what to fix is the first step. Our BASELINE Website Audit checks your site for AI Agent Readiness, AI Crawler Visibility, page speed, and accessibility — and gives you a clear list of what needs attention and why.
Learn about the BASELINE Audit →
Questions? Email us →
About Presentek
Presentek has been helping businesses design and build better online experiences for more than 25 years. If this post raised questions about your own site, here are the services most relevant to what was covered.
Websites & Applications
Website development, modernization, security and performance, and AI-related services — including making AI-coded prototypes production-ready. If the technical gaps described in this post need hands-on fixing, this is the right starting point.
Solution Design & Consulting
If you are thinking about how AI could be incorporated into your website or application — not just accommodated by it — our AI Solution Design service takes you from concept to proof-of-concept to production-ready.
Marketing, Product & Sales Tools
Brand, go-to-market, content, and digital marketing deliverables, plus product launch kits and sales enablement tools. Useful context if your website work is tied to a broader marketing or product initiative.