Your Website Now Has Two Audiences: Humans and AI

Your website now has two audiences.

The first audience is the one you originally built your site for: humans with browsers. The second one is new and is growing in importance and in numbers: AI agents and crawlers. 

Both of these audiences are important, but they need very different things.

• Humans need a fast, polished experience. They judge the page in a few seconds on layout, usability, images, and how quickly the page responds.

• AI agents, on the other hand, don’t care about visual layout. They want structured data, fresh content, and a clean response they can easily parse.

Earlier in this blog post series, we argued that a good portion of application logic belongs at the front door; that is, at the edge of the network every request already crosses. Serving two different website audiences is a perfect test for this argument. 

With Akamai Functions at that front door, paired with Bot Manager’s classification and other Akamai services, you can recognize which audience each request comes from and give that audience exactly what it needs — all without sacrificing response time, forking your origin, or maintaining a second rendering pipeline.

There is a better process. By using Akamai Functions as the glue that connects services such as Bot Manager, you can create a response tailored to each audience, right at the edge, without sending every request to the origin. You can make both your human and AI audiences happy.

Many websites are built using a client-side rendered JavaScript framework meant to serve humans. When queried, the server sends back a skeleton: a nearly empty HTML document plus script tags. A browser downloads those scripts and executes them. Then, the browser builds the page, fetches the product data, and writes the prices and descriptions into the document. The full content of the page only appears after the client-side code runs.

The traditional setup

That works fine for humans, but most AI crawlers never see the full content because they don't execute JavaScript. Independent testing of the major crawlers finds that GPTBot, ClaudeBot, and most of their peers read only the raw HTML. What these bots get from the product page is the skeleton itself: an empty content div and a list of script tags.

As a result, the page ranks well in classic search, but AI assistants either skip the site or fill the gap from somewhere else, resulting in users getting incorrect answers.

The Akamai Functions solution

With Akamai Functions as the glue, this process looks different.

First, Bot Manager identifies the AI crawler at the edge. The request then routes to a function that takes Bot Manager's classification, pulls the product data, coordinates the structuring on Akamai Cloud, and caches the machine-ready response at the edge so the next crawl is a cache hit.

Classify, render, and cache all happen in one pass. The data is rendered into clean HTML with schema.org markup or plain markdown — the navigation, scripts, and styling that only matter to a human is removed. The bots are happy: Their version of the page is derived from the same source as the human page, but is in the format they need; i.e., generated and cached at the edge rather than maintained as a second site.

The human visitors never see any of this process. Their page appears just as they need it, unchanged (Figure 1).

AI crawlers frequently revisit sites to retrieve updated dynamic information, including prices, product availability, and recent content updates.

The traditional setup

Traditionally, sites have relied on caching, which has always been a trade-off between freshness and speed. AI traffic makes the trade-off even more pronounced. Teams often try to tune this balance with the cache time to live (TTL) setting. But if they set the TTL too long, the AI assistants serve last week's price. If they set TTL too short, every crawler revisit becomes an origin hit, which is paid for in load, egress, and latency (multiplied across every AI crawler). 

Neither answer is ideal.

The Akamai Functions solution                                                                  

That trade-off can be removed by using Akamai Functions to assemble the response. The fields that change often live in the key-value (KV) store, replicated across Akamai Cloud regions.

When a price changes, a commerce or content system writes the change once, and it propagates to every region. The function reads those fields fresh on every request (a small hop, since the KV store lives in the same Akamai Cloud region as the function). As a result, fields that change often (price, stock level, headline) are served fresh, while fields that don’t change come from the edge cache.

The function becomes the glue between the KV store's live fields and the cached page, assembling one AI-ready response from both. The frequent crawler gets near real-time data on every request, with no per-crawl trip to the origin. The origin writes a change once and serves the occasional cache refill; it stops paying for every re-crawl (Figure 2).

Both of your audiences, humans and AI, need dynamic pages to be fast. These pages — a personalized landing page, an inventory-aware product page, and a campaign page assembled per visitor, for example — are the pages most likely to convert a human and most likely to be quoted by an AI assistant. 

They are also typically the slowest pages you serve.

The traditional setup

In a centralized architecture, personalization, inventory checks, and content assembly require round trips to a central origin region, adding 50 to 100 milliseconds per hop. Combining multiple requests increases page rendering time by 250 milliseconds or more for both humans and AI crawlers. As a result, Core Web Vitals slip, and search rankings, paid search quality scores, and conversions suffer. 

AI crawlers rarely wait or retry, so a slow response can mean that the page gets skipped altogether.

The Akamai Functions solution

With Akamai Functions, the logic all runs closer to the user. The function looks up the visitor's segment in the KV store, calls the recommendation model running on GPUs in Akamai Cloud, checks inventory, and assembles the page — all without leaving Akamai's network.

Akamai's benchmarks put the typical personalization round trip at 200 milliseconds in a centralized setup; moving it to the edge can cut that to just 40 milliseconds, with inference costs falling by up to 86%. Using Akamai Functions accelerates page rendering, improves Core Web Vitals, optimizes organic search rankings, lowers paid search costs, increases conversions, and delivers fast responses to AI crawlers (Figure 3).