What to Do When Your Competitors Are Scraping Your Prices

It’s a problem: Your competitors scrape your pages, parse your catalog, and undercut your prices. You try to respond by adjusting prices to compete. But your competitors just scrape and undercut you again. It’s a downward spiral, and it’s costing you customers and sales.

Solutions are difficult. You can’t hide your data: Shoppers can’t buy what they can’t see. And you can’t block the bots. A block tips off the scraper that it’s been spotted, and it inevitably just rotates IP addresses/fingerprints, and then it returns. You could try implementing complex rules, but then you risk turning away the search indexers and partner integrations that are supposed to get through. All of these solutions have downsides.

What’s needed is a solution that keeps your pricing and catalog proprietary and allows you to selectively decide what data each visitor sees. With Akamai Functions, you can do just that. You can identify bots, decide which ones are good and which ones are bad, decide what data each category sees, and do it all right at the edge where every request already crosses, before the bots reach your data.

Let’s look at how it works.

Real users and search indexers (the good bots) are identified and pass through to your cache or origin. There’s no ranking hit, no customer loss, and no broken partners. Traffic classified as a scraper (or other bad bots) is caught and routed to Akamai Functions.

Next, Akamai Functions acts as a rewriting proxy. It builds and returns a modified response with proprietary data redacted, obfuscated, or summarized. There’s no blocked page and no error. The scraper bot gets no signal that it has been caught.

Classification, rewriting, and caching all happen in coordination, and all at the internet’s front door (the edge), with the function acting as the glue between Bot Manager’s verdict and the response. And the response is cached at the edge for the next time a scraper asks for the same page.

With this solution, you are in full control of who sees your data.

Let’s look at three examples of how this might work in the real world, including how to:

• Obfuscate pricing for competitor scrapers

• Protect your catalog and feeds from aggregators

• Truly control who gets your data

A rival’s bot is hitting your product pages every three minutes. It’s a spy inside your store: The bot reads your prices, feeds those prices to its repricing system, and automatically adjusts its prices to undercut yours.

In a traditional solution, you respond by blocking the bot by IP address and fingerprint. Unfortunately, the bot often recognizes that this has happened. It simply rotates to a new address and hits your site again. You respond by writing another rule, blocking another IP, but the bot changes again. And so on.

Eventually, the rule gets so complex that it ends up blocking real shoppers or good search crawlers. Your solution becomes an escalating cycle where you’re perpetually behind and can never win.

With an Akamai Functions solution in place, Bot Manager intercepts the request at the edge and scores it as a scraper. Because you want your pricing to remain proprietary, the request goes to Akamai Functions, which rewrites just the price in the response, swapping the real number for a stale, randomized, or withheld value. The rest of the response is left intact.

Because the scraper is never blocked, it’s not aware that it has been detected. It receives and ingests the customized data, giving its repricing system the numbers you chose to show it. The shopper-facing page is never touched. 

You effectively detected the spy and wrote its report — all without it knowing.

An aggregator has targeted your site. Unlike in the first example, aggregators aren’t after a single price to undercut. They want your whole catalog — SKUs, descriptions, images, availability, and prices — so that they can rebuild the catalog and relist your products somewhere else.

Retail catalogs are their prize: More than 45% of AI bot traffic on Akamai networks targets commerce customers, drawn to the clean, structured product, price, and availability data that ecommerce pages publish by necessity.

In a traditional design, your catalog pages and product APIs are wide open by necessity. You might try to watch for and block bulk bot requests, but origin rate limiting is easy for bots to evade. They simply slow down and spread requests across addresses.

Inflicting even more pain, every bot scrape that gets through adds origin load and egress cost. You’re not only serving the data that will be used against you, but you’re also paying to serve that data.

A better solution is to use Akamai Functions tied to Bot Manager’s classification. With this design, you can deploy a set of content-shaping rules and decide per request what each client receives.

For example, when Bot Manager detects that the caller is a scraper bot, it works in conjunction with Akamai Functions to return a degraded response: partial fields, no stock counts, and a summary in place of the fully structured feed. When Bot Manager classifies the caller as a shopper or trusted partner, the function returns the full, original response.

One function, deployed across Akamai Cloud regions, can apply the same policy on every route a scraper might try (the web page and product API alike) and absorb that traffic before it reaches your origin.

You’ve become too aggressive in your response to bot scrapers, and you’re accidentally blocking traffic you want to let through. Price-comparison partners you have agreements with, your own mobile apps, and authorized resellers all need the real catalog and live prices to do their jobs.

In traditional designs, access tends to be all or nothing. You open the data wide enough for partners and competitors alike to collect it. Or you lock it down and break the integrations your sellers depend on. Neither works well.

In traditional designs, access tends to be all or nothing. You open the data wide enough for partners and competitors alike to collect it. Or you lock it down and break the integrations your sellers depend on. Neither works well.

With Akamai, the function connects identity and authorization to Bot Manager’s classification. Keys, signatures, and allowlists are checked at the edge against state in the Akamai EdgeKV store. A trusted client gets the full data, and everyone else gets the degraded or rewritten view. You decide who sees what, per request, in one place, rather than spreading that logic across the applications at your origin.

The solution also grows with you. AI-powered bot traffic is already up 300% year over year, and the commerce industry alone saw more than 25 billion AI bot requests in a single two-month span. 

The next clients at your front door might be good bots of a new kind: AI shopping agents that act on a customer’s behalf and need the same authenticated access to real prices and live inventory that your partners have today. Akamai allows you to say yes to the agents that are there to buy and no to the bots that are there to scrape, using the same decision point at the same front door.