Vector Search Can’t Tell “Yes” From “No”

Two sentences can only disagree if they're about the same thing. For instance, “The battery is replaceable” and “Paris is in France” don't disagree. They're just unrelated.

To truly disagree, everything in the sentence has to match except the one key difference between them. Sometimes, that’s only one word.

Vector search measures that matching, which means it can tell that two sentences are about the same subject, such as batteries, but not necessarily whether they agree. “The battery is replaceable” and “The battery is not replaceable” look very similar because almost all the same information is there.

That’s why contradictory statements can end up ranking almost identically in vector search. 

An embedding turns a sentence into a few hundred numbers, which means some information gets compressed or lost. What remains depends on what the embedding model was trained to recognize. 

For many retrieval models, that training emphasizes whether two pieces of text are about the same subject.  A question is paired with a passage that answers it, while incorrect answers come from completely unrelated topics. Over time, the model gets rewarded for spotting the “same subject.” However, it gets much less practice spotting “opposite claims,” especially when the difference comes down to a single word, like “not.”

The way embeddings represent a sentence can make that problem worse. For example, “The policy was approved” and “The policy was not approved” share every word but one. When those words are represented together in an embedding, the one word that reverses the meaning can have very little influence on the overall similarity score.

To see how this plays out in practice, we ran 16 test sentence pairs through an embedding model (Figure 1):

Suppose that you build a RAG-based customer support assistant that answers questions using your company’s documentation: the current manual, a 2019 version that nobody deleted, some old blog posts, and a few forum threads. A customer asks: Can I replace the battery myself?

In the corpus, the assistant finds:

• “The battery can be replaced by the user with a standard screwdriver.” — 2024 manual

• “The battery is sealed and cannot be replaced by the user.” — 2019 manual

Only one of these is right. But to an embedding model, they’re still highly similar because they’re about the same part and the same action. As a result, both can be returned as answers, and the outdated answer may rank just as highly as the current one. Nothing in the score knows that one of them is five years out of date.

Here’s another example: If a retrieval query is run against a corpus like the one below, the outdated answer will rank higher than the current one four times out of six (Figure 2).

This contradiction problem isn’t limited to text. Consider an image-based RAG system like Miranda. If a user provides a picture of a garment and asks, “Find me something that looks like this, but only something that can be tumble-dried,” the image embedding can easily find visually similar items. However, the retrieval process must still distinguish between “can be tumble-dried” and “cannot be tumble-dried,” bringing us right back to the exact same challenge.

The usual RAG setup is straightforward: Split documents into chunks, embed them, retrieve the top few matches, and paste them into the prompt. Along the way, other common steps like deduplication and filtering can also affect what the model ultimately sees.

A few other ways that conflicting information can get lost include:

• Top-k retrieval can hide one side of the disagreement

• Chunking can strip away important context

• Removing duplicates can delete the fix

• The model can't recover information that retrieval left out

An agent can deal with a disagreement if the retrieval system shows that one exists. It can check dates, search again, compare sources, or come back and ask you for clarification. But it can't do any of that if search returns only one version.

To mitigate this, the search layer should surface the conflict instead of silently choosing between the two sources. This takes more than one signal, including:

• Vectors that find candidates, and nothing more

• Keyword searches for exact details like numbers, versions, or error codes (an embedding gives “five” and “ten” a 0.92 similarity score, but keyword search can distinguish them)

• Metadata filters for date, version, plan, region, and source

• A conflict list built when documents are loaded, so known disagreements can be flagged before a user ever asks a question

• A rule for picking between sources that the agent can explain

None of these techniques is complicated on its own, but the mistake is expecting the vector search to do all five jobs.

Instead of making vector search do everything, the goal is to build a retrieval process that can recognize conflicts, preserve the context around them, and give the agent enough information to respond appropriately.

These patterns become especially important as RAG and agentic applications move into production. Akamai offers several capabilities that can support these workflows:

• Akamai Valkey Managed Database supports low-latency vector retrieval and contextual data access for AI applications

• Akamai Functions runs application logic and AI pre- and post-processing close to users 

• Akamai Cloud for AI provides distributed compute, data, serverless functions, and GPU-powered inference for AI applications at global scale 

• Akamai AI Professional Services helps enterprises accelerate AI adaption through fixed-scope, outcome-focused engagements that deliver clear value 

Together, these capabilities give teams more flexibility to build retrieval and inference workflows that preserve context and handle conflicting information more deliberately.