Beyond Vector Search: Retrieval That Respects Business Rules.

01

First, separate relevance from eligibility

Embeddings represent text as vectors so you can search for nearby meanings. That’s useful when a customer says “I keep getting bounced back to sign-in” and the runbook says “authentication redirect loop.”

But vector distance is not a reliable implementation of status = open, a timestamp comparison, or an exact error-code match. It also cannot decide whether a user is entitled to see a record. Those conditions need explicit enforcement.

For our example, divide the request into two groups:

The Boolean meaning matters too. “AUTH-1042 or a login loop” permits either branch. If the user asks for “AUTH-1042 only,” the code becomes a hard exact-match condition. A search system that quietly rewrites one into the other has changed the question.

Resolve relative time once, using an explicit time zone and an as_of timestamp. Decide what “EU customer” means in your data model: billing region, account region, or incident location. If those interpretations materially change the result, ask rather than guessing.

02

Let different retrieval methods do different jobs

BM25 covers the lexical side. It scores documents using query terms, with factors for term frequency, how common a term is across the collection, and document length. It is commonly implemented over an inverted index. This is the sparse branch: documents contribute term evidence for the vocabulary they contain. Rare technical terms can be especially useful signals.^R1

That makes BM25 a strong companion for product names, configuration keys, and distinctive error messages. It still isn’t an exact-equality operator. Tokenization may split punctuation or normalize text. Preserve error codes in a dedicated keyword field when exact identity matters.

Dense search covers the semantic side. An embedding model helps retrieve passages that express a related idea using different words. The query and document encoding must be compatible, including any query/document modes the chosen model requires.^R2

Structured queries cover the rules. Use indexed metadata filters for fields already in the search index. Use a relational query or a dedicated domain service when the request depends on joins, authoritative status, or calculations. “Accounts with two overdue invoices and an active enterprise contract” should not become a vague similarity query.

Hybrid search combines lexical and semantic retrieval.^R3 A practical version runs the two searches concurrently over the same authorized, eligible corpus, then combines their candidates. The two branches should refer to the same document and chunk identities so their results can be reconciled.

03

Give both indexes something worth finding

Before tuning search, inspect what you indexed. A chunk reading “This fixed it” is almost useless if the product version, error message, and actual fix were cut into other chunks.

For incident records, preserve a meaningful unit: the incident title, a coherent event or resolution passage, and enough surrounding context to interpret it. Attach stable document and chunk IDs, tenant and access metadata, timestamps, region, status, source URL, and content version.

Keep lexical text and embeddings tied to the same content version. If one index contains a corrected runbook while the other still serves an old copy, fusion can combine contradictory evidence. Treat deletions and permission changes as ingestion events too. An inaccessible or deleted document shouldn’t survive indefinitely in a search cache.

Prefer filtering within candidate retrieval where the engine supports it. Retrieving ten global neighbors and discarding nine forbidden records afterward leaves one result; it doesn’t magically discover nine eligible replacements.

Engines differ in how approximate nearest-neighbor search interacts with filters, so measure recall under your actual tenant and permission distribution.^R4

If your architecture requires broader internal retrieval, keep those candidates inside the trusted retrieval service, authorize them before any external reranker or model sees their text, and overfetch only within a controlled budget. Recheck permissions when necessary to cover stale index metadata.

04

Merge rankings without pretending their scores match

Now you have two lists. BM25 might assign a document 12.7 while the vector search returns a similarity of 0.82. Adding those numbers directly makes the result depend on incompatible score scales.

Reciprocal rank fusion, or RRF, combines list positions instead. Each document gets a contribution of 1 / (c + rank) from every list where it appears; ranks start at one. Missing from a list means no contribution. The constant c softens the difference between high and lower positions.^{R5 R8}

Here’s a toy example with c = 60. These are calculated illustrations, not measured search results. Change the ranks or the constant to see how fusion responds.

Candidate B wins because both lists support it. That agreement is helpful, but it doesn’t prove B answers the question. An exact-match route may still be needed for identifier lookups, and rank fusion cannot recover a document that neither branch retrieved.

The following small Python function illustrates fusion over lists of stable chunk IDs:

def rrf(rankings, c=60):
    scores = {}
    for ranking in rankings:
        seen = set()
        for rank, chunk_id in enumerate(ranking, start=1):
            if chunk_id in seen:
                continue
            seen.add(chunk_id)
            scores[chunk_id] = scores.get(chunk_id, 0.0) + (
                1.0 / (c + rank)
            )
    return sorted(scores, key=lambda x: (-scores[x], x))

The final ID sort makes ties deterministic. Deduplication within each list prevents repeated IDs from collecting extra votes. Choose candidate windows and the fusion constant using evaluation; don’t mistake an example default for a universal optimum.

To wire the branches together, the orchestration can look like this. search_bm25 and search_dense are application adapters, not SDK methods. Both must enforce the same trusted scope, including permissions and validated business filters, and return chunks with consistent IDs and versions. query_vector comes from your compatible embedding model.

import asyncio

async def hybrid_candidates(query, query_vector, scope):
    lexical, semantic = await asyncio.gather(
        search_bm25(query, scope=scope, limit=50),
        search_dense(query_vector, scope=scope, limit=50),
    )
    by_id = {hit["id"]: hit for hit in lexical + semantic}
    ranked_ids = rrf([
        [hit["id"] for hit in lexical],
        [hit["id"] for hit in semantic],
    ])
    return [by_id[chunk_id] for chunk_id in ranked_ids[:40]]

The shortlist now has one entry per chunk, ready for reranking. Handle an empty list before invoking the reranker. Adapters should reject an invalid scope rather than retrying without its filters.

05

Reranking asks a more focused question

Fusion asks how well a candidate placed across retrieval methods. A reranker looks more directly at how a candidate relates to the query.

A common design uses a cross-encoder: it processes the query and a candidate together, allowing detailed interaction between their words. That’s more expensive than retrieving precomputed document vectors, so you usually apply it to a shortlist.^R6

Try retrieving 50 candidates from each branch, fusing the union, and reranking the top 40. Then assemble perhaps six to ten useful passages within a token budget.

Cohere Rerank is one hosted option. Its v2 API accepts a query, document strings, a model identifier, and a requested result count. Returned indices map back to the documents you submitted.^R7

import os
import cohere

# candidates: authorized chunks in their original order
client = cohere.ClientV2(api_key=os.environ["COHERE_API_KEY"])
response = client.rerank(
    model=os.environ["COHERE_RERANK_MODEL"],
    query=query,
    documents=[c["text"] for c in candidates],
    top_n=min(8, len(candidates)),
)
selected = [candidates[r.index] for r in response.results]

This fragment assumes a nonempty shortlist and a model available to your account. Add a deadline, error handling, and an empty-result branch in the application. Only send content to a hosted service if your data-handling policy permits it.

Keep identifiers alongside the submitted strings. Don’t sort the candidate array before mapping returned indices, or your citations may point at the wrong text. Also watch document length: the API documents truncation limits, so a decisive sentence at the end of an oversized passage may never be considered.^R7

06

Assemble evidence instead of filling a bucket

The top eight chunks may all repeat the same paragraph from one incident. Passing them all to the generator spends tokens without adding much evidence.

Deduplicate overlapping passages, retain their provenance, and include the surrounding section when a fragment depends on it. For a comparison question, seek coverage of both sides. For a troubleshooting question, distinguish symptoms, the proposed fix, and confirmation that the fix worked.

You also need a way to stop. If the retrieved material doesn’t establish whether SSO was enabled before the login loop began, say that evidence is missing. Nearby text about SSO doesn’t establish the sequence of events.

For exhaustive requests, retrieval needs another route entirely. “How many unresolved incidents meet these criteria?” requires an exact count over the eligible records. “List all of them” needs pagination or an export. A top-k semantic shortlist can support explanation; it cannot justify a claim that every match was found.

07

Add an agent only where another step can help

An agentic retrieval system can choose tools, reformulate a query, or fetch follow-up evidence based on what it finds. That flexibility is valuable when the next step genuinely depends on the previous result.

For example, the first search finds the error code but lacks the deployment version. The agent can read the authorized incident record, extract the version, then retrieve the relevant runbook section. An explicit state machine can implement the same controlled workflow; you don’t need open-ended autonomy to benefit from multiple steps.

Keep the non-negotiable filters in application code. A query rewrite can improve wording, but it must not drop tenant restrictions, broaden access, or turn “unresolved” into “all incidents.”

Stop sooner when evidence is sufficient. If the remaining gap is ambiguity in the user’s question, ask. More searching won’t decide which business definition they intended.

08

Measure the stage that actually failed

Start with a labeled set of real questions: exact codes, paraphrases, date ranges, joins, multilingual phrasing, ambiguous requests, and questions with no answer. Include access-control cases as a separate, mandatory evaluation.

Compare lexical-only retrieval, dense-only retrieval, hybrid fusion, and hybrid plus reranking on the same set. Then inspect three different outcomes:

• Candidate recall: Did the eligible evidence enter the candidate set? Recall@K measures how much of the labeled relevant evidence appears in the first K results.
• Ranking quality: Did useful evidence reach the top? Metrics such as nDCG@K account for graded relevance and position; reciprocal-rank metrics help when finding one first useful result matters.
• Answer quality: Does the answer follow the evidence, cite the right source, satisfy the filters, and abstain when support is missing?

Track latency at the 95th percentile, cost per request, index freshness, and failure rates alongside quality. Looking at the stages separately tells you where to spend effort.

At larger corpus sizes, an approximate nearest-neighbor index can avoid an exhaustive vector scan, trading some recall for speed. Tune that tradeoff on your workload rather than assuming the closest results will always be found.^R2

Run independent retrieval branches concurrently. Cache with keys that include the relevant identity or permission scope, filters, and corpus version. Give reranking a timeout and decide whether an authorized fused shortlist is an acceptable fallback for the use case. If it isn’t, return an explicit limitation. A missing authorization decision must never become a reason to search more broadly.

09

Make the system answer the actual question

Return to our support request. The system should resolve access, region, tier, status, and time first; search for the permitted code or symptom; fuse and rerank the eligible evidence; then explain what the records support.

The practical payoff is easy to recognize. A reader gets the incident they needed, understands why it matched, and can follow the citation. Your team can explain how it was selected and diagnose what happened when selection failed.

That’s a much more useful target than making cosine similarity look impressive in a demo.