Embedding Drift: Detecting When “Meaning” Moves

Meaning moves, and it moves quietly

An embedding model turns language into a point in a few hundred or a few thousand dimensions, and your whole retrieval stack assumes those points stay put. They do not. When a vendor ships a new embedding version, when you re-chunk a corpus from 512 to 256 tokens, when a domain vocabulary shifts because the business entered a new market, the geometry of the space moves under you. The same question that used to land next to the right policy document now lands one neighborhood over, and the retriever returns something plausible but wrong. The vector index is still healthy, the query still returns k results in the same 40 milliseconds, and every dashboard you own stays green.

That is what makes drift dangerous: nothing throws an error. A finance assistant that once resolved "beneficial owner" to the KYC handbook starts pulling the marketing glossary, and relevance at the top rank quietly falls from 0.91 to 0.78 over three weeks. Nobody notices until a user escalates or an auditor asks why a regulated answer cited the wrong source. By then the drift has been shipping wrong context to a language model for a month, and you have no baseline to prove when it started or how far it went.

Drift itself is inevitable. Models improve, corpora grow, and the language your users type evolves. What is optional is the silence. The entire discipline here is to convert an invisible, gradual shift into a small number of numbers you watch on every deploy, so a change of meaning shows up as a moved needle rather than as a support ticket. You do not prevent drift; you make it observable, attributable, and reversible before it reaches a customer.

Four probes and a trigger policy

You do not need a research pipeline to detect drift. You need four lightweight signals, each cheap to compute and each catching a different class of movement, plus a trigger policy that says what happens when a signal fires. Canary queries catch relevance regressions on the questions that matter most. Centroid tracking catches whole intents sliding. Anchor pairs catch two concepts collapsing into one. Distribution checks catch a global shift from a model or pipeline swap. The table below maps each probe to what it catches, a concrete metric, and a threshold you can start with and tune per intent.

The probes are cheap because they reuse work you already do. Canaries are just a saved eval run. Centroids are a mean over 30 to 50 seed vectors you recompute nightly. Anchor pairs are two cosine calls. None of it needs a GPU or a labeled test set beyond a couple of hours of curation, and all of it runs on a cron in seconds. The discipline is not the math; it is committing to run the probes on every deploy and to stamp every embedding, index, and centroid with a model version ID so you can always compare today to a known-good yesterday.

Take a worked example. A support-search team runs 20 canary queries nightly. After a routine embedding-provider upgrade from v2 to v3, MRR on canaries drops from 0.88 to 0.79, and the centroid for the "billing dispute" intent moves 0.11 in cosine distance while the "billing question" centroid barely moves. The anchor pair "chargeback" versus "charge" narrows from a 0.22 gap to 0.09. Three probes agree: v3 has collapsed a distinction the old model preserved. Because the change is caught in shadow mode before cutover, the team keeps v2 live, adds a rerank step, re-tests, and only then promotes v3, avoiding a live regression that would have hit roughly 4,000 daily searches and every downstream answer built on them.

Stand up drift detection in a week

• Build 15 to 30 canary queries from real escalations and high-volume tickets, store the expected top document IDs, and evaluate them on every deploy and on a nightly cron so regressions surface within a day.
• Compute a baseline centroid for each of your top 5 intents from a curated seed set of 30 to 50 documents, snapshot it with a model version ID, and track cosine distance against it over time.
• Define 10 anchor pairs, 5 positive and 5 negative, that encode distinctions you cannot afford to lose, and alert when any pair narrows past 30% or inverts sign.
• Add a single drift_score panel to your retrieval dashboard that shows canary MRR, the worst centroid delta, and the tightest anchor gap, so drift is one glance away for anyone on call.
• Write a trigger policy with named steps and owners, from warn through shadow test, rerank, re-embed, and roll back, and never upgrade an embedding model without a shadow window of at least a week.

Where drift programs go wrong

• Offline metrics only: batch benchmarks look great while live quality slides. Fix: run canaries against the production index on a schedule, not just in CI.
• No preserved baselines: you upgrade the model and cannot compare because the old index is gone. Fix: keep the previous embeddings and centroids checkpointed with version stamps for at least two releases.
• One threshold for everything: a single drift bar treats a marketing FAQ like a regulated KYC intent. Fix: set per-intent sensitivity, tighter on high-stakes topics.
• Overfit canaries: a frozen golden set stops representing real traffic and goes green while users suffer. Fix: refresh 20% of canaries quarterly from recent incidents under review.
• Detection without remediation: alerts fire and nobody knows what to do. Fix: bind every threshold to a named action and owner in the trigger policy, and log the remediation and its measured effect.

Five moves to make this week

• Create 15 canary queries from recent escalations and wire them to a nightly run.
• Compute and version-stamp centroids for your top 5 intents.
• Stand up 10 anchor pairs and alert on any inversion or 30% gap narrowing.
• Add a drift_score widget to the retrieval dashboard with canary, centroid, and anchor views.
• Write the trigger policy: warn, shadow test, rerank, re-embed, roll back, each with an owner.