Dumping every document into one flat vector index and hoping cosine similarity sorts it out leaves most of your signal on the table. A contract is not a blog post, it has parties, clauses, and cross-references, and a support corpus mixes docs, tickets, and release notes with different vocabularies. A single general embedder averages across tasks it should handle separately, and precision suffers. Small, task-specific adapters over a shared base model route and rank far better, lifting retrieval precision without the cost or lock-in of one monolithic model. When your corpus has structure, stop flattening it away.
Flat vector search wastes the structure you already have
The default retrieval design is one embedding model, one index, one similarity search. It is simple and it works until the corpus has structure that the flat index flattens away. A contract is not a blog post: it has parties, clauses, defined terms, and cross-references. A support corpus mixes product docs, tickets, and release notes, each with different vocabulary and different notions of relevance. A single general embedder treats all of it as undifferentiated prose, and precision suffers because the model is averaging across tasks it should be handling separately.
Small adapters offer a middle path between one monolithic model and a fleet of full fine-tunes. You keep a single frozen base encoder and train tiny task-specific heads (LoRA-style adapters, often under 1 percent of the parameters) for routing and ranking. Each adapter is cheap to train, cheap to swap, and specialized for one slice of the retrieval problem, so the structure in your corpus becomes signal instead of noise.
The reason a flat index underperforms is subtle. Cosine similarity in a general embedding space rewards surface-level lexical and topical overlap, so a release note that shares vocabulary with a legal query can outrank the clause that actually answers it. The structure you would use to disambiguate, document type, party, effective date, product line, is sitting in metadata the flat search never consults. Adapters exist to put that structure back into the ranking.
Route by type, then rank with a specialized head
The pattern is two-stage. A lightweight router adapter classifies the query and selects the right index or sub-corpus, then a ranking adapter tuned for that slice reorders the candidates. Because adapters share the frozen base, you pay for one encoder in memory and load only the small deltas per task. The table contrasts the flat baseline with the structured-adapter approach on the dimensions that decide production fit.
| Dimension | Flat single index | Small adapters | Practical effect |
|---|---|---|---|
| Precision@5 | Averaged across all types | Tuned per document type | Typical lift of 8-15 points |
| Training cost | Full fine-tune to specialize | Under 1 percent of params per head | Hours, not days, per slice |
| Serving footprint | One large model | One base + small deltas | Swap adapters without reload |
| Adding a new type | Retrain the whole embedder | Train one new adapter | Ship incrementally, low risk |
| Metadata use | Ignored or bolted on | First-class routing signal | Fewer cross-type false matches |
The lift compounds because errors are correlated. Cross-type false matches, a release note surfacing for a legal query, are exactly what a router removes, and they tend to be the most damaging results a user sees.
Work it through on a support corpus of 120,000 documents split across product docs, resolved tickets, and release notes. The flat baseline embeds everything into one index and returns precision@5 of 0.61 on a labeled query set, with roughly a third of the misses being cross-type: a query about a billing error surfacing a release note that merely mentions billing. Add a small router adapter, a 3-class query classifier trained in about two hours on a few thousand labeled queries, that sends "how do I" questions to product docs and "it broke" questions to tickets. That alone lifts precision@5 to 0.70 by cutting cross-type false matches. Then add a per-slice ranking adapter, each well under 1 percent of the base parameters, tuned on relevance labels for that slice, and hard-filter by product and version from metadata rather than hoping the vector encodes them. Precision@5 climbs to 0.74, a 13-point gain over the flat baseline, while memory stays flat because all three adapters share one frozen encoder and swap in as small deltas. No full fine-tune, no second large model, just structure turned back into signal.
What makes this economical is the shared frozen base. Because the router and every ranking head are small deltas over one encoder, a new document type does not force a retrain of the whole system; you train a fresh adapter in an afternoon, validate its precision against the flat baseline for that slice, and load it beside the others with no additional model in memory. The result is a retrieval stack that grows the way the corpus does, one structured slice at a time, instead of a single monolithic embedder you dare not touch because every change risks the tasks that were already working.
Add adapters where the corpus structure is real
- Segment the corpus by the structure that already exists, document type, source system, or clause category, and confirm each segment has a distinct relevance notion before adding a head.
- Train a small router adapter first. Even a 3-5 class query classifier that picks the right sub-index removes the worst cross-type errors cheaply.
- Keep the base encoder frozen and shared. Train ranking adapters as LoRA-style heads so each stays under a percent of parameters and swaps in without a reload.
- Feed structured metadata (party, date, product, status) as explicit filters alongside the vector score, not as text hoping the embedder notices it.
- Measure precision@5 and precision@10 per segment against the flat baseline, and only keep an adapter that clears a real margin, for example 5 points or more.
Where adapter retrieval goes wrong
- Adding adapters where the corpus has no real structure. Fix: verify segments have distinct relevance before splitting; a homogeneous corpus does better flat.
- Letting adapters diverge onto different base encoders. Fix: pin one frozen base so vectors stay comparable and memory stays flat.
- Ignoring metadata and hoping the vector captures it. Fix: apply party, date, and type as hard filters combined with the similarity score.
- A router that misroutes silently. Fix: log routing decisions, alert on low-confidence classifications, and fall back to searching all indexes when unsure.
- Evaluating only global average precision. Fix: report per-segment precision@k, since a global average can hide a slice the new adapter made worse.
Confirm the adapters are earning their place
- Corpus segments each have a distinct, verified notion of relevance.
- A router adapter selects the sub-index and logs low-confidence routes.
- All adapters share one frozen base and swap without a reload.
- Structured metadata is applied as filters, not left to the embedder.
- Precision@5 is measured per segment and each adapter clears a real margin.