Retrieval architecture for a low-traffic, high-scrutiny RAG assistant

Why this ADR exists

Almost every RAG tutorial reaches for a managed vector database in the first paragraph. That is the right call at scale, and it is why I nearly did the same thing here before working out what the index would actually weigh.

A vector store earns its keep when the corpus is large, changes independently of deploys, and gets queried often enough for the index to pay for itself. This corpus is small, it only changes when I publish, and it gets queried a handful of times a week. None of the conditions hold, so I went looking for what does fit.

The sizing argument

The whole decision turns on one calculation, done before any code was written:

~20,000 tokens of MDX
  ÷ ~167 tokens per chunk (semantic split, ~40 token overlap)
  = 120 chunks
  × 1024 dimensions × 4 bytes
  ≈ 480 KB of raw float32

Under half a megabyte. That fits in the smallest serverless runtime several times over, and comparing a query against 120 vectors is arithmetic that finishes in microseconds. The managed option would have spent more time on the network round-trip than the whole in-process scan takes.

I did not want this to rest on me remembering the limit, so the build checks it: npm run ingest fails above 4,000 chunks, roughly 33× the current corpus. When that fires, it is the trigger to move the retriever to pgvector.

Why hybrid rather than pure vector search

Dense retrieval on its own does badly on the questions this assistant actually gets. Someone asking "what did he use for rate limiting?" wants a specific term, and embeddings are lossy about precisely those rare tokens: proper nouns, library names, error codes.

So retrieval runs two rankings and combines them:

• BM25 over tokenised chunks, which is very hard to beat on exact terms and needs no model call.
• Dense cosine similarity over Voyage embeddings, which handles paraphrase and questions that share no vocabulary with the source text.

They are combined with reciprocal rank fusion rather than a weighted sum of scores. Summing scores would mean calibrating the two systems against each other, and BM25 scores and cosine similarities have no shared scale to calibrate on. RRF only looks at rank position, which avoids the problem and means I do not have to retune weights every time the corpus changes.

Why generation runs on a chain rather than one model

The first version of this called Claude directly, and the write-up said so. What changed is not a quality judgement — it is that the retrieval half is the part worth showing, and the retrieval half costs nothing to run. Paying per token so a stranger can watch a demo work was the only standing cost left in the system, and it was attached to the least interesting component.

So generation now tries providers in order and takes the first that answers: a free tier by default, a second free tier behind it, and metered providers last, where they only bill if both free tiers are down. The switch is reported in the trace rather than hidden, because a demo that quietly degrades is making a claim it has not earned.

Two things this cost me, both worth stating. Open-weight models hold citation discipline less reliably — one would not stop emitting 【2†L4-L9】 no matter how the prompt was worded, so the stream is normalised in code as well as asked for in the prompt. And ranking providers by cost rather than quality means the default answer is not the best answer available; that is the trade I chose, knowing the passages under it are identical either way.

What I would do differently at 100× the corpus

Move the index to pgvector, keep the hybrid strategy, and feed fusion from an ANN recall set instead of a brute-force scan. The retrieval interface is already shaped for that swap, which is why I drew the boundary where I did.

One caveat worth stating: I have not load-tested this past a single user, since a single user is the real workload. So the latency figures below tell you the pipeline is fast, not that it holds up under concurrency.