RAG and data · 2026-09-22
Calculating two-stage retrieval cost with Voyage Rerank 2.5
Covers how the total token cost of retrieving initial candidates with embeddings and reranking them with Voyage Rerank 2.5 in a RAG pipeline is calculated, and how it lowers the token count sent to the language model.
Why rerank gives a different hit rate than embedding
Embedding search vectorizes the query and each document separately for speed; that makes a fast first pass over millions of documents possible, but it can miss a subtle semantic relationship between query and document. Voyage Rerank 2.5 reads the query and each candidate document TOGETHER and produces a relevance score; that gives a more accurate ordering but costs more than embedding since it requires a separate evaluation per candidate.
That's why the typical pattern is to run a cheap, fast first pass with embeddings to narrow candidates down to 50-100, then rerank that small set; reranking the entire collection directly raises cost unnecessarily.
The cost formula: query × document count + total document tokens
The model page states billing is based on total processed tokens: (query token count × number of documents evaluated) + the sum of tokens across all documents. That means the first term grows fast if your query is long and you're reranking many documents, while cost approaches the document token sum when reranking few documents with a short query.
A single request evaluates at most 1000 documents; narrowing the first pass to 50-100 candidates with embeddings not only improves hit rate, it also lowers the document count the query gets multiplied by, keeping rerank cost under control.
- Cost = (query tokens × document count) + total document tokens.
- Narrowing candidates to 50-100 with embeddings controls rerank cost.
- A single request evaluates at most 1000 documents.
The real savings show up on the language-model side
Although rerank costs more than embedding, sending only the most relevant few documents to the language model lowers the total token count the generation model processes (and therefore its cost). Weighing the rerank stage's own cost against the token cost you save on the generation model gives a more accurate picture than looking at rerank price alone.
Frequently asked questions
Can I use rerank alone without embedding?
Technically yes, but reranking the entire collection quickly becomes expensive due to the (query tokens × collection size) formula; narrowing with embedding first is recommended.
What happens with a candidate set larger than the 1000-document limit?
A single request evaluates at most 1000 documents; if your candidate set is larger, you need to apply a tighter pre-filter at the embedding stage to bring it under that limit.