FinOps for AI: Calculating the Real Cost of Your RAG Pipeline

A RAG query looks small and is not. The user types one line, so the feature feels cheap, but every request quietly stuffs the retrieved passages into the prompt before the model writes a single word. Price a typical retrieval query of 6,000 retrieved tokens and a 200 token answer across the 42 models we track, and input is between 78 and 97 percent of the cost of that call, with a median of 86 percent. The part you are watching, the answer, is the small end of the bill.

That single ratio explains most of what follows. If roughly 86 percent of your spend is retrieved context, then the levers that matter are the ones that change how much context you retrieve and what you pay for it. Changing the model helps. Retrieving less helps more, and it is free.

Retrieve less before you do anything else

Most pipelines retrieve more than they need because the default top-k in the tutorial was ten and nobody revisited it. Here is what that costs. Recorded on August 10, 2026, GPT 5 Mini was $0.25 per million input tokens and $2.00 per million output. At 50,000 queries a month with a 200 token answer, the monthly bill moves like this with nothing changed but how much you retrieve:

Cutting retrieval from six passages to two takes the bill down by more than half, on the same model, with the same answers, at the same volume. Whether your answers survive that cut is an evaluation question rather than a cost question, and it is worth an afternoon to find out: reranking a wider candidate set and passing fewer, better passages usually improves the answers and lowers the bill at the same time. That is the rarest kind of change and it is sitting in most pipelines untouched.

Then cache the part that repeats

A RAG prompt has two very different halves. The instructions, the output schema, the tool definitions and any always-attached policy or style document are identical on every request. The retrieved passages change every time. Only the first half can be cached, so the saving depends on how large that fixed block is relative to the passages, which is a number you can measure today.

Of the 42 models we price, 28 publish a cache-read rate and 14 do not, and the median discount among those that do is 90 percent off standard input. For a pipeline with a large fixed preamble that is the difference between two invoices you would describe differently. The caching audit guide covers how to check what you are actually earning rather than what you hope to.

One structural note specific to RAG: put the retrieved passages after the fixed instructions, not before them. A prompt that opens with the passages has no stable prefix at all and can never cache anything, and this ordering mistake is common because it reads more naturally to a human.

The two layers we do not price

Generation is the layer this site can price, because published per-token rates are what we capture. Two other layers are real and are not in any figure here.

Embeddings are billed per token like everything else: once to index your corpus, and again for every query you embed. Per call it is small enough to ignore and at corpus scale it is not, particularly if you re-index often. A pipeline that rebuilds its index nightly is paying that indexing cost thirty times a month rather than once, which is a decision worth making deliberately rather than inheriting from a cron line.

The vector database sits outside token pricing altogether. It is priced like infrastructure, by storage and queries or by the instance it runs on, and it does not shrink when you retrieve fewer passages, so none of the savings above touch it. Get that number from your provider's bill and add it to the token figure by hand. A pipeline that is cheap on tokens and expensive on infrastructure is a common and expensive surprise.

Size yours, then check it against the invoice

To model a pipeline before it exists, the project estimator prices the document-question shape from plain answers, including what caching would save and what the same work costs across three model tiers. If you would rather work in tokens directly, the RAG Document Search preset in the advanced calculator loads an input-heavy scenario you can tune to your own chunk sizes and query volume.

Once it is running, stop estimating. Export a month of real usage and drop it into the bill auditor, which prices what you actually spent and re-prices the identical tokens on every other model we track. The gap between your estimate and that number is the most useful thing you will learn about your own pipeline, and in a RAG system it is almost always retrieval size, because that is the figure nobody remembers choosing.