Inference cost definition

Inference cost in AI is the price of running a trained model on real requests, charged per token, per call, or per GPU-hour, and it recurs for the life of a deployment rather than ending when training does. The cheapest token is the one you never send, and structured content is what lets you not send it. In Sanity, a GROQ projection returns only the fields named in the query, so a classification step can be handed the two fields a question concerns instead of a whole rendered page.

How is inference cost billed?

Inference cost is billed in one of three shapes, depending on how you reach the model. The most common is per token, usually quoted per million tokens and split so that input tokens (everything in your request) and output tokens (everything the model generates) are metered separately. Output is typically priced higher than input, because generating text happens one token at a time and costs more compute per token than reading a prompt does.

The second shape is per request, per image, or per minute of audio, used by some endpoints and by products that resell model access on top of their own margin.

The third is GPU-hours or instance-hours, which is what you pay when you host a model yourself. Here you are buying time on hardware whether or not it is busy, so utilization becomes the hidden variable: the same workload has a completely different cost shape depending on how full you keep the machines.

A fourth category is worth knowing because it changes the arithmetic. Several providers now meter cached input separately from fresh input. OpenAI's prompt caching documentation describes billing across uncached input, cache write, and cached input (cache read), each at a different rate, and notes that cache-write pricing is not an additive fee: input tokens use the uncached-input, cached-input, or cache-write rate. So "input and output" is still correct, but "input, cached input, and output" is closer to how a modern bill is actually itemized.

What drives inference cost up?

Inference cost is driven by how many tokens move through the model and how many times you ask. The main factors, roughly in the order they tend to matter:

Prompt and context length. Every token in the request is billed, including the system prompt, tool definitions, retrieved documents, and the whole conversation so far. Output length. Billed separately and usually at the higher rate, and on models that emit reasoning or thinking tokens, those count as output too. Model size. A larger model costs more per token for identical work. Calls per task. An agent makes many model calls to finish one job, so the meaningful unit is cost per completed task, not cost per call. Conversation depth. Each turn resends the accumulated history; Anthropic's cost optimization guide puts it plainly: "A 40-turn task sends its first turn 40 times, so task cost grows with roughly the square of turn count." Retries. Failed parses, schema violations, timeouts, and guardrail rejections are all paid for at least twice. Cache misses. A reused prompt prefix is billed at a reduced rate, but a changed prefix pays full price for everything after the change, which is why a volatile token near the front of a long system prompt is expensive out of proportion to its size. Idle hardware, if you self-host, since reserved GPU time bills the same whether it is working or not.

How do you reduce inference cost?

You reduce inference cost by sending fewer tokens, sending them to a cheaper model, or not sending them at all. Seven levers, from the ones that cost nothing in quality to the ones that trade some away.

First, trim the context. Send the fields a question concerns rather than the whole document or page. This is the largest saving available to most content-driven systems, and it costs nothing in output quality because the removed tokens were never relevant.

Second, cache the stable prefix of the prompt so repeated context is billed at the reduced rate, and keep anything volatile (a timestamp, a session ID, a status line) out of the front of that prefix where it would invalidate everything behind it.

Third, route per request instead of sending everything to one model, reserving the expensive model for the requests that need it.

Fourth, use a smaller or specialized model for routine, high-volume steps.

Fifth, batch work that can tolerate delay. OpenAI's Batch API is an asynchronous tier offered at a lower rate with a 24-hour turnaround, aimed at evals, classifying large datasets, and embedding content repositories.

Sixth, cap output length, since output is the dearer half of the bill.

Seventh, return a decision rather than generated prose where a decision is all you need. A model that emits a label or a probability over a fixed set of answers has no sequence of output tokens to pay for. System One models, a category whose first released example is TypeSafe AI's Jev (15 September 2026), work this way: you pass state plus typed questions and get back probabilities over a fixed answer space, with no text. A guaranteed schema-valid result also removes a class of parse-and-retry code, and retries are one of the drivers in the first place.

What is the difference between inference cost and training cost?

Training cost is the one-time expense of producing a model's weights; inference cost is the recurring expense of using those weights on live traffic. The difference matters because the two are optimized by entirely different means. Training is improved through better data, better architecture, and hardware procurement, and it ends. Inference is improved through prompt design, routing, caching, and model selection, and it never ends: it scales with every user, every feature, and every retry for as long as the product runs.

The practical consequence for anyone budgeting a product is that inference cost is the line item that grows with success. A feature that works well gets used more, and getting used more is exactly what makes it expensive. There is an industry-level argument that spend is shifting from training toward inference overall, discussed by McKinsey and TechTarget among others, though how that plays out for any individual team depends entirely on whether they train anything at all.

Why is token price not the same as inference cost?

Token price is the sticker; inference cost is the bill, and a model with the lower sticker can easily produce the larger bill. A cheaper model that needs three attempts, a longer prompt, or more back-and-forth to reach the same result can cost more per finished job than an expensive model that gets there in one call. Anthropic's cost optimization guide states the rule directly: compare models "on cost per completed task, not per token."

Two related confusions are worth clearing up. Inference cost is not the same as latency: they share drivers, but asynchronous batch tiers make the same work cheaper and slower at once, while prompt caching makes it cheaper and faster at once. And inference cost is not the same as the context window, which is a limit on how many tokens fit in one request rather than a statement about what they cost. Filling a large window is permitted and expensive at the same time.

Finally, the API line item is not the whole cost of ownership. Self-hosting adds reserved capacity, idle time, ops staffing, and egress; hosted APIs add gateway, observability, and evaluation spend.

How does structured content affect inference cost?

Structured content lowers inference cost by making "send only the relevant part" an addressing problem rather than a parsing problem. When content lives as a rendered page, the smallest thing you can hand a model is the page, and you pay input tokens for navigation, boilerplate, and markup that have nothing to do with the question. When content lives as typed fields with references to taxonomy documents, you can address the two fields a question concerns and send those.

Sanity is the AI Content Operating System, and the mechanism here is GROQ, its query language, whose documentation describes the goal as being able "to describe exactly what information your application needs, potentially joining together information from several sets of documents, then stitching together a very specific response with only the exact fields you need." A projection such as selecting an ID, a title, and a release year returns three fields rather than a whole document, and the input token count falls accordingly.

The second effect is on the answer space. References to taxonomy documents give a routine classification step a bounded, enumerable set of valid answers, which is what makes it reasonable to hand that step to a small model with a short prompt instead of a frontier model with a long one. Both effects point the same way: a clean, bounded input is a cheap input.

Last updated: October 8, 2026