Time to first token (TTFT) definition

Time to first token (TTFT) is the delay between sending a request to a generative model and the first token of the response arriving, which makes it the main driver of how responsive a streaming interface feels. One component of that delay is prefill, the cost of processing everything in the prompt. In Sanity, content is stored as discrete fields and blocks, so an application can pass the exact block a question concerns instead of a whole rendered page, which reduces the text sent to prefill.

What is time to first token (TTFT)?

Time to first token (TTFT) is the time between a client sending a request to a generative model and that client receiving the first token of the response. A token is the small chunk of text (roughly a word fragment) that a language model emits one at a time. Because streaming interfaces show output as it arrives, TTFT is the length of the silence the user waits through before anything appears on screen.

IBM describes the interaction as two phases, one with no visible activity and one where generation has started, with TTFT measuring how long the transition takes. Its analogy is useful: in conversation, a delay before someone starts speaking is more noticeable than pauses later on. The AWS Well-Architected Agentic AI Lens puts it plainly, calling TTFT "the dominant perceived-performance signal."

TTFT only exists as a separate number when the response is streamed. If the server returns the whole answer as one block, there is no first token to observe early, and TTFT collapses into total latency. The benchmarking tool GuideLLM handles this by excluding objectives that cannot be evaluated, such as time to first token on a non-streaming backend.

How is TTFT different from total latency and tokens per second?

Time to first token (TTFT) measures the wait before generation starts, while end-to-end latency measures the wait until the complete response has arrived, and tokens per second measures the speed of generation in between. NVIDIA states the relationship directly: end-to-end request latency equals time to first token plus generation time.

The rate metrics go by several names. Inter-token latency (ITL) and time per output token (TPOT) both describe the gap between consecutive output tokens after the first one. NVIDIA defines inter-token latency as averaging the time between consecutive output tokens, excluding the first. IBM notes the practical consequence: a model can decode each subsequent token quickly and still feel sluggish if the initial delay is high.

TTFT is not prefill latency. Prefill latency covers only the model's prompt-processing phase. TTFT also includes queue wait, network transport, the first sampling step, and any gateway work inside your measurement boundary. Prefill is often the largest single component, but it is still only a component.

Tracking only total latency hides regressions users notice. AWS makes this point and lists treating TTFT as interchangeable with end-to-end latency in KPIs and alarms as a named anti-pattern, noting that a faster model reduces total duration but leaves TTFT unchanged when the pre-inference pipeline is the bottleneck.

What makes time to first token slow?

Time to first token (TTFT) is the sum of several delays, and IBM decomposes it as queueing delay, prefill time, initial decoding time, and network and API overhead. Each delay has its own fixes.

Network and transport. Every request pays a round trip to the endpoint, plus connection setup and TLS. The model does not change this.

Queueing. Under load, requests wait before they are served. IBM notes this delay is influenced by concurrency and the system's ability to manage different workloads. The IETF benchmarking terminology draft gives it vocabulary: queue depth, queue wait time, and head-of-line blocking.

Prompt length, through prefill. Before the first token can be sampled, the model computes attention across the entire prompt and builds a key-value cache for reuse during decoding. IBM calls prefill often the largest contributor to TTFT, especially for longer prompts, and NVIDIA says TTFT grows with longer prompts because of that cache computation. NVIDIA's list of what actually makes prompts long is worth memorizing: the user query, the system prompt, chat history, chain-of-thought reasoning, and documents retrieved by a RAG pipeline.

Model size. Larger models need more computation per step, as both IBM and OpenAI's latency guide note.

Prompt caching. Reusing a precomputed cache for a shared prompt prefix skips that part of prefill. The practical rule from OpenAI's guidance is to put stable, reusable content at the start of the prompt and dynamic content such as retrieval results later.

How do you measure TTFT properly?

Measuring time to first token (TTFT) takes four things: a declared measurement boundary, percentiles instead of averages, a stated cold or warm condition, and a stated workload.

First, declare where you are measuring. The IETF draft defines three boundaries for the system under test: the model engine (excluding network transport, authentication, routing, and safety filtering), the application gateway (the engine plus request handling, authentication, rate limiting, input validation, and output filtering, reflecting what users observe when calling an API endpoint), and the compound system (a gateway plus orchestration, retrieval, tool execution, and multi-step reasoning). The draft says testers must declare the boundary and must not directly compare metrics from different boundaries without adjustment. Note the status: this is an Internet-Draft, work in progress, not an RFC.

Second, report percentiles. An average hides the tail users complain about. GuideLLM treats percentiles and confidence intervals as first class, and writes objectives in a form such as "TTFT under 200ms for 99% of requests," which is an illustration of how to phrase an objective rather than a target anyone should assume.

Third, separate cold from warm. A first request with a cold cache, a cold connection, and a possibly cold autoscaled worker is a different measurement from steady state. Both the IETF draft and GuideLLM define explicit warm-up and measurement windows.

Fourth, state the workload. NVIDIA warns that concurrency, maximum batch size, request rate, and sampling settings all change the measured numbers, and that client-side tools differ in how they define and calculate metrics, which makes results from different tools hard to compare.

Why do agent pipelines have bad TTFT?

Agent pipelines have poor time to first token (TTFT) because every model call in the chain pays its own TTFT, and serial handoffs add their full duration to what the user waits for. AWS states that multi-agent pipelines amplify the TTFT problem when every upstream agent must fully complete before the user-facing agent begins, and names blocking those workflows until every upstream agent finishes as an anti-pattern that turns coordination delay into user-visible latency.

Tool use adds a second kind of stall. AWS describes an event stream that opens a tool-use content block, streams the tool's input as deltas, and then pauses while the agent runs the tool and feeds results back. A client that gets no signal during that gap shows the user partial output followed by silence.

AWS also makes the point that matters most for application teams: by the time the first token leaves the model, an agent can spend the entire TTFT budget on work that happens before inference begins, including context assembly, prompt construction, retrieval, tool execution, and serial pre-checks, and streaming recovers nothing from that window. The biggest wins, AWS says, come from the pre-inference path: compressing retrieval, narrowing retrieved context, and parallelizing independent steps.

The standard mitigations are structural, not model-level. OpenAI's latency guidance suggests combining sequential steps into a single prompt to avoid extra round trips, and splitting steps that are not strictly sequential into parallel calls.

Do all models have a time to first token?

No. Time to first token (TTFT) is a metric for autoregressive models, the ones that generate output one token at a time with each token conditioned on the ones before it. The IETF benchmarking draft is explicit that its metrics apply to transformer-based autoregressive language models, and that other architectures such as diffusion models, encoder-only models, and non-autoregressive decoders need different terminology.

Models that return a decision rather than generating text are in that other category. A System One model, for example, takes state plus typed questions and returns probabilities over a fixed answer space with no prose. There is no stream, so there is no first token to wait for, and end-to-end latency is the only meaningful measure of how fast it responded.

The practical consequence for anyone comparing systems is that TTFT and end-to-end latency are not interchangeable across architectures. Comparing a streaming model's TTFT against a decision model's total response time compares two different quantities. Where responses stream, track TTFT and generation rate separately. Where they do not, one number covers it.

TTFT is also separate from what a request costs. Prompt caching happens to improve both latency and inference cost, which is why the two get conflated, but a cheap request can be slow and a fast one can be expensive.

Last updated: October 7, 2026