time to first token
In plain wordsHow long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.
The interval between a request's arrival and the delivery of the first generated tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. of its response. It comprises queueing delay, the prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. computation over the whole prompt (growing linearly with prompt length in most layers and quadratically in attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.), generation of the first token, and network transfer. Because streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end. displays tokens as they are produced, TTFT governs how responsive a chat or voice interface feels, while time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow. governs how fast the rest of the answer arrives. It is reduced by prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper. of repeated prefixes, faster prefill hardware, scheduling that prioritizes newly arrived requests, and disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them.. For reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning., which may think silently before answering, some benchmarks report a separate time to first answer token.
With the 50,000-token contract cached, TTFT dropped from four seconds to under one.
Origin
In use by 2023 as LLM serving matured: Databricks' October 2023 guide to inference performance defined it alongside time per output token, and serving research such as DistServe (2024) adopted the TTFT/TPOT pair as its latency targets.
Related
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
- time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end.
- tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant.
- prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper.
Sources
- Agarwal, M. et al. (2023). LLM Inference Performance Engineering: Best Practices. Databricks blog. databricks.com/blog/llm-inference-performance-engineering-best-practices
- Zhong, Y. et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. arXiv:2401.09670. arxiv.org/abs/2401.09670
Mentioned in
latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system., model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., prompt cachingprompt caching, n.: Saving the work a model did reading the start of a prompt, so later requests that begin the same way can skip that reading and come back faster and cheaper., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., voice agentvoice agent, n.: An AI system you talk to out loud, for example on a phone call, that listens, answers in a natural-sounding voice and can carry out tasks such as booking appointments or checking an order., chunked prefillchunked prefill, n.: A serving trick that splits the reading of a long prompt into smaller pieces and slots them between other users' replies, so nobody's answer freezes while a huge document is read., disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough., service-level objectiveservice-level objective, n.: A specific, measurable target for how well a service should perform — for example, 99.9 percent of requests answered within two seconds — used to decide when engineers must act., time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.