service-level objective
In plain wordsA specific, measurable target for how well a service should perform — for example, 99.9 percent of requests answered within two seconds — used to decide when engineers must act.
A reliability target: the level that some measured quality of a service, such as availability, latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system. at a given percentile or error rate, should meet over a stated period. In site reliability engineering, the measured quantity is the service-level indicator (SLI), the SLO is the internal goal for it, and a service-level agreement (SLA) is a contractual promise to customers, usually looser than the SLO and backed by penalties such as service credits. The shortfall an SLO permits is the error budget: a service with a 99.9 percent monthly availability target may be down about 43 minutes a month, and teams slow risky releases when the budget is spent. For language-modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. serving, typical SLOs cover time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear., time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow., throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second., availability and the share of rate-limitedrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later. requests, and they drive capacity planning, batching policy and routing.
Our SLO is a p95 time to first token under 800 milliseconds for paid users.
Origin
Service-level agreements and objectives come from telecommunications and IT service management of the 1980s and 1990s, later codified in frameworks such as ITIL. Google's book Site Reliability Engineering (2016) popularized the SLI–SLO–error-budget vocabulary, which teams serving machine-learning models adopted.
Usage note
People often say SLA when they mean SLO. Some teams also set quality SLOs, such as a minimum evaluation score or a maximum hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true. rate, which are harder to measure continuously than latency or availability.
Related
- latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.
- model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
- model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
- rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later.
- throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.
- time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.
Sources
- Beyer, B., Jones, C., Petoff, J. & Murphy, N. R. (eds.) (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media.
Mentioned in
model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm., rate limitrate limit, n.: A cap on how much a customer may use an online AI service in a given time — so many requests or words per minute — beyond which extra requests are refused until later., canary deploymentcanary deployment, n.: Releasing a new model or software version to a small slice of users first and watching closely, so that any problem hits only a few people before the change reaches everyone., goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.