inference engine

Expert systems

The general-purpose reasoning program inside an expert systemexpert system, n.: A program that stores a human specialist's know-how as many if–then rules and uses them to give advice or make decisions in a narrow field, such as diagnosing infections or configuring computers. or rule-based systemrule-based system, n.: A computer system that makes decisions by following a list of 'if… then…' rules written by people, rather than by learning from examples., which matches the rules of a knowledge baseknowledge base, n.: A store of facts and rules that a program can reason with: like a database, but organized so that new conclusions can be worked out from it, not just looked up. against the facts at hand and fires those that apply, working by forward chainingforward chaining, n.: A way of reasoning that starts from known facts and keeps applying rules to them, adding each new conclusion, until it reaches an answer or nothing new can be concluded. (from data toward conclusions), backward chainingbackward chaining, n.: A way of reasoning that starts from the question ('is this true?') and works backward, looking for rules that would prove it and then checking whether their conditions hold. (from goals toward supporting facts) or a mix. It also performs conflict resolution among applicable rules, may propagate uncertainty measures, and records the chain of rule firings so that the system can explain how it reached an answer. Separating a general-purpose inference engine from domain knowledge made reusable expert system shells possible, such as EMYCIN, which was MYCIN with its medical rules removed.

We swapped in a new rule set without touching the inference engine.

Model serving

Software that executes a trained neural network to produce outputs, especially one optimized for serving large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.: it loads weights, schedules requests, manages the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., and applies techniques such as continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. and speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality. to raise throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second. and cut latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system.. Examples include vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., SGLangSGLang, prop. n.: A free, open-source engine for running AI language and image models on servers, built to be fast when many requests begin the same way, as in chat and agent workloads., llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models. and TensorRT-LLM. Here 'inference' means running a model forward, not logical deduction.

Switching inference engines doubled our tokens per second on the same GPUs.

Probabilistic reasoning

An algorithm or library that computes posterior probabilities or most probable explanations from a probabilistic graphical modelprobabilistic graphical model, n.: A way of describing how many uncertain quantities depend on one another by drawing them as a network, which makes it practical for a computer to reason about all of them together. or probabilistic programprobabilistic programming, n.: Writing a statistical model as an ordinary-looking program that includes random choices, and letting the system work out automatically which hidden values best explain the observed data., using exact methods such as belief propagationbelief propagation, n.: A way for a network of uncertain quantities to reach conclusions by having each node pass short messages to its neighbors about what it currently believes, until the beliefs settle down. and variable elimination or approximate ones such as Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability. and variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem..