TensorRT-LLM

NVIDIA's open-source engine for serving large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. as fast as possible on its own GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.. Much of the speed comes from model code written against hand-tuned, fused CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI. kernels, including FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip.-style attention; for serving it adds paged KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. management, in-flight batching (NVIDIA's name for continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish.) and speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., supports quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. down to INT4 and FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory. as well as INT8 and FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits., and splits large models across GPUs and nodes with tensor, pipeline and expert parallelism. It began as a compiler-style workflow that built an optimized TensorRT engine for each model and hardware configuration, and later added a PyTorch-based runtime for faster model support. It often leads raw-speed comparisons on NVIDIA hardware at the cost of portability, and serves as a back end for NVIDIA's Triton Inference Server and Dynamo serving stack.

Moving the model from vLLM to TensorRT-LLM with FP8 gave us another 30 percent throughput on H100s.