LiteRT

An open-source inference engine from Google that runs trained models on the device itself rather than on a server; it was called TensorFlow Lite until 2024. Models are converted into a compact FlatBuffers file (.tflite), usually with quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. to 8-bit integers or 16-bit floating point, and executed by a small interpreter on Android, iOS, embedded Linux and the web, and, in a reduced 'Micro' variant, on microcontrollers with only kilobytes of memory. Delegates hand supported operations to GPUs, digital signal processors and NPUsNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., including Google's Edge TPU. Although it began as part of TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems., LiteRT accepts models converted from PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips. and JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips. through Google's AI Edge tools, and it underlies higher-level offerings such as MediaPipe and on-device generative-AI interfaces. It is among the most widely deployed inference engines, powering vision, speech, text and recommendation features in mobile applications.

We quantized the keyword model to int8 and shipped it as a LiteRT file on the microcontroller.