edge AI
In plain wordsDoing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
Running AI models, mostly for inference but occasionally for training, on hardware located where the data arise or are used — sensors, cameras, vehicles, robots, industrial controllers, phones, gateways or on-premises servers — rather than in remote cloud data centersAI data center, n.: A building or campus packed with AI chips, power equipment and cooling, built to train and run AI models. The biggest use as much electricity as a city.. Motivations are low and predictable latencylatency, n.: How long you wait between asking an AI system for something and getting the answer. Lower latency means a snappier, more responsive system. (a car cannot wait for a round trip to the cloud), operation without reliable connectivity, lower bandwidth costs when raw video or sensor streams would otherwise be uploaded, and privacy or data-sovereignty requirements. Limits on power, memory, heat and cost drive the use of model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., efficient architectures and specialized NPUsNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud. and other acceleratorsAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.. Systems often split work between tiers: a small model on the device filters or answers routine inputs and offloadsoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run. hard cases to an edge server or the cloud. On-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device. is the subset running on the end user's own device, and TinyMLTinyML, n.: Machine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years. the extreme case of microcontrollers.
The cameras run edge AI to flag defects on the line and send only the flagged frames to the cloud.
Origin
Grew out of edge computing and Cisco's 'fog computing' (2012), which by the mid-2010s described processing near data sources for the Internet of Things. 'Edge AI' spread from about 2017–2018 as smartphone neural engines and dedicated edge inference chips appeared, and became a standard industry category.
Usage note
Usage is loose: vendors may call anything outside a hyperscale cloud 'edge', from a phone to a regional server room. federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates. is a related approach for training across edge devices without centralizing their data.
Related
- on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device.
- TinyMLTinyML, n.: Machine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years.
- model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud.
- offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run.
- federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.
Mentioned in
on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost., wake wordwake word, n.: A word or phrase, such as 'Alexa' or 'Hey Google', that a smart speaker or phone listens for constantly and that switches it into full listening mode., YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video., Core MLCore ML, prop. n.: Apple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud., depthwise separable convolutiondepthwise separable convolution, n.: A cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step., EfficientNetEfficientNet, prop. n.: A family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing., ExecuTorchExecuTorch, prop. n.: Meta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center., liquid neural networkliquid neural network, n.: A small, brain-inspired network for time-varying data whose neurons change how quickly they react depending on their input, letting a few neurons handle tasks such as steering a car., LiteRTLiteRT, prop. n.: Google's free software for running AI models on phones, small gadgets and microcontrollers; it was called TensorFlow Lite until 2024 and is used inside many Android and iPhone apps., MobileNetMobileNet, prop. n.: A family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy., offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run., ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer., small AIsmall AI, n.: Cheap, compact AI tools built for one practical job — like advising farmers or screening patients — that poorer countries can afford to run, instead of giant all-purpose models., spiking neural networkspiking neural network, n.: A neural network that, like the brain, communicates with brief pulses, or spikes, sent only when a neuron's charge crosses a threshold. On special chips this can save a great deal of energy., TinyMLTinyML, n.: Machine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years.