depthwise separable convolution
In plain wordsA cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step.
A cheaper substitute for a standard convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location. that splits its two jobs, spatial filtering and channel mixing, into separate steps: each input channel is first filtered on its own by a k × k filterfilter, n.: In an image-recognition network, a small grid of learned numbers that acts as a pattern detector, for vertical edges, a patch of fur or a wheel, scanned across the whole picture. (the depthwise step), and a 1 × 1 convolution then combines the filtered channels (the pointwise step). For C_in input and C_out output channels the cost per position falls from k²·C_in·C_out to k²·C_in + C_in·C_out multiply-adds, close to a k²-fold saving when C_out is large (for 3 × 3 kernels, the eight- to ninefold reduction reported by the MobileNet authors), usually at a modest cost in accuracy. It is the core of efficient vision architectures such as Xception, the MobileNet family and EfficientNet, and of many models deployed on phones and other edge devicesedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.. Depthwise convolution is the special case of grouped convolution with one group per channel.
Swapping the 3 × 3 convolutions for depthwise separable ones cut the multiply-adds by about 85 percent.
Origin
Laurent Sifre devised depthwise separable convolutions during a 2013 internship at Google Brain, drawing on his doctoral work with Stéphane Mallat on scattering transforms; they were first shown publicly in an ICLR 2014 talk and detailed in his 2014 thesis. François Chollet's Xception (October 2016) reinterpreted Inception modules as a step towards them, and Andrew Howard and colleagues' MobileNets (April 2017) made them the standard building block for mobile vision.
Related
- convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers.
- neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.
- filterfilter, n.: In an image-recognition network, a small grid of learned numbers that acts as a pattern detector, for vertical edges, a patch of fur or a wheel, scanned across the whole picture.
Sources
- Chollet, F. (2016). Xception: Deep Learning with Depthwise Separable Convolutions. arXiv:1610.02357. arxiv.org/abs/1610.02357
- Howard, A. G. et al. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv:1704.04861. arxiv.org/abs/1704.04861
Mentioned in
convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., ConvNeXtConvNeXt, prop. n.: A 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers., EfficientNetEfficientNet, prop. n.: A family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing., François CholletFrançois Chollet, prop. n.: A French software engineer who created Keras, a popular tool for building neural networks, and the ARC benchmark, which tests whether AI can solve kinds of puzzles it has never seen., InceptionInception, prop. n.: A family of Google image-recognition networks, the first called GoogLeNet, whose building blocks examine each patch of an image at several scales at once while keeping computation low., MobileNetMobileNet, prop. n.: A family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy., neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.