backbone
In plain wordsThe main, usually pretrained part of a vision model that turns an image into useful internal features; smaller task-specific parts are attached to it, like limbs to a spine.
The principal feature-extractingfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare. network of a model, usually pretrained on a large dataset and shared across tasks, to which smaller task-specific components are attached. Detectors and segmenters are commonly described as backbone, neck and head: the backbone (for example ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI.-50, a vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence. or a self-supervised DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself. encoder) computes feature mapsfeature map, n.: The grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears.; the neck (such as a feature pyramid network) fuses them across scales; the head predicts boxes, classes or masks. Swapping in a larger or better-pretrained backbone is one of the most reliable ways to raise accuracy, and how the backbone was pretrained (supervised on ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom., on image–text pairs, or self-supervised) often matters as much as the head design. The word is also used for the pretrained large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. inside a vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence. and, loosely, for any base model that others build on.
Freezing the backbone for the first two epochs kept the new detection head from wrecking the pretrained features.
Origin
An anatomical metaphor that spread through object-detection papers in the mid-2010s; its first use there has not been traced. Mask R-CNN (2017) explicitly separated the 'convolutional backbone architecture' from the network head, and YOLOv4 (2020) popularized the backbone–neck–head breakdown. The older networking sense (a network's core links) is unrelated.
Usage note
The backbone–neck–head vocabulary is a convention, not a formal definition; one paper's neck may be another's head.
Related
- feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare.
- transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
- ResNetResNet, prop. n.: A family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI.
- vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
- foundation modelfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
Sources
- He, K., Gkioxari, G., Dollár, P. & Girshick, R. (2017). Mask R-CNN. ICCV 2017. arxiv.org/abs/1703.06870
- Bochkovskiy, A., Wang, C.-Y. & Liao, H.-Y. M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv:2004.10934.
Mentioned in
feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare., R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes., DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself., masked autoencodermasked autoencoder, n.: A way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.