network dissection
In plain wordsA method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people.
A method for labeling and counting interpretable units in a convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. by matching each unit's activation maps against human-annotated segmentations of visual concepts. For every unit, activations over a large probe dataset are thresholded at a high quantile and upsampled to image resolution; the intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. between the resulting regions and each concept's masks is computed, and a unit whose best overlap passes a cutoff is labeled a detector for that concept. The original probe set, Broden, combined several datasets into about 1,200 concepts in six categories: objects, scenes, parts, materials, textures and colors. Dissection showed that object and part detectors emerge in higher layers without supervision for them, compared architectures and training regimes by their numbers of unique detectors, and found that randomly rotating a layer's basis reduces interpretability, suggesting individual units form a privileged basis. GAN dissection applied the idea to image generators, where switching detected units on or off adds or removes objects.
Network dissection labeled unit 314 in the last convolutional layer as a door detector.
Origin
Introduced by David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva and Antonio Torralba at MIT in 'Network Dissection: Quantifying Interpretability of Deep Visual Representations' (CVPR 2017), building on Zhou and colleagues' 2015 finding that object detectors emerge in networks trained only to classify scenes. Bau's GAN Dissection followed in 2019.
Usage note
A unit labeled with one concept may also respond to others (polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.), and a fixed concept list cannot reveal features nobody annotated, a gap later addressed by natural-language neuron descriptions and automated interpretabilityautomated interpretability, n.: Using AI models to help explain other AI models — for instance, having a language model read examples that excite an internal unit and write a description of what it detects.. Compare testing with concept activation vectorstesting with concept activation vectors, phr.: A method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect., which probes directions rather than single units.
Related
- feature visualizationfeature visualization, n.: Making pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible.
- multimodal neuronmultimodal neuron, n.: A unit inside an AI vision model that responds to one idea however it appears — a photo, a drawing or the written name — much like the 'Halle Berry neuron' found in a human brain.
- testing with concept activation vectorstesting with concept activation vectors, phr.: A method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect.
- polysemanticitypolysemanticity, n.: When a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.
- automated interpretabilityautomated interpretability, n.: Using AI models to help explain other AI models — for instance, having a language model read examples that excite an internal unit and write a description of what it detects.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
Sources
- Bau, D., Zhou, B., Khosla, A., Oliva, A. & Torralba, A. (2017). Network Dissection: Quantifying Interpretability of Deep Visual Representations. Proc. CVPR 2017.
- Bau, D. et al. (2019). GAN Dissection: Visualizing and Understanding Generative Adversarial Networks. ICLR 2019.
Mentioned in
feature visualizationfeature visualization, n.: Making pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible., multimodal neuronmultimodal neuron, n.: A unit inside an AI vision model that responds to one idea however it appears — a photo, a drawing or the written name — much like the 'Halle Berry neuron' found in a human brain., testing with concept activation vectorstesting with concept activation vectors, phr.: A method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect.