SRAM
In plain wordsThe fastest kind of computer memory, built directly into processor chips. It is quick but takes up a lot of space, so chips hold only a little of it.
A type of fast memory used above all inside processor chips, for registers, caches and scratchpad buffers: a small array can be read within a few clock cycles, and it is made in the same logic process as the surrounding circuits. Each bit sits in a cross-coupled latch, usually of six transistors, that keeps its state as long as power is on, so unlike DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data. it needs no refresh; but the latch takes far more area than a DRAM cell, which limits on-chip capacities to megabytes rather than gigabytes. The H100H100, prop. n.: Nvidia's flagship AI chip of 2022–2024, so scarce and sought-after during the generative-AI boom that it became a unit of measure for AI computing power., for example, has 50 MB of L2 cache plus up to 228 KB of shared memory per streaming multiprocessor. AI kernels such as FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip. are designed to keep working data in SRAM and avoid round trips to high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., and some accelerators — Groq's LPU, Cerebras's wafer-scale enginewafer-scale engine, n.: A single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip. — hold models largely in SRAM for very low-latency inference. SRAM density has improved little at the newest nodesprocess node, n.: The label for a generation of chip-making technology, like '3 nanometer'. Smaller numbers mean newer, denser chips, but the number no longer measures any actual part of the chip., a growing constraint on chip design.
Origin
Semiconductor static memory dates to the 1960s; Intel's first product, the 64-bit 3101 of 1969, was a bipolar SRAM, and on-chip SRAM caches became standard in microprocessors from the late 1980s.
Usage note
Not to be confused with DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data. (dense, mostly off-chip, refreshed) or with flash memory (non-volatile storage).
Related
- DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data.
- high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough.
- memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive.
- FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip.
- wafer-scale enginewafer-scale engine, n.: A single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip.
- memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed.
- DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data.
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., tokens per secondtokens per second, n.: A speed rating for AI text generation: how many word-pieces a model produces each second. Five to ten keeps up with a fast reader; hundreds feel instant., AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible., analog in-memory computinganalog in-memory computing, n.: Doing AI math inside the memory itself, using electrical currents through stored values, instead of shuttling numbers back and forth to a separate processor. It could save lots of energy but is imprecise., CerebrasCerebras, prop. n.: An American chip company that makes a single giant processor the size of a whole silicon wafer, used to train and run AI models very quickly., DRAMDRAM, abbr.: The main working memory of computers and AI chips: dense and fairly cheap, but slower than on-chip memory and needing constant refreshing to keep its data., kernel fusionkernel fusion, n.: Merging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step., memory wallmemory wall, n.: The growing gap between how fast chips can calculate and how fast they can fetch data from memory. Processors increasingly sit idle, waiting for numbers to arrive., von Neumann bottleneckvon Neumann bottleneck, n.: The traffic jam caused by ordinary computers keeping data and processing in separate places, so information must constantly shuttle back and forth through one narrow channel., wafer-scale enginewafer-scale engine, n.: A single giant processor made from nearly a whole dinner-plate-sized silicon wafer instead of the small chips normally cut from it, so data rarely has to leave the chip.