PyTorch
In plain wordsThe most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.
An open-source deep-learning framework for Python whose defining design choice is eager, or define-by-run, execution: each tensor operation runs as soon as Python reaches it, on a CPU or GPU, while the autograd system records the operations in a computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward. rebuilt on every run and later traverses it backward for reverse-mode automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.. Models are therefore ordinary Python programs that can use loops, conditionals and standard debuggers. Its torch.nn module supplies layers and losses and torch.optim supplies optimizers, while companion libraries cover data loading, distributed training (DistributedDataParallel and fully sharded data parallelism, FSDP) and deployment, including ExecuTorchExecuTorch, prop. n.: Meta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center. for devices. PyTorch 2.0 (2023) added torch.compile, which captures Python code into graphs and generates fused kernels, often in TritonTriton, prop. n.: A free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts., recovering much of the speed of static-graph frameworks. Since about 2019 it has dominated research papers, and most open-weight language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. and libraries such as Hugging Face TransformersTransformers, prop. n.: A free software library from Hugging Face that lets programmers download and use thousands of ready-trained AI models for text, images and sound with a few lines of code. and vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models. are built on it.
We wrote the training loop in PyTorch and switched on torch.compile only once the model was correct.
Origin
Released publicly in January 2017 by Facebook AI Research (now part of Meta AIMeta AI, prop. n.: The artificial intelligence research group of Meta, the company behind Facebook and Instagram, known for the PyTorch software and the Llama models. It is also the name of Meta's chatbot.), created by Adam Paszke, Sam Gross, Soumith Chintala and Gregory Chanan as a Python successor to Torch, a machine-learning library begun in 2002 whose later versions were scripted in Lua, and influenced by the define-by-run design of Chainer (2015). Caffe2 was merged into it in 2018, and governance passed to the independent PyTorch Foundation, part of the Linux Foundation, in 2022.
Usage note
'Torch' may mean the earlier Lua library or, informally, PyTorch itself, whose Python package is named torch. PyTorch is usually compared with TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems., which led adoption until about 2019, and with JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips..
Related
- TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems.
- JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips.
- automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.
- computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward.
- TransformersTransformers, prop. n.: A free software library from Hugging Face that lets programmers download and use thousands of ready-trained AI models for text, images and sound with a few lines of code.
- ExecuTorchExecuTorch, prop. n.: Meta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center.
- TritonTriton, prop. n.: A free language from OpenAI for writing the fast inner routines of AI programs in Python-style code; its compiler handles hardware details that Nvidia's own tools leave to experts.
- TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems.
Sources
- Paszke, A. et al. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv:1912.01703.
Mentioned in
Meta AIMeta AI, prop. n.: The artificial intelligence research group of Meta, the company behind Facebook and Instagram, known for the PyTorch software and the Llama models. It is also the name of Meta's chatbot., AI stackAI stack, n.: The layers of technology an AI product depends on, from chips and data centers at the bottom, through the models, up to the apps people use, each built on the layer below., automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically., computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward., CUDACUDA, prop. n.: Nvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI., Jupyter notebookJupyter notebook, n.: An interactive document, opened in a web browser, that mixes runnable code with its results, charts and notes, so people can experiment step by step and share their work., KerasKeras, prop. n.: A free, beginner-friendly Python library for building neural networks from ready-made layers in a few lines of code, while a larger toolkit underneath does the heavy computing., Mark ZuckerbergMark Zuckerberg, prop. n.: The American co-founder and chief executive of Facebook's parent company Meta, who made it a major AI developer that released its Llama models for anyone to download., NumPyNumPy, prop. n.: The basic free Python library for working quickly with large grids of numbers; nearly every data-science and machine-learning tool in Python is built on it or copies its style., scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way., tensortensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way., TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems., TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., AI compilerAI compiler, n.: Software that translates an AI model into efficient instructions for a specific chip, rearranging and combining its calculations so the hardware is used as fully as possible., CaffeCaffe, prop. n.: An early free deep-learning toolkit from the University of California, Berkeley, popular for image recognition in the mid-2010s, in which networks were described in simple text files rather than code., Core MLCore ML, prop. n.: Apple's software framework that lets apps on iPhones, iPads and Macs run AI models directly on the device, using its own chips instead of sending data to the cloud., DeepSpeedDeepSpeed, prop. n.: Free software from Microsoft for training and running AI models too big for ordinary tools, spread over many chips; it is best known for a memory-saving technique called ZeRO., ExecuTorchExecuTorch, prop. n.: Meta's free software for running PyTorch AI models directly on phones, wearables and other small devices, without needing an internet connection or a data center., fully sharded data parallelfully sharded data parallel, n.: A feature of the popular PyTorch software that lets many chips train a model too big for any one of them, by spreading the model and its training bookkeeping evenly across all the chips., JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips., Koray KavukcuogluKoray Kavukcuoglu, prop. n.: A Turkish-born computer scientist who co-authored many of DeepMind's landmark papers and in 2026 took charge of Google DeepMind as its senior vice president., LiteRTLiteRT, prop. n.: Google's free software for running AI models on phones, small gadgets and microcontrollers; it was called TensorFlow Lite until 2024 and is used inside many Android and iPhone apps.