Ollama

An open-source tool for downloading, managing and running open-weightopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private. language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. locally on macOS, Windows and Linux. It distributes models from its own online library, typically as quantizedquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama. weights bundled with a chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse. and default settings; pulls and runs them with Docker-like commands such as 'ollama run'; and lets users customize them through a plain-text Modelfile that sets the base model, system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply. and sampling parameters. A background server exposes a local REST API, by default on port 11434, including an OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., so that editors, coding agents and other applications can use local models as drop-in replacements for cloud ones. Inference runs on the ggml and llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models. stack, with automatic GPU detection and CPU offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run.. Optional hosted cloud models were later added alongside local ones.

Install Ollama, run the 8B model, and point your editor at localhost:11434.