Ray

An open-source framework for spreading Python programs, including machine-learning workloads, across many processors and machines, so that code developed on a laptop can run on a cluster. Its core offers two primitives — tasks, which are remote stateless functions, and actors, which are remote stateful objects — whose calls return futures, while a distributed scheduler places work on CPUs and GPUs across machines, a shared object store passes data between them, and fault tolerance restarts failed work. Libraries built on the core cover machine-learning stages: Ray Data for distributed data processing and batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less., Ray Train for distributed training with PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips. and other frameworks, Ray Tune for hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., Ray Serve for model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., and RLlib for reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time.. Ray is widely used for large-scale data preparation, reinforcement-learning post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner. of language models and multi-model serving, often together with vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models..

We moved the embedding job to Ray and it ran on 64 GPUs without rewriting the processing code.