training loop

The control structure that drives iterative training: until a step or token budget is reached, it draws a mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once., runs the forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., computes the lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down., runs the backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., applies the optimizeroptimizer, n.: The rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment. step (with any gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update. or schedulelearning rate schedule, n.: A plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in. update) and clears gradients for the next iteration. Around this core sit periodic evaluation on a validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one., logging, checkpointingcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples. and, in distributed training, gradient synchronization across devices. High-level APIs such as Keras's fit method, PyTorch Lightning and the Hugging Face Trainer package the loop, while research code often writes it out for full control. In reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. and RL-based post-training, the loop alternates generating data (rolloutsrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning. from the current policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards.) with updates, and keeping the two in step is a central engineering problem.

The bug was in the training loop: we stepped the scheduler before the optimizer.