Modern training recipes call for batch sizes that no single accelerator can hold. GPT-3 trained with batches of 3.2 million tokens. Goyal et al. trained ResNet-50 on ImageNet with 8,192 images per batch. A typical fine-tuning job on a 24GB GPU, by contrast, might fit eight long sequences before running out of memory.

Gradient accumulation closes that gap by relying on a simple fact: the gradient of a sum is the sum of the gradients. Instead of processing one large batch, you process several smaller micro-batches in sequence. Each one adds its gradient to a shared buffer, and the weights update only once that buffer represents the full batch.

The technique takes a few lines of code and is easy to get subtly wrong. Its value depends on knowing why it works and where the equivalence breaks down. That understanding decides whether accumulation improves your training run or quietly degrades it.

GPU memory during training falls into four broad categories: parameters, optimizer states, gradients, and activations. The first three scale with model size and stay the same regardless of batch size. Activations are the intermediate tensors saved during the forward pass so backpropagation can use them, and they grow linearly with the number of examples processed at once. In transformers with long sequences, activations often dominate the memory budget.

Gradient accumulation exploits this asymmetry. Training already needs a gradient buffer the same size as the parameters. Each micro-batch runs forward and backward, adds its contribution to that buffer, and releases its activations before the next micro-batch begins. Peak memory therefore depends on the micro-batch size. The batch the optimizer sees is the micro-batch size multiplied by the number of accumulation steps. In PyTorch this behaviour is the default, because gradients keep accumulating across backward calls until they are explicitly zeroed.

The equivalence holds because batch loss is an average over examples. If you scale each micro-batch loss by 1/N and sum N such gradients, you recover the full-batch gradient, up to floating-point error. The cost is time. Accumulation trades memory for wall-clock time, since N micro-batches execute one after another. Throughput per optimizer step is no better than with the small batch. Accumulation enables large batches; it does not make them faster.

The equivalence also has well-documented failure modes. Batch normalization computes its statistics per micro-batch, so a model trained with a micro-batch of 4 behaves like a micro-batch of 4 no matter how many steps you accumulate. Variable-length sequences create a second trap. Averaging loss within each micro-batch and then across micro-batches weights tokens unequally, a bug that affected widely used training libraries until it was identified and fixed in 2024. In distributed data-parallel training, gradients should synchronize only on the final micro-step. Otherwise communication cost multiplies by N.

Takeaway

Gradient accumulation separates two quantities that hardware ties together: how many examples fit in memory and how many examples inform each update. The equivalence holds only for operations that are linear across examples.

In exact arithmetic, summing gradients across micro-batches gives the same result as computing them in one pass. Floating-point arithmetic is not associative, so the order and precision of each addition change the outcome. Over a handful of steps the difference is negligible. With dozens or hundreds of accumulation steps in reduced-precision formats, it can become measurable.

The core risk is swamping. When a small value is added to a much larger running total, the small value's low-order bits are lost. BF16 carries only 8 bits of significand precision, roughly two to three decimal digits. Once the accumulated gradient grows large relative to each new contribution, later micro-batches count for less than their true share, or for nothing at all. FP16 is more precise but has a narrow exponent range. Losses scaled by 1/N can therefore underflow to zero for small gradients when N is large.

Standard mixed-precision practice handles most of this. Frameworks keep master weights and gradient accumulators in FP32 while running the forward and backward passes in lower precision. With FP16, dynamic loss scaling multiplies the loss before backpropagation so gradients stay within representable range, then divides that factor back out before the optimizer step. Under accumulation, the unscaling step and gradient clipping must happen once, after the final micro-batch. Clipping each micro-batch separately changes the effective gradient.

When memory pressure tempts engineers to store accumulation buffers in BF16, the trade-off deserves scrutiny. Alternatives include keeping only the accumulator in FP32, using compensated (Kahan) summation, or reducing the number of accumulation steps. Expect results that are close to a true large-batch run but not bitwise identical, and write reproducibility tests with numerical tolerances rather than exact matches.

Takeaway

Every accumulation step is an addition, and floating-point additions lose information when magnitudes diverge. Choose your accumulator's precision based on how many steps you plan to accumulate, rather than defaulting to the compute dtype.

Once accumulation makes batch size nearly free in memory terms, the real question becomes which batch size to target. Larger is not automatically better. McCandlish et al. at OpenAI introduced the gradient noise scale, a measurable quantity that predicts a critical batch size. Below that size, doubling the batch roughly halves the number of steps needed. Above it, returns fall off sharply: each step costs more compute without a proportional reduction in steps.

The critical batch size depends on the task, and it grows as training progresses and loss falls. This is why large language model recipes often ramp batch size during training. GPT-3 increased its batch from 32,000 tokens to 3.2 million over the first several billion tokens. Early training phases and simpler tasks have smaller critical batches, while complex tasks with noisy gradients tolerate larger ones.

Changing batch size also shifts the optimal learning rate. Goyal et al. showed that for SGD on ImageNet, a linear scaling rule combined with warmup held up to 8,192 images per batch. Adaptive optimizers such as Adam often follow something closer to square-root scaling. At extreme batch sizes, layer-wise methods such as LAMB, which trained BERT in 76 minutes, become relevant. A common source of degraded results is increasing accumulation steps without revisiting the learning rate schedule.

A practical procedure follows from this. 1. Choose the largest micro-batch that fits in memory while keeping the GPU well utilized. Accumulation never makes up for an underutilized device. 2. Set a target effective batch from published recipes for similar models, measured in tokens for language models. 3. Divide the target by the micro-batch size times the number of data-parallel workers to get the accumulation steps. For fine-tuning on small datasets, keep the effective batch modest: 10,000 examples at an effective batch of 1,024 yields fewer than ten updates per epoch.

Takeaway

Batch size is a hyperparameter with a measurable sweet spot, not a resource to maximize. Because gradient accumulation removes the memory constraint, choosing the right value becomes your job rather than your hardware's.

Gradient accumulation works because gradients add up across examples. That one property lets engineers decouple optimizer behaviour from device memory and run training recipes designed for clusters on much smaller hardware.

Using it well means respecting its limits. Treat accumulation as exact only for operations that are linear across examples, and audit BatchNorm and loss normalization with that in mind. Keep accumulators in FP32. Apply unscaling and clipping once per optimizer step. In distributed setups, synchronize gradients only on the final micro-step.

Above all, choose the effective batch size deliberately. Maximize the micro-batch for utilization, aim near the task's critical batch size, and retune the learning rate whenever the effective batch changes. Memory constraints determine how you reach a given batch size, while optimization dynamics determine which size to aim for.