Practitioners who scale training across hundreds of accelerators run into a persistent anomaly. Increase the batch size while holding the epoch budget fixed, and test accuracy often degrades, even when training loss reaches comparable values. Keskar and colleagues documented this generalization gap systematically in 2017, though it had been folklore for years. It is not a hardware artifact or an implementation bug. It is a statistical property of the optimizer itself.

The standard framing treats minibatch stochastic gradient descent as a noisy approximation to full-batch gradient descent. In that view, noise is a nuisance tolerated for computational efficiency. The framing is incomplete. Sampling variance is structured. It scales predictably with batch size, and it interacts with the geometry of the loss surface in ways that bias which solutions the optimizer reaches. Remove the noise, and you change the implicit regularizer, often without noticing.

This article develops the argument in three steps. First, it derives how batch size controls gradient variance and defines the effective temperature of SGD. Second, it examines the sharp-versus-flat minima hypothesis, including the reparameterization critiques that complicate it. Third, it derives learning rate scaling rules from the stochastic differential equation view and identifies where those rules break down. The goal is not a recipe. The goal is a working model of the mechanism, precise enough to predict when large-batch training will succeed and when it will quietly cost accuracy.

Let L(θ) = (1/N) Σ ℓ_i(θ) and denote the full gradient G = ∇L. A minibatch of size B, sampled uniformly, yields the estimator g_B = (1/B) Σ_{i∈B} ∇ℓ_i. It is unbiased. If Σ denotes the per-example gradient covariance, sampling with replacement gives Cov(g_B) = Σ/B. Sampling without replacement multiplies this by a finite-population correction of roughly (1 − B/N), which matters only when the batch approaches the dataset size.

Now decompose the SGD update as θ_{t+1} = θ_t − εG − ε(g_B − G). The deterministic drift is −εG. The stochastic perturbation has zero mean and covariance ε²Σ/B. Over many small steps, this discrete process is well approximated by the stochastic differential equation dθ = −∇L dt + √(ε/B) Σ^{1/2} dW, where continuous time is t = ε × steps. Learning rate and batch size enter the diffusion term only through their ratio.

That ratio, ε/B, behaves as a temperature. Smith and Le formalized it as the noise scale g = ε(N/B − 1), which reduces to approximately εN/B when B ≪ N. Jastrzębski and colleagues reached a compatible conclusion: the dynamics, and empirically the final test accuracy, depend largely on ε/B rather than on ε and B separately. The claim holds within the regime where the SDE approximation is valid, meaning small steps and smoothly varying gradients.

A complementary quantity comes from McCandlish and colleagues. They define the simple noise scale B_noise = tr(Σ)/|G|², which compares total gradient variance to squared signal. When B ≪ B_noise, each step is noise-dominated, and doubling the batch nearly halves the steps required. When B ≫ B_noise, the gradient estimate is already accurate, and additional samples are largely wasted. Notably, B_noise tends to grow during training, because |G| shrinks faster than tr(Σ) as optimization approaches a minimum.

The critical observation is that this noise is not isotropic. Near a minimum of a log-likelihood loss, Σ approximates the Fisher information, which in turn approximates the Hessian H. Fluctuations are therefore largest along directions of high curvature. SGD does not jitter uniformly in parameter space. It shakes hardest precisely where the landscape is steepest. That alignment between noise and curvature is what turns a variance argument into a geometric one, and it sets up everything that follows.

Takeaway

Batch size is not merely a throughput parameter. Through the ratio ε/B, it sets the temperature of an implicit stochastic process, and temperature determines which regions of the loss landscape remain stable.

The flat minima hypothesis predates modern deep learning. Hochreiter and Schmidhuber argued in 1997 that a minimum occupying a wide basin can be specified with fewer bits of parameter precision, implying lower description length and, by minimum description length reasoning, better generalization. Keskar and colleagues revived the idea empirically, measuring sharpness as the maximum loss increase within a small neighborhood. Large-batch solutions were consistently sharper, and their test accuracy was consistently worse.

The noise model from the previous section supplies a mechanism. Consider a quadratic basin with Hessian H, and assume Σ ≈ H. The SDE becomes an Ornstein–Uhlenbeck process whose stationary covariance C satisfies HC + CH = (ε/B)H, giving C = (ε/2B)I. The expected excess loss is then ½ tr(HC) = (ε/4B) tr(H). At fixed temperature, sharper basins sustain higher stationary loss and are escaped more readily. Small batches simply cannot settle in narrow wells.

Backward error analysis sharpens the picture. Smith, Dherin, Barrett and De showed that SGD with random shuffling approximately follows a modified loss: the original objective plus a penalty proportional to ε/B times the trace of the gradient covariance. Near a minimum, that trace tracks the Hessian trace. SGD therefore performs implicit flatness regularization, and increasing the batch size at fixed learning rate attenuates the penalty. Zhu and colleagues further showed that curvature-aligned noise escapes sharp minima far more efficiently than isotropic noise of equal magnitude.

The hypothesis has serious critics. Dinh and colleagues observed that ReLU networks are positively homogeneous: scaling one layer by α and the next by 1/α leaves the function unchanged while arbitrarily altering the Hessian spectrum. Raw curvature is therefore not parameterization-invariant, and any sharp minimum has a flat twin computing the same function. Hoffer and colleagues added that much of the generalization gap reflects fewer parameter updates at fixed epochs. Training longer, with appropriate rate adjustment, closes a substantial portion of it.

The hypothesis survives in refined form. Scale-invariant sharpness measures and PAC-Bayes bounds, which evaluate loss robustness under perturbations proportional to weight magnitude, remain among the most predictive generalization measures in large empirical studies such as Jiang and colleagues' survey. Sharpness-aware minimization exploits the same principle explicitly. The defensible claim is not that low curvature causes generalization, but that robustness of the loss to perturbations of an appropriate scale correlates with it, and that small-batch noise biases optimization toward such robustness.

Takeaway

Flatness is not a property of a point in parameter space but of a point under a chosen geometry. The useful question is not whether a minimum is flat, but flat with respect to which perturbations.

The linear scaling rule follows directly from the SDE: to preserve temperature when multiplying batch size by k, multiply learning rate by k. A second derivation avoids continuous time. Take k consecutive steps with batch B and rate ε, so θ_{t+k} = θ_t − ε Σ_j g_{B,j}(θ_{t+j}). If gradients change little across those k steps, this approximates a single step at rate kε on batch kB. Goyal and colleagues used the rule to train ResNet-50 on ImageNet with batch 8192 in one hour.

Both derivations rest on assumptions that fail predictably. Early in training, gradients change rapidly, so the k-step approximation breaks; this motivates gradual warmup, which Goyal applied over the first five epochs. More fundamentally, gradient descent on a quadratic is stable only when ε < 2/λ_max. Linear scaling eventually crosses that threshold. Cohen and colleagues showed that full-batch training drives λ_max to hover near 2/ε, the edge of stability, meaning curvature itself adapts to the step size.

A competing prescription is square-root scaling. If the objective is to hold the per-step update variance ε²Σ/B constant, then ε ∝ √B. Krizhevsky proposed this heuristically, and Hoffer and colleagues defended it. The two rules preserve different invariants: linear scaling conserves the drift-to-diffusion ratio per unit of continuous time, while square-root scaling conserves step-level variance at the cost of reduced drift. For plain SGD in the small-step regime, linear scaling has stronger theoretical and empirical support.

For adaptive optimizers, the answer changes. Malladi and colleagues derived SDE approximations for RMSprop and Adam and found that square-root scaling is the correct analogue, because the preconditioner normalizes gradient magnitude and alters how noise enters the dynamics. At large batch sizes, every rule saturates. McCandlish and colleagues derived the optimal rate ε_opt(B) = ε_max / (1 + B_noise/B), linear for B ≪ B_noise and flattening toward ε_max beyond it. Shallue and colleagues confirmed this pattern of perfect scaling, diminishing returns, then a plateau across many workloads.

The same analysis yields a tradeoff between optimization steps S and training examples E: (S/S_min − 1)(E/E_min − 1) = 1. The critical batch size is not a constant of the architecture; it depends on the task and grows as training proceeds. This explains why Smith, Kindermans and Le could replace learning rate decay with batch size increases. Raising B lowers temperature just as lowering ε does, while reducing the number of sequential updates required. Annealing schedules are temperature schedules, whichever variable implements them.

Takeaway

Every scaling rule is a conservation law in disguise. Before choosing one, ask which quantity you actually need to hold fixed: temperature, per-step variance, or step size relative to curvature.

The batch size question resolves into a statement about stochastic dynamics. Minibatch sampling injects curvature-aligned noise at a temperature set by ε/B. That noise acts as an implicit regularizer, penalizing solutions whose loss is sensitive to perturbation. Large batches cool the process and weaken the regularizer unless the learning rate compensates.

Several principles follow. Treat ε/B, not either variable alone, as the primary control for SGD. Measure the gradient noise scale instead of guessing a critical batch size, and expect it to grow during training. Use flatness measures that respect the architecture's symmetries. Derive scaling rules from the invariant you intend to preserve, and verify where the approximations behind them fail.

Vapnik's central lesson was that generalization depends on the capacity an algorithm can effectively reach, not on raw parameter count. Batch size is one of the quiet determinants of that effective capacity. The optimizer is not a neutral search procedure. It is part of the model.