A deep network at initialization is a high-dimensional dynamical system waiting to either explode or collapse. Before a single gradient update, the choice of initial weight distributions determines whether signals can propagate forward through dozens or hundreds of layers—and whether gradient information can flow backward to update early parameters. Get this wrong, and training never starts. Get it right, and you establish the narrow corridor of stability that learning requires.

The question sounds deceptively simple: what variance should we assign to the initial weights? Yet answering it rigorously demands a layer-by-layer analysis of how random matrix products transform signal magnitudes. The mathematical framework that emerges connects random matrix theory, moment propagation, and the geometry of activation functions into a coherent theory of trainability at initialization. The results—conditions derived by Glorot, Bengio, He, and others—are not heuristics. They are precise variance prescriptions that emerge from first principles.

What makes this analysis especially relevant today is that modern architectures systematically violate the assumptions behind classical initialization schemes. Residual connections create additive signal paths. Layer normalization dynamically rescales activations. Attention mechanisms introduce data-dependent, non-stationary transformations. Each modification demands a reconsideration of the variance conditions that preserve stable forward and backward propagation. Understanding the derivations—not just the formulas—is what allows practitioners and researchers to reason about initialization in architectures that no lookup table covers.

The foundational analysis begins with a simple feedforward model: layer activations defined as h^(l) = f(W^(l)h^(l-1)), where f is a pointwise nonlinearity and each weight matrix W^(l) has i.i.d. entries drawn from a zero-mean distribution with variance σ_w². Under the assumption that pre-activations and weights are independent—valid at initialization—the variance of the pre-activation at layer l decomposes multiplicatively: Var(z_i^(l)) = n_l-1 · σ_w² · Var(h_j^(l-1)), where n_l-1 is the fan-in of layer l.

This recurrence is the critical equation. If the per-layer gain factor n_l-1 · σ_w² · E[f'(z)²] exceeds unity, activation variances grow exponentially with depth. If it falls below unity, activations shrink exponentially toward zero. Only when the product equals one does variance remain stable across arbitrary depth. The term E[f'(z)²] captures the nonlinearity's contribution—for linear activations it equals one, for ReLU it equals 1/2, and for tanh it depends on the input variance through a self-consistent fixed-point equation.

The backward pass obeys a symmetric analysis. The gradient of the loss with respect to pre-activations at layer l satisfies δ^(l) = f'(z^(l)) ⊙ (W^(l+1))^Tδ^(l+1). Computing the variance of this expression introduces the fan-out n_l instead of the fan-in: Var(δ_i^(l)) = n_l · σ_w² · E[f'(z)²] · Var(δ_j^(l+1)). Stable gradient propagation therefore requires a second, generally different variance condition—one involving fan-out rather than fan-in.

The simultaneous satisfaction of both forward and backward stability conditions is possible only when fan-in equals fan-out, which rarely holds in practice. This tension motivated the harmonic mean compromise that became the Glorot initialization. More fundamentally, the analysis reveals that depth acts as an amplifier of initialization error: a per-layer gain of 1.01 seems innocuous, but across 100 layers it produces a variance ratio of e ≈ 2.7, and across 1000 layers a ratio exceeding 10⁴.

The mathematical structure here is that of a product of random matrices. The Lyapunov exponent of this product—the average logarithmic growth rate—determines whether the network sits in the stable, exploding, or vanishing regime. The initialization variance is the primary control parameter for this exponent at t = 0. Mean field theory formalizes this by tracking the entire distribution of pre-activations across layers, yielding phase diagrams that delineate ordered (collapsed), chaotic (exploding), and edge-of-chaos (stable) regimes as functions of weight and bias variance.

Takeaway

Depth converts small per-layer variance mismatches into exponential signal collapse or explosion. The initialization variance is not a hyperparameter to tune—it is a stability condition derived from the multiplicative structure of deep compositions.

The Xavier (Glorot) initialization emerged directly from setting the forward-pass gain factor to unity under the assumption of a linear activation or a symmetric nonlinearity like tanh operating near the origin. For a layer with fan-in n_in and fan-out n_out, the forward stability condition gives σ_w² = 1/n_in, while the backward condition gives σ_w² = 1/n_out. Glorot and Bengio's compromise was the harmonic mean: σ_w² = 2/(n_in + n_out). This is optimal in the sense that it minimizes the maximum deviation from unit gain across both propagation directions.

The arrival of ReLU activations broke the symmetry assumption underlying Xavier initialization. ReLU zeros out exactly half the pre-activation distribution (assuming zero-mean inputs), which means the effective second moment of the derivative is E[f'(z)²] = 1/2 rather than approximately 1. He et al. incorporated this factor directly: the forward stability condition becomes n_in · σ_w² · (1/2) = 1, yielding σ_w² = 2/n_in. This is precisely twice the Xavier variance—a factor of √2 in the standard deviation—and the correction is necessary and sufficient for ReLU networks to maintain stable variance propagation.

The optimality of these schemes is conditional on specific, clearly stated assumptions: independent weights and activations, i.i.d. initialization, specific nonlinearities, and the absence of skip connections or normalization layers. Within those assumptions, the derivations are exact, not approximate. The variance conditions are necessary for stable propagation in the infinite-width limit, a fact formalized by the correspondence between wide neural networks and Gaussian processes at initialization.

Variants for other activation functions follow the same template. Leaky ReLU with negative slope α gives E[f'(z)²] = (1 + α²)/2, yielding σ_w² = 2/((1 + α²) · n_in). SELU activations require initialization that preserves specific fixed-point variance and mean conditions. The pattern is consistent: identify the second moment of the activation derivative, then solve the unit-gain equation. Each activation function defines its own critical manifold in the space of initialization hyperparameters.

What these derivations collectively establish is a principle of matched impedance—borrowing terminology from signal processing. Each layer must be initialized so that it neither amplifies nor attenuates the signal passing through it. The weight variance is the impedance-matching parameter. When the activation function changes, the impedance condition changes, and the initialization must follow. This is why no single initialization scheme is universal: the correct variance is always a function of the computational graph's local structure.

Takeaway

Xavier and He initializations are not empirical tricks—they are exact solutions to variance preservation equations under specific activation assumptions. The derivation template generalizes: compute the second moment of the activation derivative, then solve for unit gain.

Residual networks fundamentally alter the variance propagation analysis. A residual block computes h^(l+1) = h^(l) + F(h^(l)), where F is the residual function. The variance of the output is Var(h^(l+1)) = Var(h^(l)) + Var(F(h^(l))) under the assumption of uncorrelated identity and residual paths. After L blocks, the variance grows as Var(h⁽⁰⁾) · (1 + L · Var(F)) if each residual branch contributes equally. To prevent this linear variance growth from destabilizing very deep networks, the residual branch must be downscaled—either by initializing the last layer of each block to zero (the "fixup" approach) or by applying a factor of 1/√L or 1/L to the residual output.

Layer normalization and batch normalization introduce a qualitatively different regime: they dynamically enforce unit variance at every layer, making the forward-pass variance propagation analysis largely moot. However, the backward pass is not similarly protected. Gradient magnitudes through normalized layers depend on the Jacobian of the normalization operation, which involves terms of order 1/√n (where n is the normalization dimension) that can still create pathological gradient scaling in certain configurations. Moreover, the interaction between normalization and weight initialization affects the effective learning rate—smaller initial weights produce larger normalized gradients, creating an implicit learning rate amplification.

Attention mechanisms present the most complex initialization landscape. The query-key dot product Q^TK / √d_k has a variance that depends multiplicatively on the variances of the query and key projections and the dimension d_k. The √d_k scaling factor is itself an initialization-adjacent design choice: it ensures the softmax input has unit-order variance, preventing saturation into near-one-hot distributions. If the projection weights are initialized with He or Xavier variance, the pre-softmax logits will have variance d_k · σ_Q² · σ_K² · d_model, which must be normalized to O(1) for the softmax to remain in its sensitive regime.

Recent work on transformer initialization—including the μP (maximal update parameterization) framework of Yang and Hu—extends the classical variance analysis to a joint optimization of initialization scale and learning rate. The key insight is that the correct initialization is not solely determined by forward-backward signal stability but also by ensuring that parameter updates at the first step have the correct magnitude relative to the initial weights. In μP, different parameter groups (embeddings, attention projections, output layers) receive width-dependent initialization scales and learning rates that together ensure hyperparameter transfer across model scales.

The overarching lesson is that initialization theory must co-evolve with architecture design. Every structural innovation—skip connections, normalization, attention, mixture-of-experts routing—modifies the computational graph in ways that invalidate previous variance conditions. The derivation methodology, however, remains constant: track the variance (and covariance, for correlated paths) of activations and gradients through the modified graph, identify the stability conditions, and solve for the initialization parameters that satisfy them. Mastery of this methodology is what separates practitioners who can reason about novel architectures from those who rely on defaults that may silently fail.

Takeaway

Modern architectures systematically violate classical initialization assumptions. The formulas change, but the methodology does not: trace variance through the actual computational graph, identify the stability manifold, and initialize on it.

Weight initialization is not a preprocessing step—it is the mathematical precondition for trainability. The variance conditions derived from forward and backward propagation analysis define a narrow stability manifold in parameter space, and departing from it produces exponential pathologies that no optimizer can recover from in finite time.

The classical Xavier and He results remain foundational, but their true value lies in the derivation methodology they established: decompose the computational graph into composable units, propagate variance through each, and solve for the critical surface. This methodology generalizes to every architecture that will ever be built.

As networks grow deeper and architecturally more complex, initialization theory becomes more—not less—important. The researchers who advance the field will be those who can derive stability conditions for novel computational graphs from first principles, treating initialization not as a default to accept but as a design parameter to engineer.