Modern neural networks are absurdly large. GPT-scale models carry hundreds of billions of parameters, and even modest vision networks routinely exceed hundreds of megabytes. Yet a striking empirical fact keeps recurring: these same networks can often be pruned, quantized, or distilled to a fraction of their original size with negligible loss in accuracy.

This is not a happy accident of engineering. It points to something deeper about what neural networks actually do during training. If a 100-million-parameter model performs identically to its 1-billion-parameter parent, the extra parameters were not encoding useful information—they were scaffolding.

To understand why compression works, we need to leave the ad-hoc world of heuristics and enter the domain of information theory. The Information Bottleneck framework, redundancy analysis, and rate-distortion theory together give us a principled account of what learning is, why over-parameterization helps, and how much a trained network can be squeezed before its predictions degrade.

Tishby's Information Bottleneck (IB) principle reframes supervised learning as a compression problem. Given an input X and target Y, the network learns an internal representation T that minimizes the mutual information I(X; T) while maximizing I(T; Y). In plain terms: keep what matters for the task, discard everything else.

This framing has predictive power. Empirical studies of deep networks reveal two distinct training phases: a rapid fitting phase, where I(T; Y) increases as the network learns to predict, followed by a slower compression phase, where I(X; T) steadily decreases. The network is quite literally forgetting irrelevant details about its inputs.

The implication for compression is direct. If the final representation carries far less information about X than the raw input does, then the parameters encoding that representation must be highly redundant. There is simply less signal to preserve than the parameter count suggests.

This also explains why generalization tracks compression. A representation that has aggressively discarded task-irrelevant information cannot overfit to it. Compression is not just a deployment convenience—it is intrinsic to what a well-trained network becomes.

Takeaway

Learning is compression in disguise. A trained network is a lossy encoder that keeps predictive signal and discards the rest, which is precisely why it can be shrunk after training.

If networks ultimately learn compressed representations, why train them large in the first place? The answer lies in optimization geometry. Over-parameterized networks have loss landscapes with abundant well-connected minima, making gradient descent reliably find good solutions. Small networks, by contrast, sit in cramped landscapes riddled with saddle points and poor local optima.

The Lottery Ticket Hypothesis makes this concrete. Frankle and Carbin showed that within any large trained network exists a small subnetwork—a winning ticket—that, when trained in isolation from its original initialization, matches the full network's performance. The remaining 90–95% of weights functioned as an optimization aid, not as knowledge storage.

Information-theoretically, this means the effective parameter count is far below the nominal count. Many weights cluster near zero, encode duplicate features, or lie in flat directions of the loss surface that contribute nothing to the function computed. Techniques like SVD analysis of weight matrices routinely reveal that layer ranks are a small fraction of their dimensions.

This has a clean architectural implication: parameters are cheap during training and expensive during inference. The engineering play is to exploit the former for optimization tractability, then discard it for deployment.

Takeaway

Large models are not large because they need to be—they are large because gradient descent needs them to be. Once trained, most of the scaffolding can be removed.

How much can we actually compress? Rate-distortion theory gives us a lower bound: for a target accuracy loss (distortion), there exists a minimum representation size (rate) below which we cannot go. The gap between this theoretical floor and current practice is where engineering happens.

In practice, three techniques exploit different sources of redundancy. Pruning removes weights whose contribution to output variance falls below a threshold, typically achieving 10–20× size reduction. Quantization reduces the bit-width of remaining weights, exploiting the fact that neural computations are robust to precision loss—INT8 and even INT4 inference are now standard. Knowledge distillation trains a small student network to mimic a large teacher's output distribution, transferring the compressed representation directly.

Combined, these methods routinely achieve 50–100× compression on large language models with minimal quality loss. BERT-base compresses to under 15MB while retaining over 95% of downstream task performance. This is not far from theoretical limits estimated by mutual information analysis of the learned representations.

The remaining gap is diminishing. Techniques like mixture-of-experts routing and conditional computation push the effective compute per token down further, suggesting the ceiling is closer to hardware constraints than to information-theoretic ones.

Takeaway

Compression is bounded by the information content of the task, not by the size of the model. Once you know how much signal a task actually requires, you know how small the network can become.

Neural network compression is not a bag of tricks—it is the natural consequence of what learning is. Networks train large to navigate optimization landscapes, then converge on representations that discard most of what they were shown.

For engineers, this reframes the deployment question. The design pattern is: over-parameterize for trainability, then compress against the information-theoretic floor of the task. The two phases have different objectives and deserve different architectural choices.

The next frontier is training networks that are compressed from the start—architectures that respect the bottleneck principle during optimization rather than after it. Getting there will require closer dialogue between information theory and systems design than we have had so far.