The problem of priors haunts Bayesian epistemology like a persistent specter. How should a rational agent assign initial probabilities before encountering evidence? Subjective Bayesians embrace the apparent arbitrariness—any coherent prior will eventually converge toward truth given sufficient data. But this permissiveness troubles those seeking objective standards of rationality.

Edwin Jaynes proposed a bold solution rooted in information theory: the maximum entropy principle. When faced with incomplete information, we should adopt the probability distribution that maximizes Shannon entropy subject to whatever constraints our background knowledge provides. This distribution, Jaynes argued, represents the uniquely rational choice—maximally uncommitted to claims beyond what the evidence supports, importing no hidden assumptions.

The philosophical stakes are considerable. If maximum entropy provides objective prior selection, it rescues Bayesianism from charges of subjectivity while maintaining its elegant updating machinery. Critics have challenged this program on multiple fronts, questioning whether entropy maximization truly delivers the objectivity it promises. This article examines Jaynes's core argument through the Concentration Theorem, confronts the transformation invariance objections that expose deep problems, and identifies legitimate applications where maximum entropy reasoning retains its power despite these philosophical limitations.

Shannon entropy measures the average uncertainty in a probability distribution. For a discrete distribution over outcomes with probabilities p₁, p₂, ..., pₙ, entropy equals -Σpᵢ log(pᵢ). Higher entropy means greater uncertainty—more spread across possibilities. The uniform distribution over n outcomes achieves maximum entropy log(n), while a distribution concentrated entirely on one outcome has entropy zero.

Jaynes's Concentration Theorem provides the mathematical backbone for using entropy as a guide to prior selection. Consider all probability distributions satisfying certain constraints—perhaps a known mean, variance, or other moment conditions. Among these infinitely many consistent distributions, exactly one maximizes entropy. The theorem establishes that this maximum entropy distribution is maximally non-committal given the constraints.

The formal argument proceeds through counting. Consider sampling repeatedly from a true underlying distribution. The overwhelming majority of observed frequency patterns will closely resemble the maximum entropy distribution consistent with the constraints. Other distributions require specific patterns that constitute a vanishingly small fraction of possibilities. In this precise sense, maximum entropy distributions are typical while alternatives are exceptional.

This typicality argument connects to deeper principles. Adopting a non-maximum entropy distribution means privileging specific patterns without evidential warrant. You're effectively claiming the world exhibits a particular structure beyond what your constraints entail. The maximum entropy distribution makes no such additional claims—it spreads probability as uniformly as possible given what you know, encoding ignorance where ignorance genuinely exists.

Consider a concrete example: you know only that a die's average roll is 3.5 (the fair value). The uniform distribution over faces maximizes entropy subject to this constraint. Any other distribution—say, one favoring extreme outcomes 1 and 6—builds in additional structure. Jaynes argues this additional structure requires justification. Without it, rationality demands maximum entropy.

Takeaway

The maximum entropy distribution is uniquely typical among all distributions satisfying given constraints—adopting any alternative means privileging specific patterns without evidential warrant.

The most serious challenge to maximum entropy's objectivity emerges from coordinate transformations. Shannon entropy is not invariant under reparameterization. If we describe the same physical situation using different variables, maximum entropy recommends different probability distributions. This dependence on arbitrary representational choices undermines claims that maximum entropy provides an objective rational standard.

Consider a simple case: a parameter θ ranging from 0 to 1, about which we know nothing beyond these bounds. Maximum entropy over θ yields the uniform distribution. But suppose we equivalently describe the situation using φ = θ², also ranging from 0 to 1. Maximum entropy over φ gives a uniform distribution over φ—which transforms back to a non-uniform distribution over θ concentrated toward smaller values.

The mathematics is straightforward but the philosophical implications are severe. Both parameterizations are equally legitimate descriptions of the same situation. Yet maximum entropy delivers incompatible recommendations depending on which description we arbitrarily employ. The principle cannot be objectively rational if its outputs depend on representational conventions that carry no epistemic significance.

Jaynes recognized this problem and proposed solutions involving transformation groups. When a problem exhibits symmetries—invariance under certain transformations—we should require our prior to respect these symmetries. This additional constraint can sometimes select a unique prior independent of parameterization. The approach works elegantly for highly symmetric problems like coin flips or ideal gases.

However, most realistic inference problems lack the requisite symmetry structure. When transformation groups don't constrain the prior uniquely, we're back to making choices that maximum entropy was supposed to eliminate. The objectivity program requires either finding symmetries in every problem (implausible) or imposing them by fiat (subjective). Neither option delivers the foundational objectivity Jaynes sought.

Takeaway

Maximum entropy recommendations change under coordinate transformations, meaning the supposedly objective prior depends on arbitrary representational choices—a fatal flaw for foundational claims.

Acknowledging that maximum entropy fails as a foundation for objective Bayesianism doesn't render it useless. Within appropriate domains, the principle provides defensible methodological guidance. Statistical mechanics offers the paradigmatic success case, where maximum entropy distributions describe thermal equilibrium with remarkable accuracy.

The physics applications work precisely because the relevant symmetries and constraints are physically motivated rather than philosophically imposed. Microcanonical ensembles maximize entropy subject to fixed energy. Canonical ensembles maximize entropy subject to average energy constraints. These constraints emerge from physics—conservation laws and thermodynamic principles—not from abstract rationality requirements.

Beyond physics, maximum entropy methods prove valuable when we can justify the constraint structure on substantive grounds. In image reconstruction, spatial smoothness constraints reflect genuine prior knowledge about typical images. In linguistic modeling, frequency constraints capture real patterns in language use. The key insight: maximum entropy succeeds as a tool for encoding domain knowledge, not as a prior-free objectivity generator.

This reframing preserves maximum entropy's practical utility while abandoning its foundational pretensions. When experts can articulate what they know in constraint form, entropy maximization provides a principled method for extracting everything those constraints imply—nothing more, nothing less. The output remains sensitive to which constraints get included, but this sensitivity correctly reflects that knowledge encoding requires substantive choices.

The methodological lesson extends beyond maximum entropy specifically. Formal methods in epistemology rarely deliver the foundational objectivity their proponents sometimes claim. Their value lies in making assumptions explicit, deriving their consequences rigorously, and identifying where substantive choices enter. Maximum entropy clarifies that prior selection requires non-trivial decisions. It organizes those decisions through constraint specification rather than eliminating them entirely.

Takeaway

Maximum entropy succeeds not as a foundation for objectivity but as a rigorous tool for encoding domain knowledge—extracting everything specified constraints imply while making substantive choices explicit.

The maximum entropy principle represents the most sophisticated attempt to ground Bayesian reasoning in objective rationality. Jaynes's Concentration Theorem establishes genuine mathematical content—entropy-maximizing distributions are uniquely typical given constraints. This result merits attention regardless of one's philosophical commitments.

Yet transformation variance reveals that maximum entropy cannot deliver foundational objectivity. Prior selection remains irreducibly dependent on representational choices that carry no epistemic warrant. The program aimed to eliminate subjective elements from Bayesian inference but merely relocates them to parameterization and constraint specification.

What remains is still valuable. Maximum entropy provides powerful methodology for domains with well-motivated constraint structures, particularly physics and information theory. Its philosophical legacy lies not in solving the problem of priors but in clarifying its structure—showing precisely where and how substantive assumptions enter probabilistic reasoning.