Traditional signal processing has long relied on discrete representations: pixels for images, samples for audio, voxels for volumetric data. These grid-based formats impose a fundamental coupling between representational fidelity and memory consumption, scaling poorly with dimensionality and offering no principled mechanism for querying between sample points. The discretization is a computational convenience, not a mathematical necessity.

Implicit neural representations invert this paradigm. Rather than storing signal values at fixed locations, a coordinate-based network f_θ: ℝ^d → ℝ^k parameterizes the signal as a continuous function, with weights θ encoding the entire signal. Querying becomes forward evaluation; storage becomes model size, decoupled from spatial resolution.

Yet naive multilayer perceptrons fail catastrophically at this task. Standard ReLU networks trained on coordinate inputs exhibit severe spectral bias, converging rapidly to smooth, low-frequency approximations while ignoring the high-frequency content that constitutes visual and geometric detail. Understanding why requires examining the neural tangent kernel and its eigenspectrum, which reveals how architectural choices—positional encoding, periodic activations, hash grids—act as principled modifications to the inductive bias, expanding the class of representable functions at prescribed frequencies.

The spectral bias of coordinate-based MLPs admits a precise characterization through the neural tangent kernel (NTK). For a wide network trained under gradient descent, learning dynamics are governed by the eigendecomposition of the NTK evaluated on training coordinates. Crucially, the NTK of a standard ReLU MLP on low-dimensional inputs is dominated by low-frequency eigenfunctions, with eigenvalues decaying rapidly for high-frequency components.

This decay dictates convergence rates: components aligned with large eigenvalues are learned in early iterations, while high-frequency modes require exponentially more optimization steps. In practice, this manifests as blurry reconstructions that never sharpen regardless of training duration.

Fourier feature mappings resolve this by preprocessing coordinates through γ(x) = [cos(2πBx), sin(2πBx)], where B is a matrix sampled from a chosen distribution. The composed kernel k(x, y) = h(Bx − By) becomes a stationary kernel whose bandwidth is controlled by the spectrum of B.

Sampling B from a Gaussian with variance σ² yields a Gaussian kernel with bandwidth proportional to σ, effectively flattening the NTK eigenspectrum across a targeted frequency band. The choice of σ becomes a hyperparameter trading off approximation of high-frequency detail against generalization noise.

This reframing illuminates positional encodings not as engineering tricks but as explicit priors over signal bandwidth. The transformer's sinusoidal encoding, NeRF's frequency embedding, and random Fourier features all instantiate the same principle: modulating the kernel spectrum to align learnable functions with the target signal class.

Takeaway

Positional encoding is not a heuristic—it is a deliberate kernel design choice that reshapes what functions gradient descent can efficiently learn.

Sinusoidal Representation Networks (SIRENs) replace ReLU activations with sin(ω₀ · Wx + b), embedding periodicity into every layer rather than only at the input. This seemingly minor change induces profound consequences for representational capacity and derivative regularity.

The derivative of a SIREN with respect to input coordinates remains a SIREN, since d/dx sin(f(x)) = cos(f(x)) · f'(x) preserves the sinusoidal structure. Consequently, gradients, Laplacians, and higher-order differential operators are themselves neural networks with matching expressive power—a property no ReLU architecture possesses, since ReLU derivatives are piecewise constant and vanish for higher orders.

This makes SIRENs uniquely suited for supervising signals through differential constraints: fitting a function while simultaneously matching its gradient field, solving Eikonal equations for signed distance functions, or training via the Poisson equation. The network can represent solutions to partial differential equations natively.

Initialization requires careful analysis. Sitzmann et al. derive that weights sampled from U(−√(6/n), √(6/n)), combined with a first-layer frequency multiplier ω₀ ≈ 30, preserves the distribution of activations across depth and prevents the sinusoidal nonlinearity from either saturating or collapsing to linear behavior.

The spectral behavior of SIRENs differs qualitatively from Fourier-featured MLPs. Rather than a fixed input frequency basis, SIRENs learn frequency compositions throughout depth, producing a hierarchy of oscillatory features. Empirically this yields sharper reconstructions of natural signals but demands greater care in optimization landscape design.

Takeaway

When your loss depends on derivatives, choose an activation whose derivatives are equally expressive—regularity of the parameterization is not a detail but the substrate of what can be learned.

The choice between implicit and explicit signal representations reduces to a tradeoff along three axes: memory scaling, query cost, and editability. Discrete grids offer O(1) lookup but O(N^d) storage in d dimensions; implicit networks reverse this, offering compact parameterization but requiring a forward pass per query.

For high-dimensional signals—radiance fields, deformable geometries, 4D light transport—the exponential blowup of discrete storage renders implicit representations not merely elegant but necessary. A 512³ voxel grid consumes gigabytes; a comparably expressive MLP fits in megabytes.

Yet pure implicit representations exhibit slow training and inference. Recent hybrid architectures—instant-NGP's multi-resolution hash grids, TensoRF's tensor decompositions, plenoxels' sparse grids—recognize that the continuous parameterization need only be applied where signal complexity demands it. Learnable feature grids interpolated and decoded by small MLPs achieve orders-of-magnitude speedups while preserving continuity.

The theoretical picture unifies through the lens of function approximation over reproducing kernel Hilbert spaces. Discrete representations correspond to Nyquist-Shannon sampling of bandlimited signals; implicit networks correspond to overparameterized approximation with learned bases; hybrids interpolate between these regimes by factorizing the signal into fixed structural priors and learned residual complexity.

Editability introduces further asymmetry. Modifying a pixel in an image is trivial; modifying a spatially localized region of an implicit representation requires either fine-tuning subject to catastrophic forgetting or architectural mechanisms for spatial disentanglement, an active research direction.

Takeaway

Continuous and discrete representations are not competitors but complementary parameterizations of the same underlying function space—the art lies in choosing where continuity earns its cost.

Implicit neural representations reframe signal modeling as function approximation in the truest sense: the network is the signal. This shift dissolves long-standing boundaries between graphics, geometry processing, and machine learning, since a common mathematical object—the coordinate network—now serves purposes previously requiring distinct data structures.

The theoretical foundations remain incomplete. Spectral analysis via NTKs offers insight in the infinite-width limit, but practical networks operate in the feature learning regime where kernel approximations break down. Understanding how activation choice, initialization, and architectural priors interact to shape the learned function class is an open frontier.

What is clear is methodological: representational power emerges from principled alignment between the inductive bias of the architecture and the spectral, geometric, and differential structure of the target signal. Progress will follow those who treat network design as a form of applied harmonic analysis.