Modern AI has a power problem that better chips alone cannot solve. Training a frontier language model consumes energy measured in gigawatt-hours. Inference, which means running the model millions of times a day, often exceeds that cost over a deployment's lifetime. As models scale, energy has shifted from an operational footnote to a first-order design constraint.

Most engineers treat efficiency as a downstream concern. They choose the architecture for accuracy, then quantise, prune, and distil until it fits the budget. This sequence leaves significant gains on the table, because the largest determinants of energy consumption are fixed the moment you decide how data flows through the network.

The architecture determines how many operations execute, what precision they use, and, most importantly, how often data must travel between memory and compute. These choices compound across billions of inferences. Understanding them requires looking past parameter counts and FLOPs toward the physical cost of computation. That means knowing where the joules actually go, how to measure what they buy, and why the target hardware should shape the design from the very start.

The most useful reference point for energy-aware design comes from Mark Horowitz's widely cited 2014 analysis of operation costs at a 45nm process node. A 32-bit floating-point multiply consumed roughly 3.7 picojoules. An 8-bit integer add consumed about 0.03. Reading 32 bits from off-chip DRAM consumed around 640 picojoules, more than a hundred times the cost of the arithmetic it fed.

Absolute values have shifted with newer process nodes, but the hierarchy has held. Data movement dominates computation. An operation's energy is determined less by what it computes than by where its operands live: registers, on-chip SRAM, or external memory. Each step outward in that hierarchy costs an order of magnitude or more.

This reframes how layer types compare. Standard convolutions reuse each weight across many spatial positions, which gives them high arithmetic intensity: many operations per byte fetched. Depthwise separable convolutions, popularised by MobileNet, cut FLOPs by roughly eight to nine times. However, their depthwise stage performs few operations per byte loaded, so on many accelerators they deliver smaller energy savings than the FLOP reduction suggests. Fully connected layers at small batch sizes are worse still, because every weight is fetched once and used once.

Attention follows the same pattern at scale. Its quadratic score matrix is expensive less because of the multiplications than because naive implementations write it to high-bandwidth memory and read it back. FlashAttention computes identical results while tiling the work to stay in on-chip SRAM, which cuts memory traffic substantially. Similarly, the key-value cache makes autoregressive decoding memory-bound. This is why grouped-query attention, which shares keys and values across heads, reduces energy per token without altering the core mechanism.

Takeaway

Count bytes moved before you count operations. An architecture's energy profile is written in its memory access patterns, and FLOPs are only the visible part of the bill.

Leaderboards rank models by accuracy, which implicitly treats compute as free. Suppose a model gains 0.3 percentage points on ImageNet while doubling inference energy. It appears strictly better on a leaderboard, yet for most deployments it is the worse choice. Efficiency-aware evaluation requires a different objective: accuracy delivered per unit of energy, measured under realistic conditions.

The first step is replacing proxies with measurements. FLOPs and parameter counts correlate poorly with energy because they ignore memory traffic, precision, and hardware utilisation. Two models with similar FLOPs can differ several-fold in joules per inference. Measure energy directly with on-device power counters such as NVIDIA's NVML or Intel's RAPL. Alternatively, follow the methodology of MLPerf Power, which standardises measurement across systems.

The second step is evaluating along a Pareto frontier rather than a single ranking. Plot accuracy against joules per inference for every candidate. Models below the frontier are dominated, because another option is both more accurate and cheaper. EfficientNet shows why measurement matters here. EfficientNet-B0 slightly exceeded ResNet-50's accuracy with roughly a tenth of the FLOPs. Yet its depthwise convolutions often ran slower on GPUs than that reduction implied, which is exactly the gap that energy measurement exposes.

Finally, define the operating point before comparing. Energy per inference depends on batch size, sequence length, precision, and whether the workload is latency-bound or throughput-bound. A model that is efficient at batch size 256 in a data centre may be wasteful at batch size 1 on a phone. Report accuracy per joule under the conditions you will actually deploy, and the rankings frequently reorder themselves.

Takeaway

A model's accuracy is only meaningful alongside the price paid for it. Evaluate on a frontier, not a leaderboard, because the best model is the one nothing else dominates at your operating point.

Energy efficiency is not a property of an architecture in isolation. It emerges from the interaction between the model's computational pattern and the hardware executing it. A design that suits a GPU's tensor cores may be wasteful on a mobile NPU, and the reverse is also true. Treating hardware as an afterthought guarantees a mismatch.

The roofline model provides the framework. Every processor has a peak compute rate and a peak memory bandwidth. Their ratio defines the arithmetic intensity at which a workload shifts from memory-bound to compute-bound. Layers below that threshold leave arithmetic units idle while still drawing power. Kernel sizes, channel widths, and layer fusion opportunities should therefore be chosen to place critical layers near or above the hardware's ridge point.

Small dimensional choices matter. Tensor cores on NVIDIA GPUs operate on tiles, so matrix dimensions in multiples of 8 for FP16 avoid padding and wasted cycles. Mobile NPUs often accelerate a fixed set of operators and fall back to the CPU for anything else. A single unsupported activation function can therefore force costly data transfers. Precision support varies too, and INT8 is efficient only where the hardware has native integer pipelines.

Hardware-aware neural architecture search formalises these constraints. Google's MnasNet built measured latency on Pixel phones directly into its search objective. It found models 1.8 times faster than MobileNetV2 at slightly higher accuracy. ProxylessNAS and Once-for-All extended the approach by producing specialised subnetworks for each target device from a single trained supernet. The lesson applies beyond automated search: profile on the target hardware early, and let those measurements constrain the design space.

Takeaway

There is no universally efficient architecture, only architectures well matched to their hardware. Design for the chip you will deploy on, not the one you trained on.

Energy efficiency is an architectural property, not a post-processing step. The decisions that matter most are made before training begins: how data flows, how often it moves, and how well the computation maps to silicon. Compression techniques refine a design. They rarely rescue one.

Three principles follow. Prioritise memory access patterns over raw operation counts, because data movement dominates the energy budget. Evaluate candidates on an accuracy-per-joule frontier at your real operating point. Profile on target hardware from the first prototype, and treat its constraints as design inputs.

As AI deployment scales, the cost of each inference multiplies across billions of requests. The most sustainable architecture is not necessarily the smallest one. It is the one whose structure fits both the task and the machine.