If you are building AI apps, your model’s internal numbers might be breaking your code. New research reveals that activation spikes in modern large language models are far more volatile and unpredictable than previously thought, posing a silent threat to efficient deployment.

Researchers analyzed 27 checkpoints from eight different open-source model families to measure how large internal data values can get during processing. These activations are crucial for low-bit quantization, a technique used to shrink models so they run faster and cheaper on consumer hardware. The study tracked these values across embeddings, attention layers, and final outputs to understand the true dynamic range of modern AI systems.

The findings were startling. Activation peaks varied by nearly four orders of magnitude between models of similar size. While some models stayed small, others like Gemma3-27B-it spiked to massive levels. Surprisingly, Mixture of Experts architectures showed significantly lower peaks than dense models, defying expectations. Most critically, the residual stream consistently carried the highest values, meaning ignoring it during optimization leads to catastrophic errors.

The takeaway is clear: maximum activation magnitude is a specific property of a model’s architecture and training, not just its size. Before deploying open-weight models with low-bit quantization, developers must measure and report these peaks. Relying on old assumptions about activation scaling will result in poor reconstruction and unstable inference. Always verify the dynamic range of your specific model family to ensure reliable performance.