Imagine training a massive AI model but using significantly less energy to achieve better results. That is the promise of SMELT, a new architectural approach that challenges the conventional wisdom that bigger always means better. By rethinking how transformer layers are organized, researchers have found a way to squeeze more intelligence out of the same computational budget, potentially reshaping how we build efficient large language models.

The core idea involves Looped Transformers, which increase a model’s effective depth by iterating through a shared block of layers multiple times rather than stacking them linearly. Previous studies often conflated this architectural advantage with simply throwing more computing power at the problem. To address this, the researchers designed SMELT, or Sparse MoE Transformer with middle layers Looped Twice. They ensured that every version of the model used the same amount of processing power, parameters, and memory cache. This strict control allowed them to isolate the true benefit of looping, proving that reusing layers is not just a trick but a viable path to efficiency.

The results were striking. When scaled up to models with 54 billion non-embedding parameters, SMELT demonstrated a faster drop in loss relative to compute usage. This translates to a 6.8 to 18.0 percent saving in training FLOPs on the optimal frontier. The benefits were not just theoretical; they transferred to real-world benchmarks, particularly in coding tasks. The model performed even better with longer contexts and more examples, driven by a mechanism where the second pass through the layers reduces attention sinks and focuses processing on relevant content.

This research offers a practical recipe for the future of AI development. It proves that looping can improve transformers even under strict budget constraints, turning depth reuse into measurable gains. As the industry races toward larger models, SMELT suggests that smarter architecture design can deliver superior performance without the exponential cost, making powerful AI more accessible and sustainable.