Imagine a high-speed highway where traffic lights ignore congestion, causing gridlock even when lanes are technically empty. This is the hidden bottleneck in serving modern large language models. Researchers have identified a critical flaw in how current systems route requests to processing units, revealing that balancing load alone is not enough to keep AI models running at peak speed.

The issue lies in how mixture-of-experts models operate. These models contain multiple specialized sub-networks, or experts, that activate differently for each request. Existing routers assign tasks based solely on worker load, ignoring that different requests require different experts. This mismatch causes latency spikes because workers must constantly load and switch between heavy expert weights, even if they are not overloaded in terms of simple task count.

To solve this, researchers developed ELDR, an expert-locality-aware decode router for prefill-decode disaggregated serving. By analyzing early signals from a request, ELDR predicts which experts will be needed and directs the task to the worker best equipped to handle them. This approach minimizes the costly overhead of switching between different model components, ensuring that each worker processes a consistent set of experts.

The results are significant. In tests across three MoE models and two workloads, ELDR reduced median TPOT by 5.9-13.9% compared to the strongest of four load-balancing baselines. Crucially, this speed boost comes without altering the model’s output quality. As AI models grow larger and more complex, smart routing that respects expert locality will be essential for delivering the fast, responsive experiences users expect.