Your Video Generator Is Stuck on Frame One
Based on research by Yusuf Dalva, Pinar Yanardag
Video generation models have a fatal flaw: they are too obsessed with the first frame. This initial image acts as a rigid anchor, locking the camera angle and suppressing motion to maintain static consistency. The result is often a visually shallow video where the scene feels stuck, unable to evolve naturally as time progresses.
Researchers have identified that this issue stems from how the model’s attention cache works. The first frame’s data occupies a privileged position, drawing disproportionate attention and effectively freezing the composition. To fix this, they introduced an adaptive state, a hidden latent that the model denoises alongside content at every chunk but never renders. This state evolves alongside the video content, allowing the scene to change dynamically without losing coherence.
The innovation lies in treating time as relative rather than absolute. Instead of referencing a frozen starting point, the model generates its own scene anchor at each step by attending to both the previous state and current content. This creates a recurrence in the generation process where the denoising action serves as the transition function and the KV cache serves as the carrier, requiring no external module. The system updates its internal reference continuously, ensuring that motion and camera movement are no longer dampened by an unyielding initial view.
Experiments show that this adaptive approach substantially improves video dynamics. By allowing the scene to progress naturally, the generated videos feature richer motion and more realistic evolution. This breakthrough suggests that future video models can break free from the tyranny of the first frame, producing more engaging and dynamic visual narratives.