AI has mastered generating individual game assets and writing simple scripts, but it still struggles to build entire games from scratch. This gap exists because professional game engines are complex beasts, and until now, there was no large-scale data to teach AI how to handle them. Researchers have now changed that by creating JamSet and JamBench, the first datasets specifically designed to test AI on project-level code engineering within professional game engines.

The team realized that Game Jam competitions, where developers race to build complete games under tight deadlines, were a goldmine of open-source material. By leveraging the Godot engine’s text-based format and headless execution mode, they sifted through over 240,000 repositories to distill 8,133 verified projects. From these, they selected 300 manually verified projects to form JamBench, a benchmark that evaluates AI not just on writing code, but on whether that code actually runs correctly and behaves as intended.

The results reveal a startling capability cliff. While AI models handle small projects with an 80.4 percent success rate, that figure plummets to a mere 5.7 percent for larger, more complex games. Surprisingly, even advanced code agents that improve compilation rates fail to improve runtime behavior. This suggests the real bottleneck is not syntax, but architectural design. The AI can write the pieces, but it cannot yet assemble them into a functioning whole.

JamSet proves to be effective training data, offering a clear path forward for developers. The study highlights that until AI masters the high-level architecture of game design, it will remain a tool for snippets rather than a partner for creation. All data and code are now public, inviting the community to tackle this final frontier of game development automation.