We assume AI agents can handle complex real-world tasks, but a new benchmark reveals they struggle significantly when rules change mid-game. Researchers have introduced AdaPlanBench, a dynamic testing environment that exposes the fragility of large language model planning when constraints are revealed progressively rather than upfront.

The study focuses on how agents adapt when they must plan and re-plan under dual constraints from both the physical world and the user. Unlike traditional benchmarks that list all rules at the start, AdaPlanBench hides constraints until an agent proposes a plan that violates them. Built on 307 household tasks, the system uses a multi-turn protocol where agents receive iterative feedback. This forces the AI to infer hidden rules from its mistakes and revise its strategy on the fly, mimicking the unpredictable nature of real-life interactions.

The results show a stark gap between current capabilities and reliable performance. Among ten leading large language models, the best performer achieved only 67.75% accuracy. Performance degraded noticeably as more constraints accumulated, with user-specific rules proving particularly difficult to track. Failures often stemmed from weak physical grounding and reduced effectiveness.

This research establishes a critical testbed for interactive planning, highlighting that current AI agents lack the robustness needed for dynamic environments. The findings suggest that while models can follow static instructions, they remain unreliable when forced to adapt to evolving, dual-layered constraints. Until physical grounding and adaptive reasoning improve, AI will continue to stumble when real-world rules are not fully known in advance.