We assume AI agents are robust, but they are actually fragile. When a tool fails, most models do not recover; they just keep failing in loops. This disconnect between idealized testing and messy reality is the core problem researchers are finally tackling with a new benchmark called ToolMaze.

ToolMaze moves beyond the happy paths where everything works perfectly. It tests how LLM agents handle dynamic replanning and anomaly recovery when tools break. The benchmark uses a two-dimensional design to measure this. It combines topological complexity with a taxonomy of tool perturbations. These perturbations are either explicit or implicit, and either transient or permanent. This setup separates true systematic replanning from blind trial-and-error.

The results reveal a startling vulnerability. Performance degrades across nearly all models when tools are perturbed. The sharpest drops occur under implicit semantic failures. Here, agents suffer from systemic over-trust in corrupted outputs. The Perturbation Recovery Rate plummets by around thirty-seven percent in these scenarios. Furthermore, complex task structures trap agents in futile loops. They cannot escape the error because they cannot distinguish between a temporary glitch and a fundamental failure.

The most critical finding is that fault tolerance does not scale well. Agentic fault-tolerance improves at a rate of only 3.66 times slower than basic task execution. This highlights dynamic replanning as a distinct bottleneck. It is a problem that neither model scaling nor better prompting can solve. The data and code for ToolMaze are available at https://github.com/Zhudongsheng75/ToolMaze.