184 real multi-agent failures, scored against the benchmark's labels
Who&When labels which agent caused each failure and at which step. Every method below sees the same runs, uses the same model, and never sees the answer.
By kind of run
How the comparison is kept fair
Same model for every LLM method, and no ground truth in any prompt.
The baseline prompt is the benchmark's own, copied from its code. The strongest baseline adds step numbers, because the official prompt makes the model count steps and it miscounts.
Exact step matching. The official scorer counts "12" as correct for "1"; this one does not.
Step-type priors are estimated leave-one-out, so a run's own label never informs its score.
95% bootstrap intervals; paired exact McNemar tests on the same runs.
Recorded from FalkorDB. For live queries, run docker compose up from the repo.