LLM Benchmarking Under Scrutiny: MUD Experiment Reveals Judge Inconsistency

An independent research paper investigates the use of Multi-User Dungeons (MUDs) — text-based adventure games — as an environment for evaluating Large Language Models (LLMs). The experiment aimed to assess LLM performance in complex, interactive scenarios.

The core methodology involved having LLMs play within a MUD, with their actions and responses then scored across several behavioral dimensions. Crucially, two of these dimensions relied heavily on another LLM acting as a classifier or judge.

A significant finding emerged when these LLM-classifier-dependent scores were re-evaluated. One frontier model's ranking dropped six positions upon their removal. Further investigation revealed a low agreement between the primary LLM judge and a second, independent judge, with per-model agreement ranging from 85% down to 22%.

The researchers noted a particularly low aggregate kappa (0.04 on probe detection), indicating significant noise in the instrument. This noise disproportionately affected models from the same family as the LLM classifier, raising questions about potential bias or an echo chamber effect in LLM-judged evaluations.

"The divergence between the two judges is the finding we think generalizes to other judge-based benchmarks."

The team emphasizes this work is a proof of concept, not a validated benchmark, citing limitations such as only 50 runs per model, overlapping confidence intervals, and the absence of human raters. Everything from the paper and data (CC BY 4.0) to the code (MIT) is publicly available, reflecting the researchers' commitment to transparency and collaborative improvement. The entire experiment was conducted on personal computers with approximately $99 in API credits.

For a Tokyo-based professional leveraging LLMs, this research underscores a critical caveat: benchmarks relying solely on AI judges may present an incomplete or even misleading picture of model capabilities. It suggests that claims of superior performance, especially those from vendors using proprietary LLM-based evaluation, warrant careful scrutiny.

This implies that robust internal evaluation frameworks, potentially incorporating human oversight or diverse, non-LLM-based metrics, are essential for any business professional in Tokyo making decisions about deploying or fine-tuning LLMs for specific Japanese-language tasks. Relying on public leaderboards without understanding their evaluation methodology could lead to suboptimal model choices.