13 Mainstream LLMs Compete Side by Side: The Multi-Agent Collaboration Benchmark Tears Off the Mask Hiding Their True Capabilities

A freshly released benchmark just rubbed all the "AI Agents can already collaborate autonomously" hype from the past six months into the ground.

The ALEM benchmark (Agents in Long-horizon Environments for Multi-agent coordination), launched in early July, dropped 13 mainstream LLMs into a Minecraft-like open world where they teamed up to explore, trade resources, craft tools, build structures, and fight monsters. The result was rather awkward—most models achieved an average normalized return of only about 6%. For comparison, a baseline multi-agent reinforcement learning (MARL) agent trained for 1 billion environment steps only barely outperformed the zero-shot Gemini 3.1 Pro under the hardest setting.

This statement has two layers of meaning. First, it puts out the LLM hype fire: stop claiming that agents can already autonomously do team collaboration—data doesn’t support it. Second, it tempers the MARL hype fire: if a policy trained with 1 billion steps barely matches a general‑purpose LLM that has never seen the environment, this line of work deserves a cost‑benefit re‑evaluation.

What exactly does this benchmark measure

ALEM’s ambition is to answer a question long avoided: Can LLM agents actually collaborate in long‑horizon, open‑ended worlds?

Over the past two years, there has been an explosion of multi‑agent benchmarks, but most stick to three patterns: social reasoning games like Werewolf, task‑based collaboration like databases or coding, or bilateral negotiation games. The common issue—the goal space is too narrow. Tasks have clear definitions; collaboration paths are largely fixed. As long as a model can follow the script, it looks fine.

ALEM makes the scenario heavy. It simulates an evolving sandbox world: maps to explore, resources to trade, tools requiring multi‑step synthesis, structures requiring group building, and monsters requiring team defense. There is no explicit "task completed" signal—only a composite normalized reward: you score points by surviving, producing, collaborating, and resisting threats.

Researchers deliberately distinguish two dimensions: long‑horizon task competence (can a single agent persist to finish a complex task?) and coordination ability (can multiple agents effectively collaborate?). The former has improved tremendously in recent LLMs—Claude 4.8, GPT‑5, Gemini 3.1 all perform decently on single‑agent long‑range tasks. But ALEM’s ablation experiments show something surprising: coordination ability is an independent bottleneck; it doesn’t automatically emerge as single‑agent capabilities improve.

More concretely, the authors dissected the collaboration harness—removing communication, shared memory, or role assignment separately. The result: communication mattered the most. In short, the models aren’t unintelligent—they just fail at deciding what to say, when to say it, and to whom.

What the 13‑model report card says

You can view the full leaderboard on the project page; here are several notable signals.

Gemini 3.1 Pro is the standout. Entering ALEM zero‑shot, it held its own against the MARL agent trained for 1 billion steps in the toughest collaborative scenarios. For Google, that’s a solid technical endorsement—it suggests Gemini 3.1 implicitly learned some transferable collaborative priors through world modeling and long‑term reasoning, rather than brute‑forcing via in‑context learning.

GPT and Claude families performed steadily but without surprises. They scored higher in tool‑use chains requiring long reasoning paths, but struggled on meta‑coordination behaviors like "interrupting proactively" or "yielding the floor." This matches another observation: USC’s MPDF framework found typical multi‑agent failure modes of "talkativeness," "early lock‑in," and "oscillation"—pitfalls to which conversational models are naturally prone.

Open‑source models were uneven. DeepSeek and Qwen flagships made the upper ranks, but some seemingly large‑parameter open models collapsed in function‑call executability. This aligns with prior MultiAgentBench findings—Meta‑Llama‑3.1‑70B once posted an oddly high coordination score (CS = 75) but a task score (TS) of only 0.21, because it could barely execute calls and just "held meetings" in text chat.

Why coordination is hard: not an IQ issue but a social one

Placing ALEM’s findings alongside recent academic work paints a clearer picture.

Problem 1: the communication protocol itself hasn’t been optimized. Most current multi‑agent frameworks make agents "send a snippet of natural language into shared context." That barely works with up to three agents; past five, the context window becomes a noise field. In ALEM’s ablations, cutting the communication channel tanked coordination scores; keeping it fully open induced floods of low‑information chatter like "I’ll go check the east side." Communication is either none or excessive—rarely just right.

Problem 2: missing a meta‑policy layer. USC’s MPDF (Meta‑Policy Deliberation Framework) highlights a deeper gap—today’s agents can "execute protocols" but not "decide protocols." They don’t know when to Persist, Refine, or Concede. Hence common failure patterns: a minority with the right answer gets drowned out by a wrong majority; or everyone makes irrelevant micro‑suggestions, wasting computation.

Problem 3: evaluation metrics have finally changed. Since MultiAgentBench, researchers have moved beyond final task scores to separate dimensions like "planning quality," "communication efficiency," "milestone progress," and "competitive play." ALEM goes further by embedding these into normalized rewards. That means if one "foreman" model does all the work while others loaf, the group doesn’t score high—collaboration must be proven genuine.

What this means for developers

Here are several takeaways applicable to engineering practice.

1. Stop picking multi‑agent backbones by single‑model scores. ALEM clearly shows that an individual model’s SOTA solo performance barely correlates with its behavior in a 5‑agent team. If you’re building multi‑agent applications (customer support, research assistants, code collaboration, etc.), test in real multi‑agent settings—don’t trust "general leaderboards."

2. Communication protocols are the next major engineering lever. ALEM’s "communication bottleneck" implies that a well‑designed protocol (structured messages, turn‑taking control, confidence reporting) could yield more improvement than switching to a stronger model. This aligns with MoA and sparse‑communication‑topology studies—architectural gains remain untapped.

3. Include process metrics when evaluating your own agent systems. Don’t just log "task success rate"; also track "who drove progress," "communication rounds," "information redundancy," and "minority‑opinion adoption rate." Many seemingly "working" multi‑agent systems turn out to be one agent doing all the work.

4. Cross‑model collaboration is the next frontier. A hidden gem in the ALEM leaderboard is that mixed‑vendor teams (e.g., Gemini as planner, Claude as executor, GPT as verifier) outperformed single‑vendor teams on some tasks. This benefits from "cognitive diversity" stemming from different pretraining corpora. For aggregation platforms, that’s good news—services that let one API key call GPT, Claude, Gemini, DeepSeek, etc. (like OpenAI‑compatible aggregation hubs) are lowering the barrier to multi‑model hybrid agent systems.

Plenty of open questions remain

There are still caveats worth noting.

ALEM’s environment is still somewhat "gamified" and not directly reflective of real enterprise collaboration (e.g., agents co‑authoring reports or debugging distributed systems). Whether the observed "6% average return" extrapolates to real‑world performance awaits replication. Also, the 13‑model roster hasn’t yet covered some newly released domestic flagships—future leaderboard updates will be worth watching.

A trickier issue: as agent collaboration grows more complex, interpretability drops sharply. MPDF introduced differential credit assignment to attribute team utility to individuals, but it’s far from auditable. In accountable domains like finance or healthcare, that’s a necessary hurdle.

In one sentence

ALEM’s greatest value isn’t a new leaderboard—it’s that it decomposes the vague notion of "LLM multi‑agent collaboration" into measurable, ablatable, falsifiable capabilities. Whoever can raise both coordination and task scores in the coming year earns the right to claim they’re truly building Agents.

As for products still pitching "our system will collaborate like human teams"—feel free to throw your system into ALEM; numbers don’t lie.

References

• New LLM Coordination Benchmark – r/MachineLearning — ALEM release post with TL;DR, ablation results, and project link
• alem-world/alem-env – GitHub — Open‑source repo for the ALEM environment with reproduction scripts and interaction traces
• 8 Task Collaboration Modes in LLM Multi‑Agent Systems — Overview of MoA, sparse‑communication topologies, and other collaboration modes
• 05‑26 · Latest LLM Paper Digest – Zhihu — Chinese summary of recent multi‑agent coordination frameworks such as MACA