Meituan VitaBench 2.0: A Real Challenge for AI Assistants

On June 9, Meituan’s LongCat team, together with the National University of Singapore, the University of Science and Technology of China, and other institutions, officially released VitaBench 2.0. This is a major upgrade from VitaBench 1.0 released in October last year, and is the industry's first evaluation benchmark for long-term dynamic user modeling in AI agents.

Straight to the conclusion: even the most powerful large models today have a success rate of just over 50% on this benchmark. What’s more frustrating is that adding a memory system to the models actually made their performance worse.

From “Can It Work” to “Does It Understand You”

VitaBench 1.0, released last year, mainly measured whether an AI agent could complete complex tasks — for example, cross-domain operations like booking a flight or finding a restaurant. The test results were already grim back then: even the best reasoning models only had about a 30% success rate.

But that was still a single task. In real life, our relationship with an AI assistant is not a one-off transaction. You might ask it to order takeaway today, book a hotel tomorrow, and find a restaurant the next day. A truly useful AI assistant should remember that you don’t eat cilantro, you prefer quiet hotels, and that you like window seats.

VitaBench 2.0 tests precisely this capability: In long-term interactions, can an AI truly understand and remember a user’s preferences?

Dataset: Simulating 1,580 Days of Human-AI Interaction

This benchmark is backed by substantial data:

• 56 user profiles covering various ages, professions, and lifestyles
• 2000+ dynamic preference annotations — not static preferences like “likes Sichuan cuisine” but preferences that change over time
• 819 subtasks covering three major domains: takeaway ordering, in-store consumption, and online travel
• 66 executable tools for real operations like search, ordering, payment, and reviews
• An average of 2,093 interaction events per user over 1,580 days

That last number is key — 1,580 days is over four years of interaction history. This means a model must extract a user’s real preferences from large, noisy, and unstructured dialogue data, while also detecting how those preferences change over time (academically known as “preference drift”).

For example: Three years ago, Xiao Wang frequently ordered strong-flavored food, but over the past year has switched to light meals for health reasons. If the AI still recommends spicy hot pot based on his three-year-old habits, it clearly hasn’t understood the user.

Test Results: Four Painful Findings

Meituan tested mainstream cutting-edge models and uncovered several serious issues.

1. Low Performance Ceiling

In the “full-context mode” (i.e., the model has access to the user’s entire history — the most ideal scenario), performance was:

| Model | Success Rate | |-------|--------------| | Claude-Opus-4.6 | 50.3% | | DeepSeek-V4-Pro | 45.6% | | Other leading models | <45% |

Keep in mind, this is like an “open-book exam” — the model can see all historical information without having to retrieve or remember it on its own. Even then, the best model only got half the tasks correct.

2. Adding Memory Actually Made It Worse

This is the most counterintuitive finding. In theory, giving an agent a memory system (either self-managed memory or RAG retrieval) should improve its performance.

The real results were the opposite:

• Self-managed memory: Key information was often lost or distorted during updates
• RAG retrieval: Often distracted by superficially similar but irrelevant information

Both approaches performed worse than just dumping the full context to the model.

This suggests current memory mechanisms are still primitive⁠ — they can store information but cannot truly understand what is important, outdated, or related.

3. Chain-of-Thought Doesn’t Help Personalization

Enabling chain-of-thought reasoning (as in DeepSeek-R1, o3, etc.) often boosts logic tasks. But in VitaBench 2.0, it offered no systematic advantage.

The reason is clear: personalization isn’t about deduction but about:

• Filtering useful information from noise
• Detecting preference changes over time
• Actively asking questions when information is lacking

These abilities go beyond step-by-step reasoning. You can’t deduce that a user recently became vegetarian — you have to ask them.

4. Doesn’t Ask Questions

This may be the most fatal flaw. When humans lack enough information to decide, we naturally ask follow-ups. Current models prefer to guess.

Tests show Claude’s success rate in active-interaction tasks plummets from 46.0% to 27.4%. Models would rather make a potentially wrong decision than admit uncertainty.

Worse yet, even when given a user’s real preference data directly, models still struggled to integrate and apply this information in multi-step tool invocation scenarios.

Error Analysis: From “Can’t Use Tools” to “Doesn’t Understand Users”

Meituan’s error analysis revealed an interesting trend:

As models improve, error types are shifting structurally.

Taking the DeepSeek series as an example:

• DeepSeek-R1: High proportion of tool-use errors
• DeepSeek-V4-Pro: Tool errors drop significantly; preference-understanding errors dominate

In other words, the “can it use tools” problem is being solved, but “does it understand the user” is now the main bottleneck.

For the industry, this implies the next phase of AI agent competition will shift from tool usage skills to user understanding — whether it can read between the lines, remember long-term preferences, and detect preference shifts.

Technical Details: How VitaBench 2.0 Is Designed

Evaluation Framework

VitaBench 2.0 models agent-environment interaction as a Partially Observable Markov Decision Process (POMDP), quantifying task complexity in three dimensions:

1. Reasoning complexity (C_reason) — size of the task’s search space
2. Tool complexity (C_tool) — number of tools needed and interdependencies
3. Interaction complexity (C_interact) — number of user-agent exchanges required to clarify needs

Scenario Coverage

The three chosen domains are highly representative:

• Takeaway ordering: High frequency, fast decisions, stable preferences
• In-store consumption: Medium frequency, more complex decisions, context-dependent preferences
• Online travel: Low frequency, highly complex, highly personalized preferences

They cover decision types from simple to complex and vary in preference modeling difficulty.

Evaluation Method

Traditional evaluation compares the final database state, which misses non-state-changing actions like recommendations or route planning.

VitaBench 2.0 uses a rubric-based sliding window evaluator:

1. Break task goals into atomic rubric items
2. Slide a window over full dialogue history
3. Track completion status of each rubric
4. Judge task completion with an “all-or-nothing” rule

This captures process details, not just final outcomes.

Comparison with VitaBench 1.0

| Dimension | VitaBench 1.0 | VitaBench 2.0 | |-----------|---------------|---------------| | Focus | Single complex task | Long-term dynamic user modeling | | Time span | Single session | 1,580 days | | User preferences | Static needs | Dynamically evolving preferences | | Core challenge | Reasoning + tool use | Personalization + proactive interaction | | Best model success rate | ~30% | ~50% |

VitaBench 1.0 asked: “Can AI complete complex tasks?”
VitaBench 2.0 asks: “Can AI truly understand the user?”

Why Does This Benchmark Matter?

For Researchers

VitaBench 2.0 points toward the next direction in agent research: from tool use to user understanding.

Current memory mechanisms (whether vector databases or self-managed memory) are still primitive and far from practical. Designing memory systems that can correctly extract, update, and retrieve user preferences is a worthy research topic.

Also, teaching models to proactively ask questions instead of guessing when uncertain may require fundamental changes to training objectives and reward functions.

For Developers

If you’re building an AI assistant product, VitaBench 2.0’s findings are worth noting:

1. Don’t over-rely on long context — longer context doesn’t mean better preference extraction
2. Be cautious with RAG — in personalization, RAG can backfire
3. Prioritize proactive interaction design — better for the model to ask than guess
4. Handle preference drift — preferences change; systems must detect and adapt

For the Industry

Meituan clearly built this benchmark to pave the way for its AI assistant products. But objectively speaking, its design is rigorous, the dataset is substantial, and the findings are valuable.

It reveals the true deployment bottleneck for AI agents: The problem isn’t that models aren’t powerful — it’s that they don’t understand users well enough.

Any company aiming to build AI assistants or agents should take this as a reminder: the technical focus may need to shift from “making models smarter” to “making models more human-aware.”

Open-Source Resources

VitaBench 2.0 is fully open-sourced, including:

• Code repository: https://github.com/meituan-longcat/vitabench
• Dataset: https://huggingface.co/datasets/meituan-longcat/VitaBench
• Online leaderboard: vitabench.github.io (continuously updated)

Interested developers can test their agents against this benchmark or conduct further research using the dataset.

Final Thoughts

When VitaBench 1.0 launched last year, the problem was “AI can’t handle complex tasks.” This year’s 2.0 version says, “AI can’t remember who you are.”

In a sense, that’s progress — at least tool-use problems are being addressed. But the newly exposed problem is equally tricky: how to make AI truly understand users, not just execute instructions.

A 50% success rate isn’t terrible, but considering this is in an “open-book” scenario, real-world performance could be worse.

For ordinary users, this means you shouldn’t expect AI assistants to truly “get you” in the short term. For developers, it’s an opportunity — whoever cracks user understanding first will gain a real moat in the AI assistant race.

---

References:

• Meituan LongCat VitaBench GitHub Repository — Source code and documentation
• VitaBench Dataset - Hugging Face — Evaluation dataset download
• Meituan LongCat team releases VitaBench - Zhihu Column — Technical deep dive article