Harvey Open-Source Legal AI Benchmark LAB: 1,200 Tasks Approaching Real Law Firm Workflows

The legal AI company Harvey, valued at $3 billion, has open‑sourced its own yardstick for measuring the intelligence of legal agents.

This week, Harvey released Legal Agent Benchmark (LAB)—an open‑source benchmark for long‑horizon legal agents. The first version includes 24 practice areas, 1,200+ tasks, and over 75,000 evaluation criteria handwritten by lawyers. Both the code and dataset are available on GitHub. Interestingly, Harvey intentionally withheld a leaderboard this time—in their words, “we’ll refine the scoring together with the community before making it public.”

Why Harvey Is Doing This

For context: Harvey was founded in July 2022—a few months before ChatGPT—and has become the most successful AI company in legal applications, bar none. In 2024 its ARR reached $50 million and by early 2025 was approaching $100 million; in February it closed a $300 million Series D led by Sequoia, with a $3 billion valuation. The client roster speaks volumes: 28 of the U.S. Law Week Top 100 firms now use Harvey.

So when Harvey says “we’ve built a benchmark for legal agents,” that statement carries far more weight than a benchmark released by a research lab or general model vendor—they are actually the ones gathering daily feedback from top‑tier firm workflows.

Existing legal evaluations mostly stop at “single‑round QA” or “document classification”—for example, identifying clause types in a contract or answering a statutory question. Such tests help pick base models but are nearly useless for judging whether an agent can truly do a junior lawyer’s job. Real legal work? A due‑diligence memo in an M&A project may require reading dozens of contracts, tracing hundreds of cross‑references, tailoring risk disclosures to business objectives, and going through several partner review rounds. It’s a long‑horizon, multi‑document, multi‑review process—not something one‑shot tests can capture.

LAB aims to close that gap.

What LAB Actually Measures

Each LAB task has three parts:

• An instruction – simulates how partners or senior associates assign work, often vaguely: e.g., “help me prepare a risk analysis for next week’s client meeting.”
• A client case packet – includes all relevant materials: contracts, emails, minutes, prior judgments, etc.
• A delivery requirement – the agent must submit a reviewable work product, not a chat reply.

This structure deliberately mimics the internal “assignment – execution – review” collaboration chain in big‑law firms. As Harvey’s product lead Aatish Nayak once said, they want the agent interaction to feel like a colleague, not a tool—LAB’s task design follows that philosophy.

How are the 1,200 tasks distributed? They cover 24 fields including litigation, corporate, capital markets, M&A, compliance, IP, tax, labor, and real estate. The cost behind that figure is significant—Harvey says all 75 thousand evaluation criteria were written manually by practicing lawyers, averaging 60+ scoring items per task. Anyone who has done legal data annotation knows how expensive that is—no wonder academia has struggled to build serious long‑horizon legal‑agent tests.

No Leaderboard—A Smart Restraint

The choice to omit a leaderboard in the first release deserves attention.

A typical benchmark launch pairs a table showing GPT, Claude, Gemini ranked top‑to‑bottom—with the sponsor’s model conveniently on top. Harvey intentionally did the opposite, stating in its blog: “the dataset will be continuously updated; we want to work with the community to ensure results intuitively reflect real agent performance.” In coming weeks they’ll partner with researchers to run baselines and will also publish standardized submission guidelines so progress can be tracked.

This restraint has clear logic. Legal‑agent outputs are long documents—memos with citations, drafted clauses—not multiple‑choice answers. How to consistently apply 75 k expert scoring criteria across wildly different submission formats is itself a research problem. A premature leaderboard would likely reward “who best templates LAB’s format,” polluting the signal. Setting up the scoring pipeline and submission norms first, then publishing rankings, makes far more sense.

There’s another layer: Harvey is a heavy OpenAI user (they’ve publicly said they access OpenAI via Azure). As the benchmark initiator, their own placement on any leaderboard would be sensitive. Leaving that to research partners is a tactful move.

What This Means for Different Players

For model vendors: LAB is a tough test of long‑horizon‑agent capability. Until now, AI‑agent claims often cited SWE‑bench or GAIA—mostly coding or general‑knowledge tasks. Legal work is special: zero tolerance for hallucination, every citation traceable, reasoning must withstand courtroom scrutiny. A completely different proving ground. Expect leading model firms to chase LAB scores—who leads in Corporate vs. Litigation may soon become a new enterprise‑sales pitch.

For agent builders: Teams developing legal‑domain agents have long struggled with proof of effectiveness—firms rarely spare hours for A/B trials. LAB now offers a public benchmark dimension. Since the 1,200 tasks are organized by specialty, niche products (tax‑focused, IP‑focused, etc.) can demonstrate value within their slice without competing head‑on with generalist tools.

For law firms: Harvey’s blog puts it plainly—one goal of LAB is helping firms measure AI ROI. Nearly all major firms are signing AI‑tool budgets, yet partners can’t answer, “exactly how many billable hours did this save?” LAB provides a structured language: “In corporate law, the best current agent can autonomously complete X% of tasks, partially complete Y%, and fail Z%.” Such graded metrics are far stronger than vague claims like “AI saves 30% of time.”

A Broader Signal

Zooming out, LAB reflects Harvey’s enduring product philosophy. CEO Winston Weinberg often stresses that law is not a scene solvable by “wrapping GPT in an app”—the gap between foundation models and industry needs is deeply underestimated. Harvey’s edge has never been the base model itself but the composite AI system above it—hundreds of model calls chained into workflows, line‑level citation tracing, and agent collaboration mimicking firm division of labor.

LAB essentially publicizes Harvey’s internal definition of “what makes a good legal agent.” This transparency may even help Harvey’s business—once everyone measures by the same yardstick, its system strengths become undeniable. In single‑round QA Harvey may look close to Claude, but in LAB’s long‑form deliverable tasks, the gap widens.

At a macro level, LAB marks maturation of vertical‑domain agent evaluation. For two years we’ve seen endless marketing like “our agent beat GPT‑4 on xx benchmark,” but most were short, simplistic, or context‑free. LAB’s creation—lawyers writing 75 k scoring criteria, simulating full workflows—shows the industry finally realizes: genuine productivity evaluation costs money, time, and must be defined by domain experts.

For developers calling GPT, Claude, Gemini, DeepSeek via OpenAI Hub, LAB offers a fresh lens—if you’re building legal, compliance, or due‑diligence agents, run the LAB task set and see which base model performs best in your sub‑domain before choosing your path.

The dataset and code are open on GitHub at harveyai/harvey‑labs. Anyone curious can read how those scoring rubrics were written—their richness alone surpasses most legal‑AI papers.

References

• Harvey LAB open‑source repository – GitHub: full code and dataset for LAB benchmark
• Harvey releases long‑horizon legal‑agent benchmark – Linux.do: community discussion on LAB launch