🔍 Read the full analysis: What Z.ai’s Legal AI Benchmark Results Say About Its Ranking on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria task pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, placing seventh among 81 models in the archived version. It passed 92.5% of individual criteria, but the benchmark counts a task as complete only when every criterion passes. Z.ai’s claimed 12.5-percentage-point gain over its base model matches the listed scores; its open-source ranking claim is not substantiated by a named comparison set in the supplied material.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on an archived Vals AI leaderboard for Harvey’s Legal Agent Benchmark, recording a 19.17% all-criteria pass rate. The result puts the legal-focused model ahead of Z.ai’s base GLM 5.3 Flash entry, but the score means it met every requirement on fewer than one in five evaluated tasks—not that it achieved only that level on every individual element of the work.
The leaderboard score comes from an all-or-nothing grading rule: a task passes only when every criterion in its rubric passes. Vals’ archived listing reports GLM Legal at 19.17% of tasks, rounded to 19.2% in Z.ai’s documentation. The evaluated set contains 120 private tasks, according to Vals’ description of the benchmark procedure.
That task score sits alongside a 92.5% criteria pass rate for GLM Legal in the same leaderboard version. These figures measure different things. A model can satisfy most individual requirements across a set of tasks while failing many entire tasks because at least one detail—such as a citation, deadline or conclusion—did not meet its rubric. Vals says each criterion counted as passed only when both of its two AI judges passed it.
The archived table lists the model’s cost at $0.15 per test. It places GLM 5.3 Flash, the base model, 33rd with a 6.67% task pass rate and a listed cost of $0.57 per test. Subtracting the two scores gives the 12.5-percentage-point improvement claimed by Z.ai. The figures are leaderboard results, not a guarantee of performance on a firm’s own legal work.
What the Ranking Measures
The ranking offers firms and developers a comparative measure of how models handled a defined set of legal-agent tasks. But the 19.17% figure is an end-to-end completion rate under strict grading, not a general measure of legal knowledge or the share of every answer that is correct. The 92.5% criteria score gives a different view: GLM Legal often met individual requirements while still missing at least one requirement somewhere in a task.
That distinction matters for legal work, where an otherwise useful draft can still be unsafe to rely on if a citation is unsupported or a deadline is wrong. Harvey’s benchmark says its rubrics check items including facts, conclusions, citations, deadlines, dollar amounts and formatting. Firms considering the model would need to assess how it performs on their own materials and retain review by qualified legal professionals.
The low listed test cost may draw attention, but price alone does not establish value. Procurement decisions also depend on data handling, integration, verification and review costs. The supplied material does not establish what client-data protections or contractual terms would apply in a particular deployment.
Products Worth Considering
Logitech MX Brio Ultra HD 4K Webcam, Graphite
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. Its public release includes 1,250 tasks across 24 legal practice areas, graded against more than 75,000 expert-written rubric criteria. A task pairs an instruction with client-matter materials and asks an agent to produce a work product for review, Harvey says.
For its public leaderboard, Vals AI uses a private holdout set that it says mirrors the public task mix and contains 120 tasks. Internet access is disabled during evaluation. Two AI judges—GPT 5.5 and Claude Sonnet 4.6—grade submissions; Vals describes the reported task score as the average of the judges’ pass rates. These procedures make the result specific to that evaluation setup.
Z.ai’s developer documentation describes GLM-5.3-FlashX-Legal as a model for legal workflows including research, contract review and drafting. The company says it used legal-domain mid-training and post-training, including reinforcement learning in tens of thousands of task environments modeled on legal workflows. Z.ai’s AutoClaw blog announced a beta available by application on October 13, 2026; Vals lists the release date as October 12. The documentation page itself carries no date.
“An all-criteria pass rate of 19.2%”
— Z.ai, in its developer documentation
Limits of the Published Evidence
The supplied material does not identify the full details of the GLM Legal submission, the exact run date or whether the leaderboard entry has since changed. The ranking cited here is from Vals’ archived version updated October 18, 2026; a later Vals page is also noted as updated October 30, but its relevant results are not provided in the source material.
Z.ai describes the result as state-of-the-art among open-source models, but the documentation does not name the comparison set. The supplied material also says it found no published weights for the legal model, leaving the meaning of “open-source” in that comparison unclear. The score supports the stated ranking in the archived table; it does not, by itself, verify the broader claim.
A source author’s separate four-task spot check reported citations to rulings that the author could not find for the Z.ai model. The author also reported that two Claude models cited a real ruling but described its holding incorrectly. This was a small, non-benchmark check, not evidence of error rates across the leaderboard. The available material does not specify what client-data terms or legal disclaimers apply across Z.ai’s services.
What Firms Should Verify
The next useful development would be a current, directly comparable Vals leaderboard entry with its evaluation version and model details clearly identified. Readers should also look for published weights or a clear account of the open-source comparison set before treating Z.ai’s state-of-the-art claim as independently established.
For organizations testing the model, the benchmark points to concrete checks: verify every citation and material conclusion, test the system on representative documents, and establish who can access client data and under what terms. Z.ai recommends connecting the model to external services for laws, cases and company data, making retrieval quality and source checking relevant parts of any deployment assessment. The source material does not provide a date for a broader release beyond the application-based beta announcement.
Key Questions
What does GLM Legal’s 19.17% score mean?
It means the model passed every rubric criterion on 19.17% of evaluated tasks in the archived Vals leaderboard. Under Harvey’s all-pass rule, one missed criterion makes the whole task a failure.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The measures use different units. The 92.5% figure counts individual criteria, while the task score requires every criterion within a task to pass. A task with one failed citation or deadline does not count as complete.
Did GLM Legal improve on Z.ai’s base model?
In the archived table, GLM Legal scored 19.17% and GLM 5.3 Flash scored 6.67%. The difference is 12.50 percentage points, matching Z.ai’s stated improvement for those listed results.
Does the ranking prove GLM Legal is safe to use for legal work?
No. The benchmark tests performance on a specific set of tasks and does not establish reliability on every legal matter. Human review and citation verification remain necessary, and the source material does not specify the data terms for a particular deployment.
Is Z.ai’s open-source leadership claim verified?
The archived leaderboard supports the model’s seventh-place ranking among 81 entries, but the supplied material does not show the comparison set behind Z.ai’s claim of state-of-the-art performance among open-source models. It also reports no published weights for the legal model.
Source: ThorstenMeyerAI.com
