Leaderboard

Results across five agent systems and four foundation models. See the Overview for task structure and evaluation protocol.

Base model

Partial Judge Final

1

DuMateDuMate

Partial

90.3%

Judge

77.7%

Final

81.5%

2

Hermes

Partial

90.0%

Judge

76.3%

Final

80.4%

3

Claude Code

Partial

86.1%

Judge

74.9%

Final

78.3%

4

OpenClaw

Partial

82.8%

Judge

75.9%

Final

78.0%

5

OpenCode

Partial

77.0%

Judge

65.7%

Final

69.1%

Results on 200 DuMateBench tasks. Final is computed as 0.3 × Partial + 0.7 × Judge. The best result in each column is shown in bold.

Metric

The leaderboard reports three task-level metrics, and macro-averages each metric across tasks.

Metric	Symbol	Computation	What it captures
Partial pass rate	P	Mean of satisfied atomic checks	Deterministic requirement coverage
Judge score	J	Macro-average over supported target artifacts	Artifact correctness, completeness, and quality
Final score	F	0.3P + 0.7J	Weighted benchmark result

Partial pass rate (P)

Each task is associated with a set of atomic checks generated from the task requirements and reviewed by human annotators.

Check dimension	Examples of what is checked
Output existence and location	Required artifacts are present at the expected paths.
Format validity	File types, schemas, and parsable structures are valid.
Required or forbidden content	Required facts, fields, and constraints are satisfied.
Document structure	Headings, sections, layout, and organization meet the task requirements.
Spreadsheet values and formulas	Cell values, formulas, ranges, and calculated outputs are correct.
Protected-file integrity	Files that should remain unchanged are preserved.

The task-level Partial pass rate is the proportion of atomic checks that are satisfied:

$$P_t=\frac{1}{n_t}\sum _{i=1}^{n_t}{c}_{t,i},c_{t,i}\in \{0,1\}.$$

A task is considered complete only when all of its deterministic checks pass. Partial pass rate therefore gives credit for progress while keeping full completion strict.

Judge score (J)

An artifact-specific LLM judge evaluates semantic, organizational, and perceptual properties that fixed checks cannot capture.

Scoring element	Protocol
Rubric structure	Each reviewed rubric contains 3–16 atomic criteria with normalized weights.
Anchored levels	Each criterion is scored on an anchored 0–4 scale.
Evidence	The judge records a score and supporting evidence for every criterion.
Cannot assess	A criterion marked as cannot_assess contributes no positive score.
Missing artifact	A missing expected artifact receives a score of zero.
No artifact-specific judge	The deterministic score is used as the task score when no judge applies by design.

The task-level Judge score is the macro-average over supported target artifacts. A missing expected artifact receives a score of zero. When no artifact-specific judge applies by design, the deterministic score is used as the task score.

Final score (F)

The Final score gives 30% weight to deterministic requirement coverage and 70% weight to artifact quality:

Component	Weight	Meaning
Partial pass rate P	30%	Deterministic requirement coverage
Judge score J	70%	Artifact-specific quality

$$F_t=0.3P_t+0.7J_t.$$

The benchmark reports P, J, and F separately, then macro-averages each metric across the 200 tasks.

The macro-average is defined as:

$$\bar{M}=\frac{1}{200}\sum _{t=1}^{200}{M}_t,M\in \{P,J,F\}.$$