Research benchmark · DuMate

DuMateBench

AI Agent Benchmark for Real-World Work

Evaluate agents on realistic tasks and environments.

Explore the benchmark →View leaderboard

Static, open documentation Version TBD

DuMateBench evaluates whether autonomous agents can coordinate tools, produce artifacts, and recover from failures across realistic end-to-end workflows.

OverviewScope, task design, and evaluation protocol →LeaderboardResults across five agents and four base models →PaperRead the benchmark paper and citation details →DatasetOpen the released task data →

Why DuMateBench

DuMateBench is built around three core contributions.

The benchmark is designed around the gap between clean, isolated demos and the messy, multi-step work agents must complete in practice.

01

Real-session compositional workflows

200 executable tasks are reconstructed from anonymized, privacy-screened multi-turn sessions, preserving interaction history and workspace state across 8 broad scenarios and 17 fine-grained task types.

02

Reproducible complexity

Isolated Docker environments model insufficient tools or resources, unstable network and tool behavior, and noisy workspaces with distracting data.

03

Comprehensive, diagnostic evaluation

Five agent frameworks and four base models form 20 configurations, measuring end-to-end performance, workspace-noise robustness, efficiency, and failure modes.

Environment & capability coverage

Designed for complex computer work.

Tasks combine multiple capabilities and run under controlled environmental constraints.

200Executable tasks

8Broad scenarios

17Fine-grained task types

20Agent-model configurations

Environmental complexity

InsufficientMissing tools or dependencies, constrained resources, and limited preinstalled capabilities.

UnstableTransient network, API, OCR, or tool failures that require recovery.

NoisyHistorical, duplicate, irrelevant, outdated, or conflicting files and data.

These conditions are instantiated in isolated Docker containers so reliability can be evaluated under controlled and repeatable constraints.

Capability coverage

Text and codeText generation, coding, and software development · Text 160 · Code 88

Web and informationWeb retrieval, information extraction, and network/API recovery · Web 86

Documents and spreadsheetsDOCX, PDF, PPTX, and Excel workflows · DOCX reading 27 · Excel editing 18

Files and mediaFile organization, image, video, and audio generation/editing · Files 34 · Multimedia 46