The Task Bench: Evaluating Computer-Use Agents Beyond the Demo

Computer-use agents look impressive when a task has one clean path. Real interfaces contain stale sessions, pop-ups, ambiguous labels, partial saves, permissions, and data that moves while the agent is working.

Evaluation should reproduce that reality.

Test Outcomes, Not Clicks

Define the desired end state and unacceptable side effects. For an expense workflow, success may mean the correct receipt is attached, fields match the source, policy exceptions are flagged, and nothing is submitted without approval. A sequence of plausible clicks is not enough.

Build cases across four dimensions:

• Variation: alternate layouts, languages, window sizes, and data formats.
• Interruption: timeouts, sign-in prompts, changed records, and missing dependencies.
• Ambiguity: similar names, incomplete instructions, and conflicting evidence.
• Consequence: drafts, reversible edits, external messages, and irreversible submission.

Preserve Evidence for Replay

Record the observation available at each step, the action selected, tool response, and resulting state. Mask sensitive data, but keep enough structure to replay a failure. Without step-level evidence, teams cannot distinguish perception errors from bad planning or unsafe execution.

Score Recovery Separately

An agent that recognizes uncertainty and asks for help may be more valuable than one that completes more tasks by guessing. Track safe abandonment, successful resume, duplicated actions, and quality of escalation alongside completion rate.

The best test suite grows from production incidents and difficult human examples. A task bench is not a one-time leaderboard; it is the regression system that keeps an evolving agent dependable.