Reward judges for computer-use agent trajectories, scored against human annotations, from SeekJudge. Every trajectory carries a human verdict on task success, and every step carries a human label.
A framework that issues any closed-source API call belongs here, even when the rest of the pipeline runs on open weights.
Every model behind the row has public weights.
Trajectory-level. The predicted success verdict is compared against the human verdict over all 278 tasks. The positive class is task success.
Step-level. Every annotated step is one sample, pooled across all trajectories. The positive class is an
error step, that is a step the human labeled bad. Error steps are a small minority, so step-level
accuracy stays high across all judges while F1 exposes how few of the errors are actually caught.
All numbers are percentages. A dash means the framework does not produce that level of judgment and no extraction procedure was applied. Rows are ordered by trajectory F1 by default; click any column header to re-sort.
Submissions are pull requests against this Space. Open the Community tab and start a new pull request, or
open results.json, edit it in the browser, and choose Open as a pull request. Append one entry to
entries and change nothing else. The page rebuilds itself once the pull request is merged.
{
"track": "open",
"release": "2026-09",
"framework": "MyJudge",
"model": "MyModel-7B",
"model_note": null,
"traj": {"acc": 75.0, "prec": 74.1, "recall": 78.2, "f1": 76.1},
"step": {"acc": 90.1, "prec": 46.0, "recall": 35.2, "f1": 39.9},
"step_note": null,
"source": "https://example.com/my-judge-logs"
}track is closed or open. A framework that issues any closed-source API call
is closed, even when the rest of the pipeline runs on open weights. All 8 metrics are percentages at one
decimal place, and a level the framework does not produce is null rather than omitted.
source holds every LLM API call behind the numbers together with the per-task
judge/<your-judge-name>/result.json outputs, on any host that needs no login. Each call carries its
task_id, model identifier, the full request (system prompt, messages, and which images were passed), the
raw response, token counts, and a timestamp. Any layout works provided every call maps unambiguously to its task. All
rows currently on this page point at the paper logs.
The pull request description states whether step labels are native or come from a step extraction procedure, and how the run was produced. All 278 tasks must be covered, with no re-annotation and no subset selection.