Same Task Bench — Round 1 evidence

Claude Code vs Codex CLI · same repo, same 4 tasks, one rubric locked before any agent ran · 8GB Intel laptop

This page exists so the numbers in the video can be checked without taking anyone's word for them. Every file linked here is the raw output of the run, not a summary written afterwards.

Results

Ranking rule, fixed before the runs: correctness first (hidden tests, then rescue accuracy), speed as tiebreaker. Codex is 2.3× faster; Claude is the only perfect score.

Check the wall clock yourself

The harness wrote a Unix timestamp at the start and end of each run. Nothing else determines the times shown on screen:

$ cat runs/claude-code-r2/time_start runs/claude-code-r2/time_end
1786212883
1786213556

$ echo $(( 1786213556 - 1786212883 ))
673        # = 11:13

Same for Codex: 1786212816 - 1786212524 = 292 = 4:52. Machine-readable summary: runs/WALL-CLOCK.json

Check the scores yourself

• harness/results/claude-code-r2.json — official Claude Code record (peak RSS 13,725,696 B = 13MB)
• harness/results/codex.json — Codex CLI (peak RSS 8,744,960 B = 8MB, task 3 = 42/45)
• harness/results/claude-code.json — Run 1, reference-only (see caveat below)
• reference.json / selftest.json — scorer calibration
• harness/ — the scoring scripts themselves

The identical input every agent received

repo-template/ — the OrderDesk repo exactly as handed to both agents: orderdesk/app.py, tests/test_basic.py, data/orders.json, 15 scanned invoices with their OCR text, and docs/POL-114.md (the policy the reconciliation bug violates). data/big_orders.json is not included here: it is a 200,000-row file generated at setup time by harness/make_big_orders.py, which is published so you can regenerate it byte-for-byte. Tasks: TASKS.md.

Full agent transcripts

• claude-code-r2 transcript
• codex transcript
• claude-code Run 1 transcript

Absolute filesystem paths have been replaced with placeholders such as <bench>. Nothing else in these logs was edited. (The harness records the same output as both transcript.log and stdout.log per run; only transcript.log is linked here — see the run directory for both.)

Limitations, stated up front

• One run per agent, not five. Five full runs on an 8GB laptop take most of a day. Single runs cannot separate skill from variance — treat this as one observation, not a ranking.
• Run 1 of Claude Code is reference-only. Our --permission-mode acceptEdits setting denied it Bash, so it never ran pytest and hand-traced everything instead. That is not the same conditions Codex got, so the official record is the re-run (claude-code-r2) with execution enabled. Run 1 is published anyway because deleting an inconvenient run is how benchmarks become marketing.
• Plans differ (Claude MAX vs ChatGPT Plus). Both were already held; nothing was purchased for this test. Cost per task is not compared.
• Rounds 2+ are not here yet. Antigravity CLI, GitHub Copilot CLI and Aider are scheduled. This board is updated, not replaced.

Codex's three misses, in full

All three are money fields; all 15 vendor names were correct. Each is an OCR misread that arithmetic cross-checking would have caught: