Methodology
What the agents had to build, how every run was measured, how the videos were judged, and where the numbers need care.
The task in brief
Every agent gets the same written spec (about 450 lines) and the same short brief. It has to write a Three.js program that procedurally builds a stylised but recognisable Prague: the Castle with St. Vitus Cathedral, Malá Strana, the Vltava and Charles Bridge with its statues.
A camera drone sits on the ground in the Third Courtyard of Prague Castle, takes off and follows a 115-point GPS route, about 1.6 km long: down the Bull Staircase, through the castle gardens with a hairpin turn, down the New Castle Steps, around St. Nicholas Church, along Mostecká, through the Malá Strana bridge tower, across Charles Bridge between the statues and through the Old Town Bridge Tower. The film ends with a climbing shot that looks back over the bridge to the Castle, in golden-hour light.
Everything visible must come from the agent’s own code at runtime: no downloaded models, photos, map tiles or AI-generated images. The spec asks for at least 1500 animated pedestrians, a tram, boats and birds, a collision-free camera and a deterministic renderer.
The flight is rendered headless in Chrome, frame by frame, on the GPU when one is available and in software otherwise, then encoded into an MP4. The agent also delivers a contact sheet, a top-down route map and a REPORT.md with its decisions and known issues.
Nobody intervenes during a run. The brief says the agent is done only when the MP4 exists and passes its own checks. The full spec is not published here.
Where a preview video has a soundtrack, that audio is synthesised by the agent’s own code; it is not a downloaded track.
Hard requirements H1–H6
A script checks every delivered MP4 independently of the agent. A failure in any of these fails the run.
| H1 | MP4 container, H.264 High profile, yuv420p, moov atom at the front (faststart) |
|---|---|
| H2 | Exactly 1080 × 1920 (portrait 9:16) with square pixels |
| H3 | 30 fps, constant frame rate |
| H4 | Duration between 55 and 60 s; the target is 58.0 s (1740 frames) |
| H5 | Peak bitrate at most 20 Mbps over any 1-second window, file under 200 MB |
| H6 | AAC-LC audio track, 48 kHz stereo (it may be silent) |
The spec has six more requirements (H7–H12: procedural assets only, route fidelity, determinism, the take-off, no camera clipping, visible pedestrians). They cannot be checked reliably by a script, so they show up in the judges’ scores instead.
Harnesses and effort
A harness is the program the model runs in: it gives the model a shell, file tools and a loop. Each model runs in the harness its vendor ships, started from the same orchestrating session with the same brief.
Effort policy: Claude and Codex models run at high effort, everyone else at the middle setting of their provider (medium, or high where only low, high and max exist). Gemini 4 Hydrogen was run at xhigh on request. The effort is part of every run’s name.
| Harness | Models | Billing |
|---|---|---|
| Claude Code | Claude Fable, Opus, Sonnet, Haiku | Anthropic API list prices |
| Codex | GPT-6 and GPT-6.1 models | OpenAI API list prices |
| Antigravity | Gemini 4 Argon and Hydrogen | Google API list prices |
| Cursor | Grok 4.7 only | no local usage data |
| OpenCode | free OpenCode Zen models, Kimi K3.1 Pro (OpenCode Go), Qwen3.8 Max (OpenRouter) | free, subscription or billed |
Hardware
Every run uses the same small desktop: an Intel Core i3-10100T (4 cores, 8 threads, 35 W) with its integrated Intel UHD Graphics 630, driven through Mesa and ANGLE in headless Chrome. No discrete GPU.
The renderer matters more than the CPU. The same scene took about 2.9 s per frame in software (SwiftShader) and about 90 ms on the integrated GPU: 62 minutes versus 1.9 minutes for all 1740 frames. That is why the spec tells agents to try the GPU first, and why runs are only comparable on the same renderer.
What we measure
Wall time
From the first to the last timestamp of the agent’s session (Claude Code transcript, Codex rollout, Antigravity or OpenCode database). It includes rendering and encoding, because they are part of the task.
Tokens
Summed over every model response: uncached input, cache reads, cache writes and output. Output includes reasoning; OpenCode reports reasoning separately, so the two are added together here. Cursor stores no usage locally, so there are no token numbers for Grok.
Cost: list price versus billed
Tokens multiplied by the provider’s public API list price, not by any subscription, so that runs stay comparable. Exceptions are labelled: the OpenRouter run shows the amount actually billed (credit before and after), with the harness’s own estimate next to it; the OpenCode Go run was paid by a subscription and is shown at list price; free OpenCode Zen models cost $0; Gemini 4 Argon was priced at its introductory rate.
Generation speed (tok/s) and model-time share
For every model call, generation time runs from sending the request to receiving the last token. Tool execution, waiting for renders and harness overhead between calls are left out; time to first token stays in, so this is end-to-end model speed, not the pure decode rate. tok/s = generated tokens, including reasoning, divided by the summed generation time. Model-time share = summed generation time divided by wall time. Where timing sources disagree slightly, the share can come out above 100 %; charts cap it.
Tools
Tool calls by name, as each harness logs them (Bash, exec, shell and so on). Skills, subagents, web access and MCP servers were available in some harnesses; no agent used them.
Derived metrics
Cost per score point and points per dollar, output tokens per score point, cache-hit ratio (cache reads divided by all input tokens), judge spread per video, and score drift and rank stability (Spearman ρ) between panels. The Pareto frontier marks runs that no other run beats on both cost (or time) and score.
Judging
H1–H6 are a pass/fail gate, not points. Quality is scored by a blind panel of three judges from three vendors: Claude Opus 5.5 (Anthropic), GPT-6 Astra (OpenAI) and Gemini 4 Argon (Google).
From each final MP4 we extract 16 evenly spaced frames ourselves (every 3.625 s, from 0 to 54.4 s) into one 4 × 4 sheet. The sheets are anonymised with shuffled letters; the key stays with the orchestrator and is written down after judging. All three judges score every sheet with the same rubric, and the score is the median of the three totals, which keeps one outlier judge from deciding.
Whenever a new model joins, all videos are judged again with fresh letters, rather than scoring the newcomer alone. Three panels have been held so far. The earliest scores came from the orchestrator alone and gave 30 nearly free points for passing the technical checks; they are kept in the data but no longer used.
| Category | Points | What earns them |
|---|---|---|
| Landmarks | 25 | St. Vitus, St. Nicholas, Charles Bridge with statues and the bridge towers are recognisable as those buildings, not generic boxes. |
| City look | 25 | A believable historic city: varied façades, roofs, materials, lighting, shadows and water. |
| Camera & drone | 30 | The drone is visible in the chase, the take-off is shown, framing is pleasant. Heavy penalty for frames with the camera inside or against geometry. |
| Life | 10 | Plausible crowds, vehicles, boats and birds. |
| Overall | 10 | Would a viewer enjoy watching it as a short video? |
Caveats
- One run per model. There is no variance estimate yet, so a lucky or unlucky run decides a model’s place.
- The judges see 16 stills, not motion. Stutter, speed and smoothness are not judged.
- The Google judge is the least stable. In the latest panel it gave Claude Haiku 5.5 a 91 while the other two gave 53 and 43. The median absorbs single outliers like this.
- Mid-table scores drift between panels by about ±6 points, and Grok 4.7 moved 14 points between the last two. Treat small differences as a tie.
- Cursor records no token usage, so Grok 4.7 has no tokens, cost or generation speed.
- Each model runs in a different harness, so every comparison mixes the model with its harness.
- Free tiers and plan limits ended several runs. Those runs say little about the models themselves.
- Claude Opus 5.5 run 2 used an older spec and a software render. It is shown for reference and left out of comparisons.
- The judges are models from the same vendors that compete. They do not know which video is whose, but shared taste cannot be ruled out.
Metrics roadmap
Metrics we want but do not compute yet. None of them is shown anywhere on this site until it is measured.
- Reliability
pass^k and repeated runs
Run each model k times and report the mean score ± standard deviation, the share of runs that pass, and the DNF rate. One run per model cannot tell skill from luck.
Needs: several runs per model on the same spec
- Reliability
Effective cost to a passing video
What it costs on average to get one passing video, counting the failed attempts too (the “cost to complete” idea from KiloBench).
Needs: repeated runs, including the failures
- Process
Time to first MP4 and render iterations
How soon an agent has its first complete video, and how many draft and final renders or self-corrections it needs before it stops.
Needs: parsing render and encode commands out of the transcripts
- Process
Tool error rate, TTFT and pure decode speed
The share of tool calls that fail, the time to first token, and decode speed without prefill, wherever the harness logs them.
Needs: per-call timing and tool results from each harness
- Process
Code produced
Lines of code, number of files and npm dependencies in the delivered project.
Needs: a counting pass over each run’s workspace
- Video
Camera smoothness and route deviation
Jerk (the rate of change of acceleration) of the camera path, and how far the flight strays from the reference route, measured from the scene rather than taken from the agent’s report.
Needs: camera and drone positions exported per frame
- Video
Collision frames
Frames where the camera is inside or pressed against geometry, detected automatically instead of by the judges’ eyes.
Needs: a depth or raycast pass over every frame
- Video
Scene complexity
Pedestrians, colliders, triangles and draw calls from each run’s render metadata. Only some runs write this file today.
Needs: a standard render_meta.json in every run
- Video
Duration and bitrate compliance
Exact duration, average and peak bitrate, file size and audio loudness as numbers rather than a pass/fail line.
Needs: a structured version of the independent check
- Honesty
REPORT honesty
Compare what the agent claims in REPORT.md (no clipping, all checks passed, route deviation) with the independent checks and the judges’ observations.
Needs: claim extraction and a matching rule set
- Coverage
Spec soft-requirement coverage
Which of the spec’s optional and soft items (tram, boats, swans, soundtrack, title card, look back at the Castle) each video actually contains.
Needs: a checklist and a per-video review
- Cost
Energy and compute estimate
Local energy used by rendering and encoding, plus a rough estimate of the model-side compute behind the tokens.
Needs: power metering on the machine and published per-token figures
This site uses the fonts Inter and Instrument Serif, licensed under the SIL Open Font License 1.1 (see NOTICE).