Benchmark suite
The Benchmark pane runs a registry of categorized tasks — reasoning, code, refactoring, and more — against the models you pick, scores the results, and lets you export them. It’s the “measure it again next month” counterpart to the Model Council.
Running a benchmark
Section titled “Running a benchmark”- Open it with open benchmark suite.
- Pick a category (or a whole suite) from the registry.
- Pick the model(s) — installed agent CLIs work out of the box; OpenRouter models join once a key is set.
- Run and watch per-task results fill in.
Comparing two models
Section titled “Comparing two models”The two-model compare view lines both up on the same tasks so differences are visible per item, not just in the total.
Exporting
Section titled “Exporting”Results export to a file for records or sharing — useful for before/after checks when a CLI updates its default model.
Media artifacts: music and motion
Section titled “Media artifacts: music and motion”The artifact categories go beyond SVG and HTML: Artifacts · Music (Web Audio) asks models to ship a self-contained page that synthesizes a melody or drum loop in code (no audio files), and Artifacts · Motion (canvas/CSS) asks for animated scenes — a bouncing canvas ball, a CSS-keyframe storyboard. Answers render live in the sandboxed preview (click the play button in the preview to hear music — browsers require a click before audio), and the auto-check chip reads the capability at a glance: Web Audio music, Animated, or Audio + motion. No external generation APIs are involved — the model’s own code is the instrument.
Good to know
Section titled “Good to know”- Benchmarks consume real usage/quota on the agents they run — a large suite on a metered CLI counts against it.
- The registry is categorized so you can benchmark only what you care about (e.g. just code tasks).