Measure whether an AI developer tool actually saves tokens
Run a controlled Baseline-versus-Candidate test on a public repository. Compare tokens, cost, task time, quality, setup, re-index overhead, and what the agents did.
Run the same prompt suite twice
The benchmark skill pins one public repository revision, runs identical prompts with Baseline and Candidate, verifies both results, and includes setup plus required re-index work.
Install benchmark-repo in your coding agent
Use the open-source skill from the trAIce SDK repository. For a Graphify candidate, the skill also checks the Graphify CLI and agent integration before recording setup.
In Codex, ask:
Install the benchmark-repo skill from https://github.com/runtraice/traice-sdk/tree/main/skills/benchmark-repoOpen the repository and send this prompt
Use $benchmark-repo to benchmark this repository with a baseline and Graphify.You can replace Graphify with another candidate tool. The prompts recorded in the manifest are the identical task suite sent to both variants.
Inspect the comparison before upload
npx @traice/collector@latest benchmark compareThe report compares tokens, cost, task time, quality, one-time setup, refreshes, and privacy-safe activity counts. Raw observed commands, output, and paths stay on the device.
Authenticate, upload, then publish here
npx @traice/collector@latest auth login
npx @traice/collector@latest benchmark uploadUpload always creates a private workspace draft. An owner or admin reviews it before publication.
Published benchmark history
0 public, reproducible comparisons. Open any report to inspect its exact prompts and evidence.
No published benchmarks yet
Published reports will appear here after an owner or admin approves them.