AI Benchmarker

Build your own AI benchmarks. Add prompts, run them across the models you care about, rate the responses, and publish a shareable leaderboard.

Free trial included No credit card needed Your prompts stay private

Visit versionsai.com

AI Benchmarker in Raq.com

Save a list of prompts you actually care about, pick a baseline of models you trust, and add new candidate models to compare against them. Each model gets up to ten attempts per prompt, you rate every response with stars, and Raq averages the scores into a public leaderboard table at your own URL. No need to trust someone else's benchmark. Build the one that matches your work.

Use AI Benchmarker through your AI agent

Works in Claude, ChatGPT Work, Grok Bot and any AI app with connectors. Connect yours

You

Compare these AI models using our customer questions.

Your AI agent
Working in Raq.com AI Benchmarker
  • Use your customer questions. Test the work your team needs done.
  • Run the same set. Give each model the same inputs.
  • Rate the responses. Compare answer quality alongside speed and cost.

I’ll run the same questions through AI Benchmarker and put the responses together for your team to rate.

How it works

How AI Benchmarker moves work forward

Write your prompts

The questions and tasks you actually use AI for.

Add each prompt exactly as you'd send it, up to 200 in a benchmark. Paste them in one at a time, import a batch as JSONL, JSON or CSV, or start from a template.

Choose how each one is marked

Your own stars, an expected answer or an AI judge.

Rate the answers yourself after the run, give the answer you expect and check for it, or write the criteria and let a judge model score each answer. You can change any score the judge gives.

Pick the models and versions

Up to 25 models in one run.

Mix companies, sizes and reasoning levels, and keep the models you trust as a baseline that runs every time. Each prompt can be repeated up to ten times to see how steady the answers are.

Read the results

Score, cost and speed for every model.

See which model scored best, which was cheapest and which was quickest, then open any answer to read it. Publish the table as a leaderboard or share the run with a link.

Capabilities

What sets AI Benchmarker apart

Know the cost first

Every run shows the AI usage it will take before any model is called, and a cap stops it if the cost runs past that.

Re-run on a schedule

Run a benchmark every day, week or month, or let it start on its own with just the new models when one that suits it appears.

Publish a leaderboard

Put a run on a public page with your name on it, or share one run with a link, a password and an end date.

Bring your Promptfoo tests

Import a Promptfoo config as a benchmark, export one back, or start runs from your own scripts with an API key.

AI Benchmarker is included in every Raq.com plan. From $39 per user a month, with 75+ modules. AI work uses your plan's usage allowance.

Free trial included No credit card needed Your prompts stay private