OpenRouter 推出 Ori Eval：为你的项目找到最合适的模型

Jacky Liang ·8/3/2026 · Updated 8/19/2026

As more and more apps add AI functionality, the choice of which model to use for what you’re building remains just as hard, if not harder, with over 500 models you can choose from.

In practice, that choice is often made without a systematic method: a recommendation from social media, a benchmark leaderboard, or a general sense that one model is currently the strongest.

These resources are genuinely useful, but each has the same limit. A benchmark measures a fixed task set, and a recommendation reflects someone else’s application. Neither can tell you how a model behaves in your app, on your harness, on your data, and on your prompts.

Meanwhile new models ship every week, and the cost of re-evaluating them by hand is high enough that many teams delay the decision, or keep a model that no longer fits.

At OpenRouter, we know a thing or two about models.

What we’ve learned is that there is no definitive best model - only the best model for what you’re building.

Ori Eval helps you find that single model, and prove it to you.

Ori Eval will walk you through how to choose the best model for what you’re building like a friendly and experienced engineering friend - no eval experience needed.

Tl;dr

Ask a question, get an answer (with proof)

To use Ori Eval, you can simply tell your favorite coding agent:

Your agent hands the request to Ori Eval. Ori Eval explores your codebase, and it comes back with questions. Before it runs any evaluations, it works with you to help figure out what’s important to you — is it cost, performance, latency, speed, tool call accuracy, etc. It then chooses the 5 latest models that fit your requirement, and confirms with you.

Once all these necessary requirements have been gathered, Ori Eval writes a review.eval.ts file, runs your agent against the candidate models in parallel, and returns a table:

The recommendation comes with the reason: let’s say, the top bug catch rate inside your model cost criteria, and the value pick if bug review volume grows.

Don’t worry if you’ve never written an eval before

Writing good evals is not easy, that’s why we handle the annoying part for you.

Ori Eval scans your codebase for every place a model runs, and shows you what it found: the use case surface, the exact file, and the model you use there right now. It then asks where you want the eval to cover, and what matters most to you: accuracy, speed, cost, or something else?

Once Ori Eval is done interviewing you, it writes the eval file from your answers, and runs it.

It’s that easy.

Consistent scores, run after run

Because Ori Eval is an agent, it can pin the harness, the model, and the effort for the duration of a run. It also comes pre-tuned: we’ve already chosen the harness and the model that work best for eval work, so you don’t have to.

An eval checks three things

An eval file is a *.eval.ts file that runs with bun test. It checks the tools the agent called, the tools it avoided, and the quality of the answer:

For open-ended answers, an LLM-as-a-judge grades the output. Ori Eval helps you set the grading criteria and the minimum score, so even tricky open-ended questions can be evaluated.

Every bug becomes something you can test against

Tell Ori Eval about a bug in plain words: let’s say, a support agent is issuing refunds without even checking the order first — a pretty big problem.

Ori Eval writes an eval that asserts lookup_order gets called. The eval fails proving that the bug exists. You then fix the agent, and the eval passes. The assertion stays in your test suite, so you’ll always be able to catch it.

Block regressions, and re-run when the field moves

Add ori eval to a GitHub Actions workflow. It exits like bun test, so failed evals also fails the build, and a regression never reaches prod.

Since what Ori Eval writes is just code, you can also easily schedule them to run every so often. One of our early beta testers now runs model comparisons monthly. When a new model drops and it performs better than the existing model in your code base, a PR gets opened, and all you have to do is merge it and experience all the benefits without you even thinking about it.

The skill installs Ori (our pre-tuned coding agent), asks you to sign in, interviews you, and runs the eval. If you use the OpenRouter MCP server, run /spawn-ori-eval instead, and skip the URL.

To install Ori by hand:

Then run ori login. Running evals also needs Bun.