Artificial Analysis just took a swing at one of AI’s most persistent headaches: benchmarks that tell you almost nothing about how a model will perform on your actual workload. Their new platform, Optima, lets teams build custom evaluations around their own data, workflows, or even just a written description of what they’re trying to accomplish. The output isn’t a leaderboard ranking — it’s a direct comparison of models on quality, cost per task, and time per task.
The platform accepts evaluation datasets from local files or Hugging Face, pulls in AI agent traces from tools like Arize, Braintrust, and Langfuse, and can even install a skill that mines a developer’s coding environment for context. If you don’t have structured data lying around, Optima will generate test inputs and evaluation criteria from sample inputs and outputs you provide. Two scoring methods are on offer: rubric-based evaluation against objective criteria, or a pairwise comparison approach where users judge a sample of response pairs and Optima extrapolates the full ranking — the same method behind Artificial Analysis’s own GDPval-AA and AA-Briefcase benchmarks.
What sets this apart is treating cost and speed as first-class metrics rather than afterthoughts. For agentic workflows, raw token pricing is nearly useless. A model that’s half the price per token can burn through five times as many tokens retrying failed tasks. Optima’s cost-per-completed-task lens surfaces that dynamic directly. Early testers used the platform to find finance and accounting agent models that cut costs by a factor of ten without meaningful quality loss, and to test which models matched lawyer writing styles or performed best on proprietary image datasets. Pricing is refreshingly transparent: no markup on model token costs, $0.125 per criterion for rubric evaluations, and $0.375 per comparison for pairwise scoring.
The timing isn’t accidental. Epoch AI’s research showed that benchmark results swing dramatically based on implementation details that papers rarely disclose — prompt wording, temperature settings, even the agent scaffolding. A broader study of 445 benchmark papers found nearly all had methodological weaknesses, and only about 10% used complete real-world tasks. That’s a damning indictment of the status quo. Optima doesn’t fix bad benchmark design — if your evaluation criteria are sloppy, the results will be too — but it removes the excuse that off-the-shelf benchmarks just don’t fit. The question now is whether teams that have never built rigorous evaluations will suddenly start doing so just because the tooling exists.
