Artificial Analysis releases Decision Index v2 to curb benchmark gaming

Artificial Analysis launched a second version of its Decision Index, a benchmark meant to measure how useful AI models are in real decision-making tasks rather than how well they game test questions. Nearly half the index now relies on private datasets, up sharply from the first version, making memorization harder. The update adds a Decision Score that rewards models both for being right and for being well calibrated about how confident they are. Two new tests were added: one checks whether a model can correctly answer all 15 questions tied to a private document like a resume or invoice, and another checks whether a model gives consistent answers when the same question is reworded. Specialized decision-focused models performed competitively, with a few scoring in the low 70s. General-purpose large language models scored highest overall, but at a steep cost: one top model priced far above specialized alternatives and ran many times slower per task. The release suggests a widening gap between raw accuracy and practical cost-efficiency when choosing models for real business use.

Why it matters

Shows evaluators are shifting focus from raw accuracy to cost and reliability tradeoffs that matter for real deployments.

Source: Artificial Analysis