Dispositions • Big Tech & Models • Vendor Noise
Evaluating Superintelligence Models
Source: Arena Leaderboard.
The source is a live leaderboard whose rankings change over time, rather than a dated research report.
The analysis date above is the date of James's commentary.
James's take
I use model leaderboards as one input when evaluating Si tools. A ranking does not show how a model will perform in your workflow. I compare candidates on representative tasks, response time, reliable tool use, structured output, and the cost of running the process. I also test failure cases and the review a person must provide. A model change should solve a documented problem or produce an improvement that the team can measure.
Questions for your team
- Which tasks from your actual workflow will you use to compare candidate models?
- What latency and API cost limits must the model meet for this workflow to be useful?
- How will you check reliable tool calls and structured JSON output, including failure cases?
- What measurable improvement would justify changing models before the supporting data pipelines are ready?