On Artificial Analysis

Artificial Analysis has become a default leaderboard for frontier AI. When a lab releases a model, it is one of the first places people look instead of taking the vendor's launch slides at face value. That reputation is deserved in several areas: the throughput testing is useful, the pricing tracker is excellent, and there is real value in running hundreds of models through the same harness.

The trouble starts when all of that gets compressed into one score that is supposed to represent "intelligence."

The Intelligence Index is not a natural measurement. It is a collection of choices about what should count as intelligence, what deserves more weight, what gets counted twice, which old benchmarks stay after saturation, and how much credit belongs to the model versus the software wrapped around it.

I pulled apart the results from 586 model evaluations to see what those choices actually do to the index.

The 60-Second Summary

Version 4.1.1 moved the index much further toward "agentic workloads." The official Agents category rose to 34%, while General reasoning fell to 18%. That is a defensible choice, but it has consequences.

The short version is this:

• GPQA is saturated, but it still gets 6%. Frontier models are bunched above 94% and separated by fractions of a point. It cannot reliably distinguish a 27B model from a 2.4T model.
• The "24% Coding" category contains no repo-level SWE benchmark. Nothing in the index tests whether a model can enter a real multi-file codebase, trace a dependency chain, and fix a bug that spans several modules.
• SciCode is closer to a science test written in Python. After controlling for general model capability, it correlates at r ≈ 0.71 with GPQA Diamond. It measures science knowledge more than software engineering.
• GDPval gives the scaffold a lot of influence. When a fifth of the index depends on producing clean PDFs and spreadsheets, the Stirrup harness and its visual inspection loop become a large part of what is being evaluated.
• Half the index uses the same basic loop. GDPval (20%), Terminal-Bench (16%), and τ³-Banking (14%) all rely on a recurring agent-execution pattern.
• The weights barely change the ranking. Replacing AA's hand-tuned formula with a flat 10% split gives a 0.992 rank correlation and the same top ten models.

1. GPQA Is Still Here. Why?

GPQA Diamond used to be a very useful benchmark. At the frontier, it has been above 90% long enough that it is running out of room.

It contains 198 PhD-level questions across biology, chemistry, and physics. The design was elegant: subject-matter experts could solve them, while non-experts generally could not. AA runs the Diamond subset five times per model to reduce variance. That made sense two years ago, when the benchmark had more headroom.

Today the frontier leaderboard is pressed against the ceiling:

• GPT-5.6 Sol: 94.1%
• Gemini 3.1 Pro Preview: 94.1%
• Claude Opus 5: 93.7%

The top ten models are separated by roughly two points. Comparing frontier models on GPQA right now is like timing Olympic sprinters with a stopwatch that only reads full seconds. When five models sit within 0.4 percentage points of each other, the differences you're seeing are stochastic sampling noise, not capability gaps.

I standardized every benchmark in the 586-model dataset to z-scores. GPQA has the lowest standard deviation, absolute range, and IQR (0.478) of anything in the index. It has the weakest discriminative power, yet it still gets 6%.

2. The "Coding" Category Has No Coding Benchmark

The only thing in the index that touches code synthesis is SciCode at 8%. AA will point at Terminal-Bench (16%) and call it coding. I don't buy that.

By my definition, a coding benchmark should require writing or patching code. Terminal-Bench 2.1 does not mainly test that. It checks whether an agent in a Linux container can configure a server, debug environment variables, parse logs, and satisfy CLI verifiers. That is systems administration and infrastructure work, not software engineering.

SciCode does not test searching a codebase, finding which module owns a behavior, tracing dependencies, using git, modifying APIs across files, or making sense of an unfamiliar architecture. It measures scientific knowledge and mathematical reasoning through code, with some algorithm design mixed in. It is useful, but it is not a substitute for an SWE benchmark.

"So coding is scored on a single benchmark that wasn't really testing coding to begin with?"

Yeah.

"But TB2.1 also has coding in it."

Sure, technically. The coding it contains is mostly the same isolated, single-file work that SciCode already covers, with no complex interoperating architecture. If anything, that makes SciCode's place in the index harder to justify. At least SciCode still tests some algorithm design.

A benchmark such as DeepSWE is much closer to what I mean by SWE: large repositories, patches, bug fixes, feature work, and changes that span multiple files. That is the missing capability here.

3. SciCode-Verified & The GPQA Collinearity Problem

There is another problem with SciCode: SciCode-Verified (arXiv:2608.04975). Its authors report 263 defects in the original benchmark affecting 91% of the main test problems. After correcting them, reported scores moved from the 45–60% range to 84–98%. It is still a preprint as I write this, but the result is too large to wave away.

Then there is the overlap with GPQA Diamond. Depending on the control used for general model strength, the adjusted correlation lands between 0.567 and 0.71. That suggests two things:

• 1. SciCode behaves substantially like a scientific-reasoning benchmark that happens to output Python, not a code-generation benchmark.
• 2. It is AA's only dedicated code-synthesis benchmark, yet after removing general model capability, its strongest relationship is with GPQA Diamond, a pure science-reasoning test (\(r = 0.567\)).

SciCode measures scientific and mathematical reasoning through the medium of code. It does not measure repo-level software engineering. Calling it an 8% "Coding" component double-counts the science reasoning that GPQA and HLE already capture.

4. PDF-Bench: GDPval-AA v2

GDPval is an OpenAI benchmark. 1,320 professional tasks spanning 44 occupations and 9 industries. AA uses the public 220-task Gold subset.

Each run spins up a full E2B virtual machine loaded with useful packages and wraps it in AA's Stirrup harness. The agent gets web browsing, image inspection, and other tools. Its output is usually a PDF, though some tasks produce a DOCX file or spreadsheet.

The idea is to evaluate real professional work involving reasoning, research, formatting, file handling, and tool use. On paper, that is a strong design.

"So what's the problem? Sounds like a solid benchmark."

The problem is the grading.

OpenAI's own GDPval paper found that a lot of GPT-5's failures were obvious formatting screwups. Their fix: change the prompt to tell the model to render and visually inspect its own output, avoid formatting problems, and check its work. They also improved the scaffold and used best-of-four sampling.

The changes eliminated a rendering artifact that had broken more than half of the generated PDFs and reduced severe PowerPoint formatting errors. Visual self-inspection rose from 15% to 97%, and the human-preference win rate improved by five percentage points, all without changing the model weights.

The benchmark contains a wide range of tasks and multi-step workflows, but judges see the final PDF or DOCX file rather than the underlying research or calculations. That is why I call it "PDF-Bench."

This setup strongly favors vision models. A model without vision is flying blind, while a vision model can inspect its output for clipping or misalignment. If visual inspection heuristics let a smaller model beat a frontier model, the benchmark is testing the harness at least as much as latent intelligence.

What GDPval-AA v2 actually measures:

model × Stirrup scaffold × sandbox/tool competence × professional reasoning × file/artifact ability × judge preferences

It's an interesting systems test. It does not deserve 20% of the entire index.

5. Hallucinations & The "I Cannot Answer" Exploit

AA's hallucination scoring is defined as:

Now watch this.

The following function:

def handle_api_request(api_input: dict) -> str:
return "I cannot answer this question"

scores a perfect 100% on AA-Omniscience non-hallucination.

Two lines of Python beat GPT-5.6 Terra (xhigh reasoning), which scores -3. On this metric, a stub function that refuses everything outperforms a frontier model.

Automated refusal scoring is hard, and ratio-based metrics are tempting. But a metric that can be gamed by a function that refuses every question needs a refusal penalty.

6. Knowledge Is (Basically) Worthless Here

By AA's own weighting, explicit knowledge accounts for 8% of the index through AA-Omniscience-Accuracy.

Other benchmarks draw on general knowledge, especially HLE. But AA classifies HLE under "Scientific Reasoning," not knowledge, so only 8% directly rewards factual recall.

This is a philosophical choice. It prioritizes autonomous tool execution over internalized understanding. You can disagree with it, but you should at least notice it.

Parametric knowledge still matters in practice. It speeds up reasoning. It guides search queries. It filters out hallucinated tool returns. It stops the model from burning 50 steps on an execution loop because it didn't know what a REST endpoint was. Giving closed-book factual recall only 8% is a deliberate editorial decision, not a neutral measurement.

7. Half the Index Is One Agent Loop, Wearing Three Hats

AA reports 34% of v4.1.1 as "Agents." But look at how the evaluations actually execute:

• GDPval-AA v2: 20% for long-horizon artifact creation in the Stirrup harness
• Terminal-Bench v2.1: 16% for a multi-turn Linux CLI agent in the Terminus harness
• τ³-Banking: 14% for a multi-turn conversational tool-calling agent

Together, those three benchmarks make up 50.0% of the Intelligence Index.

The domain tasks differ. Office deliverables versus bash commands versus banking database calls. But the operational loop is identical across all three:

Goal Decomposition → Environment Inspection → Tool Action → Feedback Parsing → Error Recovery → State Maintenance → Loop Until Done

Giving half the index to one operational pattern biases the composite toward outer-loop scaffolding competence. A model that handles the loop well but reasons poorly can outrank one that reasons well but stumbles on tool syntax. Readers can decide whether that belongs under the label "intelligence."

8. The Correlation Problem

GDPval-AA v2 and AA-Briefcase correlate at 0.97, which makes sense because they measure nearly the same thing. AA-Briefcase is excluded from the intelligence index, so that overlap does not affect the score.

GDPval-AA v2 and Terminal-Bench v2.1 have a raw correlation of 0.91. Unlike AA-Briefcase, both contribute to the index, so this overlap matters.

Adjusting for general model strength:

model_strength = mean(z(control_benchmark_scores))
GDPval_residual = GDPval - predicted_GDPval_from_model_strength
Terminal_residual = Terminal - predicted_Terminal_from_model_strength
adjusted_r = correlation(GDPval_residual, Terminal_residual)

The result is an adjusted correlation of 0.647, which is exceptionally high. For comparison, Terminal-Bench versus HLE drops from 0.899 raw to 0.333 adjusted. Those two measure more distinct capabilities.

Adjusted correlation (\(r\)) shows how much two benchmarks overlap once general capability is removed:

• GDPval-AA / AA-Briefcase: 0.914
• GDPval-AA / Terminal-Bench: 0.647 (highest of all 45 pairs tested, and together they're 36% of the index)
• GPQA Diamond / AA-LCR: 0.624
• Humanity's Last Exam / CritPt: 0.588
• SciCode / GPQA Diamond: 0.567 (32% shared residual variance)

GPQA Diamond and AA-LCR show the second-highest adjusted correlation at 0.624. Because GPQA is compressed near 94%, most of the remaining variation it captures just tracks AA-LCR.

My personal favorite: HLE has the highest adjusted average correlation (0.257) across the whole suite. Even after removing general model strength, HLE captures more shared capability variation than any other benchmark. It's doing more work per unit of weight than anything else in the index.

"Everything's correlated, what's the issue?"

Some pairs are largely independent. τ³-Banking and AA-LCR have an adjusted correlation of −0.176, while Terminal-Bench and CritPt sit at -0.047, effectively zero. The concern is narrower: the two highest-weighted agentic benchmarks still correlate at 0.647 after adjustment, leaving 36% of the index partially redundant.

9. The Grand Irony: The Weights Barely Matter

AA uses a 20/16/14/12/8/8/6/6/6/4 weighting formula. I compared it against giving every benchmark a flat 10%.

Results:

• Pearson Correlation: 0.992
• Spearman Rank Correlation: 0.992
• Kendall's Tau: 0.927
• Top-10 Model Overlap: 10 / 10 (same models, minor ordering swaps)

Why does flat averaging match the bespoke formula? Because of the data structure. PCA shows PC1 accounts for 73.2% of total variance across the suite (80.6% if you exclude Non-Hallucination).

Good models are good at everything. Weak models are weak at everything. The specific weights encode what AA prioritizes philosophically, but they barely move the leaderboard. The ranking is dominated by general capability.

That raises an uncomfortable question: if the weights do not matter much, why spend so much effort defending them? They probably matter more for the narrative, signaling what the index values, than for the rankings themselves.

10. Humanity's Last Exam Is Sitting Right There

HLE gets 12% of the index. It probably deserves more. Consider:

• Vast headroom: 2,158 multidisciplinary questions spanning math, humanities, natural sciences. Frontier leader sits at ~55.5%, leaving 44.5% dynamic range before saturation.
• Frontier discrimination: Standardized frontier IQR of 1.695. That's 3.5× higher than GPQA's 0.478.
• General factor loading: Strong PC1 loading without ceiling effects.
• Ablation sensitivity: Removing HLE causes the largest rank shift in leave-one-out testing. It's doing the most work per percentage point of weight.

GPQA is at 94.1% with 5.9% headroom remaining. HLE is at 55.5% with 44.5% headroom. Weight belongs on benchmarks that can still differentiate models. HLE can. GPQA can't.

11. Model vs. Agent vs. System: The Taxonomy Problem

There are three distinct things being evaluated, and the index collapses them into one number:

• 1. Model Capability: Latent reasoning, mathematical deduction, knowledge retrieval, and generalization. These capabilities live in the model weights.
• 2. Agentic Capability: Multi-turn goal decomposition, state tracking, tool invocation, and error recovery across turns.
• 3. System Capability: The whole integrated stack. Model weights plus prompting harness (Stirrup), execution sandboxes, browser automation, visual feedback loops, retry mechanisms.

Standardizing the harness is normal experimental practice. You have to. But different models respond differently to prompt scaffolding, context management, and tool schemas. A model that thrives in Stirrup might underperform in a different harness, and vice versa.

Collapsing all three tiers into a single scalar means you can't tell whether a score improvement came from a better base model or from someone optimizing the harness. That ambiguity is baked into the number.

12. Independence & Incentive Loops

AA presents itself as an independent evaluator. Fair. But look at the operational structure: it works directly with frontier labs on pre-release models, labs cite AA rankings in their launch materials, one of its disclosed backers is an OpenAI Foundation director, and the largest single index component (20%) is AA's implementation of an OpenAI benchmark.

None of this is a scandal. But it makes transparency around weighting decisions and benchmark selection more important, not less.

I'm not alleging backroom deals or manipulation. What I'm saying is that AA hasn't demonstrated its definition of intelligence is sufficiently independent from the product priorities of the labs it measures.

There's an incentive loop running through the industry:

Frontier labs want third-party validation on launch day. AA wants early access so the leaderboard goes live immediately. Labs provide pre-release access. AA runs evaluations. High scores get cited in launch posts. Citations build AA's authority, which encourages the next round of early access.

AA operates openly with OpenAI, Anthropic, and Google. The arrangement resembles tech reviewing, where publications depend on early hardware access for day-one coverage. This is a structural dependency, not evidence of corruption, and it can shape editorial incentives without anyone intending it.

13. Five Things That Would Actually Help

I am not proposing another arbitrary set of weights. These five structural changes would be more useful:

14. Statistical Methodology

586 model and configuration rows across 409 underlying model families. All benchmark scores standardized to z-score units within each evaluation.

Residual Correlation Formula

To strip out general model capability ($\text{Strength}$) as a confound, I fit linear regressions for each benchmark pair $(A, B)$ against the control proxy:

Family-Cluster Bootstrap

Parameter variations and reasoning settings within the same model family aren't independent observations. All confidence intervals use a 500-replicate family-clustered bootstrap to account for that.