Preliminary report ·
Three AI models on 40 visual reasoning questions
A preliminary report on accuracy, completion, API latency and cost: one 40-question run per model, plus separate development checks of reasoning limits and response delivery.
- dated model versions
- 3
- questions per model
- 40
- original run per model
- 1
The original 40-question run
Every model used medium reasoning and an 8,192-token total output ceiling, including reasoning. A cutoff counts as unsuccessful in the raw score.
Swipe the table sideways to see all columns.
| Model | Correct / 40 | Wrong | Cutoffs | Valid / 40 | Median API time | API usage (USD) |
|---|---|---|---|---|---|---|
| Gemini 4 CarbonGoogle AI Studio | 34/4085.0% | 1 | 5 | 35/40 | 13.9 sec | $0.92 |
| Claude Fable 5.2Anthropic | 32/4080.0% | 7 | 1 | 39/40 | 4.0 sec | $1.76 |
| GPT-6 AstraOpenAI | 37/4092.5% | 3 | 0 | 40/40 | 3.7 sec | $1.80 |
Valid means a complete answer in the required format. API time includes network, queueing and generation; this median includes token cutoffs. The extra failed HTTP attempt is recorded separately. Displayed costs are rounded; downloads retain exact charges.
Why completion matters
Carbon returned one wrong answer and had five token cutoffs. Astra returned three wrong answers and had no cutoffs. The apparent gap includes different opportunities to finish, even under the same nominal ceiling.
Reasoning consumes part of the output allowance and can leave no final answer when that allowance runs out. Equal token limits and labels such as “medium” do not establish equal compute across providers. OpenRouter documentation on reasoning tokens
A post-hoc sensitivity check
On the 34 questions with valid answers from all models, Carbon and Astra each scored 33/34; Fable scored 30/34. This subset was selected after observing failures and excludes capped items. It is a sensitivity check, not an unbiased replacement ranking or evidence of equivalent ability.
Separate development checks
Positions 24, 28, 35, 36, 38 and 40 were selected because at least one model exhausted the original allowance on each. These checks tested completion, delivery, time and cost. They were not scored for accuracy: 6/6 valid responses does not mean six correct answers. They were not substituted into the original scores.
Swipe the table sideways to see all columns.
| Profile | Output ceiling | Valid / 6 | Cutoffs | Unavailable | API usage (USD) |
|---|---|---|---|---|---|
| GPT-6 AstraOpenAI · high reasoningReasoning excluded from the API response | 32,768 | 6/6 | 0 | 0 | $1.08 |
| Claude Fable 5.2Anthropic · high reasoningReasoning excluded from the API response | 32,768 | 6/6 | 0 | 0 | $0.80 |
| Gemini 4 CarbonGoogle AI Studio · high reasoningReasoning excluded from the API response | 32,768 | 3/6 | 1 | 2 | $1.05 |
| Gemini 4 CarbonGoogle Vertex global · high reasoningReasoning excluded from the API response | 65,536 | 3/6 | 0 | 3 | $0.49 |
| Gemini 4 CarbonGoogle Vertex global · high reasoningAPI progress included; text discarded locally | 65,536 | 6/6 | 0 | 0 | $1.42 |
The initial Carbon profile also had an earlier unanswered, unbilled timeout whose single recovery completed. That failed attempt remains in the measurements. Later unavailable slots were retained without retry.
One item used 51,145 output tokens
The final Carbon profile used Vertex, a 65,536-token ceiling and streamed reasoning progress with its text discarded locally. All six selected items completed. One took 10 minutes 56 seconds; another took 6 minutes 43 seconds. Both exceeded the earlier five-minute timeout boundary.
This supports a working setup for these development items. It does not prove a universal timeout cause, guarantee future completion, or provide a new 40-question accuracy score. Earlier failures remain separate.
How the study was conducted
- Same task, fresh context
- One static 1152 × 1552 image per item, containing the question and eight numbered choices. The same English prompt and strict JSON schema were used. No tools, browsing, hints or grading key were supplied.
- Frozen choices and offline grading
- A seeded balanced shuffle mapped display labels back to original options. Scores were cross-checked locally against the production grader. No returned baseline answer or cutoff was repeated.
- Transport failures stay visible
- One extra unanswered HTTP 503 in the original run was confirmed unbilled and retried identically. It took 29.516 seconds. Development profiles and recovery history are documented separately.
- Internal exploratory evaluation
- UniversalIQTest.com operates the source test and conducted this study. Request and image hashes, item timings, grading and costs were checked locally. External peer review has not been performed.
Exact model versions and original providers
google/gemini-4-carbon-20261110— Google AI Studioanthropic/claude-fable-5.2-20261109— Anthropicopenai/gpt-6-astra-20260903— OpenAI
Original protocol fingerprint: f4425b7fb5a235cd7c11fb84c4849c4f7df3d43064160f289b8434ec9478f59b
What this report can establish
It documents performance on a specific public question set under declared API settings. Completion, wrong answers, latency and cost require separate reporting. There was one original run per model, so run-to-run variation was not estimated.
Public items may have appeared in training data; contamination was not measured. The numbered static layout differs from the interactive human test. Later development profiles used different settings and collection conditions, so they do not establish a latency ranking.
No human IQ or percentile was measured or assigned to a model. Human test norms do not automatically transfer to AI systems. A future repeated full evaluation would require a new frozen collection plan. OECD discussion of human-oriented tests for AI
Measurements and citation
Recorded API usage: $4.478100 for the original run, $4.843563 for development checks and $0.334781 for technical pilots. Total: $9.656444 ($9.66 rounded). Reconciled with billing; fees and taxes excluded.
- Report (Markdown, English)
- Aggregate results (JSON)
- Per-item measurements (JSON)
- Methods (JSON)
- File hashes (JSON)
All languages share the same source files. The downloadable report, figure, JSON field names and original prompt are in English. Downloads exclude solutions, the private grading key, participant records, credentials and reasoning text. The citation below uses the original report title.
UniversalIQTest.com (2026). Three AI models on 40 visual reasoning questions: preliminary report. 14 November 2026. https://www.universaliqtest.com/ai-iq-testFrequently asked questions
- Are these AI IQ scores?
- These are raw results on 40 visual reasoning questions. No human IQ or percentile was measured or assigned. Output-limit failures and a single run prevent a general intelligence ranking.
- What does a valid calibration answer mean?
- A complete answer in the required JSON format. Calibration was not graded: six valid answers do not mean six correct answers. They did not replace original responses.
- Did models and humans take the same test?
- The figures came from the same public 40-question test. Models received static images with eight numbered choices, a fixed English prompt and fresh context. The interactive human experience was not reproduced. Translating this report does not change the experiment's language.
- Was time measured for every question?
- Yes, for all 120 original question slots. API durations include network, queueing, image processing and generation; they are not isolated thinking time or human active answering time. Extra failed attempts are recorded separately.
Explore the question format or read more about our methods.