Humanity’s Last Exam

Can the model answer the text-only portion of deliberately difficult expert questions?

The publisher's finalized HLE text-only view, preserving the official score, confidence interval, calibration error, and contamination warning fields. Higher is better. The value is the percentage reported in this lab's table.

Harness, tools, prompts, attempt count, and benchmark version can materially change the result. Compare within one reporting context.

Exact starred parenthetical from DeepSeek's V4.1-Flash release table; not merged into the full HLE value. The source prints this parenthetical value with an asterisk; its footnote says the asterisk is the pure-text HLE subset. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Claude 3.5 Sonnet (October 2024)"; source version not reported; company anthropic; rank 49; score 4.32 on publisher max score 57.94780000000001; confidenceInterval_upper 0.86; calibrationError 83; contamination message This model was used as an initial filter for the dataset.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Claude 3.7 Sonnet (Thinking)"; source version not reported; company anthropic; rank 34; score 7.89 on publisher max score 57.94780000000001; confidenceInterval_upper 1.14; calibrationError 81; contamination message Thinking budget: 16,000 tokens. Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Claude Opus 4 \n"; source version not reported; company anthropic; rank 39; score 6.26 on publisher max score 57.94780000000001; confidenceInterval_upper 1.02; calibrationError 75; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-23T15:40:08.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-opus-4-1-20250805"; source version not reported; company anthropic; rank 36; score 7.37 on publisher max score 57.94780000000001; confidenceInterval_upper 1.1; calibrationError 71; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-08-08T17:20:25.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-opus-4-1-20250805-thinking"; source version not reported; company anthropic; rank 25; score 11.26 on publisher max score 57.94780000000001; confidenceInterval_upper 1.33; calibrationError 72; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-08-08T17:19:48.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-opus-4-5-20251101"; source version not reported; company anthropic; rank 21; score 13.9 on publisher max score 57.94780000000001; confidenceInterval_upper 1.46; calibrationError 57; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-11-26T20:29:10.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-opus-4-5-20251101-thinking"; source version not reported; company anthropic; rank 9; score 26.32 on publisher max score 57.94780000000001; confidenceInterval_upper 1.86; calibrationError 55; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-11-26T20:27:42.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-opus-4-6 (Non-Thinking) "; source version not reported; company anthropic; rank 13; score 19.37 on publisher max score 57.94780000000001; confidenceInterval_upper 1.67; calibrationError 46; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. ; isNew=false; new=false; deprecated=false; entry timestamp 2026-02-17T17:06:51.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-opus-4-6-thinking-max"; source version not reported; company anthropic; rank 5; score 36.24 on publisher max score 57.94780000000001; confidenceInterval_upper 2.03; calibrationError 46; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2026-02-17T17:06:06.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Claude Opus 4 (Thinking)"; source version not reported; company anthropic; rank 27; score 10.8 on publisher max score 57.94780000000001; confidenceInterval_upper 1.31; calibrationError 73; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-24T06:34:16.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Claude Sonnet 4"; source version not reported; company anthropic; rank 45; score 5.42 on publisher max score 57.94780000000001; confidenceInterval_upper 0.96; calibrationError 76; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-23T15:40:14.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-sonnet-4-5-20250929"; source version not reported; company anthropic; rank 35; score 7.65 on publisher max score 57.94780000000001; confidenceInterval_upper 1.12; calibrationError 71; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-10-02T17:30:40.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "claude-sonnet-4-5-20250929-thinking"; source version not reported; company anthropic; rank 21; score 14.09 on publisher max score 57.94780000000001; confidenceInterval_upper 1.47; calibrationError 66; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. ; isNew=false; new=false; deprecated=false; entry timestamp 2025-10-02T17:29:42.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Claude Sonnet 4 (Thinking)\n"; source version not reported; company anthropic; rank 35; score 7.6 on publisher max score 57.94780000000001; confidenceInterval_upper 1.12; calibrationError 76; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-23T15:40:03.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "DeepSeek R1"; source version not reported; company deepseek; rank 32; score 8.54 on publisher max score 57.94780000000001; confidenceInterval_upper 1.18; calibrationError 73; contamination message none reported; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "DeepSeek-R1-0528"; source version not reported; company deepseek; rank 21; score 14.04 on publisher max score 57.94780000000001; confidenceInterval_upper 1.47; calibrationError 78; contamination message Evaluation used 32k context, 2-hour timeout with 7 retries (hosted on 16 B200s), temp 0.6, and top_p 0.95. 25 prompts timed out; 2 had connection errors. Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-06-02T19:32:49.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "DeepSeek-V3.1"; source version not reported; company deepseek; rank 21; score 12.88 on publisher max score 57.94780000000001; confidenceInterval_upper 1.41; calibrationError 68; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-09-08T23:47:59.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "DeepSeek V3 (March 2025)"; source version not reported; company deepseek; rank 48; score 4.55 on publisher max score 57.94780000000001; confidenceInterval_upper 0.88; calibrationError 80; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Fable 5.1 (xhigh)"; source version not reported; company Anthropic; rank 2; score 46.8 on publisher max score 57.94780000000001; confidenceInterval_upper 2.1; calibrationError 20; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=true; new=true; deprecated=false; entry timestamp 2026-09-03T18:19:45.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Gemini-1.5-Pro-002"; source version not reported; company google; rank 48; score 4.55 on publisher max score 57.94780000000001; confidenceInterval_upper 0.88; calibrationError 87; contamination message This model was used as an initial filter for the dataset.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Gemini 2.0 Flash Thinking (January 2025)"; source version not reported; company google; rank 38; score 6.55 on publisher max score 57.94780000000001; confidenceInterval_upper 1.04; calibrationError 82; contamination message Sampled at temperature 0.7; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Gemini 2.5 Flash (April 2025)"; source version not reported; company google; rank 21; score 12.58 on publisher max score 57.94780000000001; confidenceInterval_upper 1.4; calibrationError 81; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-17T20:00:24.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Gemini 2.5 Flash Preview (May 2025) "; source version not reported; company google; rank 27; score 10.72 on publisher max score 57.94780000000001; confidenceInterval_upper 1.31; calibrationError 83; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-20T18:30:40.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Gemini 2.5 Pro Experimental (March 2025)"; source version not reported; company google; rank 14; score 18.38 on publisher max score 57.94780000000001; confidenceInterval_upper 1.64; calibrationError 71; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature = 1.0, top_p = 0.95.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:28:45.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gemini-2.5-pro-preview-06-05"; source version not reported; company google; rank 12; score 22.06 on publisher max score 57.94780000000001; confidenceInterval_upper 1.75; calibrationError 72; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-06-05T16:29:16.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Gemini 2.5 Pro Preview (May 06 2025)"; source version not reported; company google; rank 14; score 18.38 on publisher max score 57.94780000000001; confidenceInterval_upper 1.64; calibrationError 70; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-06T16:30:01.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gemini-3.1-flash-lite-preview"; source version not reported; company google; rank 34; score 8.02 on publisher max score 57.94780000000001; confidenceInterval_upper 1.15; calibrationError 83; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2026-03-23T21:14:51.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gemini-3.1-pro-preview (thinking high)"; source version not reported; company google; rank 2; score 47.31 on publisher max score 57.94780000000001; confidenceInterval_upper 2.11; calibrationError 50; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2026-04-10T15:51:36.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Gemini 3.8 Flash"; source version not reported; company google; rank 2; score 44.95 on publisher max score 57.94780000000001; confidenceInterval_upper 2.1; calibrationError 51; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=true; new=true; deprecated=false; entry timestamp 2026-09-09T19:04:40.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gemini-3-pro-preview"; source version not reported; company google; rank 4; score 37.72 on publisher max score 57.94780000000001; confidenceInterval_upper 2.04; calibrationError 57; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-11-19T23:51:38.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "glm-4p5"; source version not reported; company zai; rank 30; score 9.64 on publisher max score 57.94780000000001; confidenceInterval_upper 1.25; calibrationError 79; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at 32K Tokens, temp = null (default temp).; isNew=false; new=false; deprecated=false; entry timestamp 2025-08-13T20:55:50.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "glm-4p5-air"; source version not reported; company zai; rank 30; score 9.41 on publisher max score 57.94780000000001; confidenceInterval_upper 1.23; calibrationError 77; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at 32K Tokens, temp = null (default temp).; isNew=false; new=false; deprecated=false; entry timestamp 2025-08-13T20:56:27.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "GPT-4.1"; source version not reported; company openai; rank 46; score 4.97 on publisher max score 57.94780000000001; confidenceInterval_upper 0.92; calibrationError 88; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-14T18:11:42.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "GPT 4.5 Preview"; source version not reported; company openai; rank 40; score 5.8 on publisher max score 57.94780000000001; confidenceInterval_upper 0.99; calibrationError 83; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "GPT-4o (November 2024)"; source version not reported; company openai; rank 50; score 2.32 on publisher max score 57.94780000000001; confidenceInterval_upper 0.64; calibrationError 88; contamination message This model was used as an initial filter for the dataset.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5.1-instant"; source version not reported; company openai; rank 38; score 6.49 on publisher max score 57.94780000000001; confidenceInterval_upper 1.04; calibrationError 68; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-11-26T20:29:58.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5.1-thinking"; source version not reported; company openai; rank 10; score 24.65 on publisher max score 57.94780000000001; confidenceInterval_upper 1.82; calibrationError 54; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-11-26T20:28:20.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5.2-2025-12-11"; source version not reported; company openai; rank 9; score 28.5 on publisher max score 57.94780000000001; confidenceInterval_upper 1.9; calibrationError 45; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-12-15T23:39:42.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5-2025-08-07"; source version not reported; company openai; rank 9; score 26.32 on publisher max score 57.94780000000001; confidenceInterval_upper 1.86; calibrationError 50; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at reasoning_effort: 'high'.; isNew=false; new=false; deprecated=false; entry timestamp 2025-08-07T21:19:06.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5.4-2026-03-05 (xhigh thinking)"; source version not reported; company openai; rank 5; score 36.47 on publisher max score 57.94780000000001; confidenceInterval_upper 2.03; calibrationError 42; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2026-03-10T21:10:15.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5.4-pro-2026-03-05"; source version not reported; company openai; rank 2; score 45.32 on publisher max score 57.94780000000001; confidenceInterval_upper 2.1; calibrationError 37; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2026-03-23T21:14:11.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5-mini-2025-08-07"; source version not reported; company openai; rank 13; score 19.74 on publisher max score 57.94780000000001; confidenceInterval_upper 1.679; calibrationError 64; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-08-22T21:45:46.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-5-pro-2025-10-06"; source version not reported; company openai; rank 6; score 33.32 on publisher max score 57.94780000000001; confidenceInterval_upper 1.99; calibrationError 49; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-11-06T22:49:17.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "GPT 6 Astra"; source version not reported; company openai; rank 1; score 54.17 on publisher max score 57.94780000000001; confidenceInterval_upper 2.09; calibrationError 40; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=true; new=true; deprecated=false; entry timestamp 2026-09-09T19:03:00.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-oss-120b"; source version not reported; company openai; rank 19; score 15.48 on publisher max score 57.94780000000001; confidenceInterval_upper 1.53; calibrationError 76; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-09-18T22:08:42.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "gpt-oss-20b"; source version not reported; company openai; rank 30; score 9.73 on publisher max score 57.94780000000001; confidenceInterval_upper 1.25; calibrationError 76; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-09-18T22:09:34.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "kimi-k2-instruct"; source version not reported; company moonshot; rank 46; score 4.68 on publisher max score 57.94780000000001; confidenceInterval_upper 0.89; calibrationError 82; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.6.; isNew=false; new=false; deprecated=false; entry timestamp 2025-07-23T20:40:41.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Llama 4 Maverick"; source version not reported; company meta; rank 45; score 5.34 on publisher max score 57.94780000000001; confidenceInterval_upper 0.95; calibrationError 84; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Mistral Medium 3"; source version not reported; company mistral; rank 49; score 4.36 on publisher max score 57.94780000000001; confidenceInterval_upper 0.86; calibrationError 78; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-13T17:48:23.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Muse Spark"; source version not reported; company meta; rank 4; score 40.92 on publisher max score 57.94780000000001; confidenceInterval_upper 2.07; calibrationError 51; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2026-04-08T16:58:25.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Nova Lite"; source version not reported; company amazon; rank 49; score 3.76 on publisher max score 57.94780000000001; confidenceInterval_upper 0.8; calibrationError 83; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Nova Micro"; source version not reported; company amazon; rank 48; score 4.41 on publisher max score 57.94780000000001; confidenceInterval_upper 0.87; calibrationError 84; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Nova Pro"; source version not reported; company amazon; rank 49; score 4.32 on publisher max score 57.94780000000001; confidenceInterval_upper 0.86; calibrationError 80; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o1 (December 2024)"; source version not reported; company openai; rank 35; score 7.75 on publisher max score 57.94780000000001; confidenceInterval_upper 1.13; calibrationError 84; contamination message This model was used as an initial filter for the dataset.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o1 Pro"; source version not reported; company openai; rank 35; score 7.71 on publisher max score 57.94780000000001; confidenceInterval_upper 1.13; calibrationError 82; contamination message 9% (196 prompts) failed due to a post-training bug and were counted as failures. OpenAI has been informed and is working on a fix. --- Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o3 (high) (April 2025)"; source version not reported; company openai; rank 13; score 20.57 on publisher max score 57.94780000000001; confidenceInterval_upper 1.71; calibrationError 36; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-16T23:15:14.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o3 (medium) (April 2025)"; source version not reported; company openai; rank 13; score 19.78 on publisher max score 57.94780000000001; confidenceInterval_upper 1.68; calibrationError 37; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-16T17:05:11.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o3 mini (high)"; source version not reported; company openai; rank 21; score 13.37 on publisher max score 57.94780000000001; confidenceInterval_upper 1.44; calibrationError 80; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o3 mini (medium)"; source version not reported; company openai; rank 28; score 10.31 on publisher max score 57.94780000000001; confidenceInterval_upper 1.28; calibrationError 81; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-10T19:29:02.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o4-mini (high) (April 2025)"; source version not reported; company openai; rank 13; score 18.9 on publisher max score 57.94780000000001; confidenceInterval_upper 1.65; calibrationError 58; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-16T23:15:25.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "o4-mini (medium) (April 2025)"; source version not reported; company openai; rank 21; score 14.53 on publisher max score 57.94780000000001; confidenceInterval_upper 1.49; calibrationError 58; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.3.; isNew=false; new=false; deprecated=false; entry timestamp 2025-04-16T17:05:18.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Qwen3-235B-A22B"; source version not reported; company alibaba; rank 23; score 11.75 on publisher max score 57.94780000000001; confidenceInterval_upper 1.36; calibrationError 74; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.6.; isNew=false; new=false; deprecated=false; entry timestamp 2025-05-01T15:50:28.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred. Official Scale AI Humanity's Last Exam text-only aggregate row; exact source model label "Qwen3-235B-A22B-Thinking-2507"; source version not reported; company alibaba; rank 19; score 15.43 on publisher max score 57.94780000000001; confidenceInterval_upper 1.52; calibrationError 78; contamination message Potential contamination warning: This model was evaluated after the public release of HLE, allowing model builder access to the prompts and solutions. Sampled at temperature 0.6 and 32768 max tokens.; isNew=false; new=false; deprecated=false; entry timestamp 2025-08-05T18:07:41.000Z. The public aggregate feed does not expose a reproducible agent configuration or cost series, so no tool setup, task-level result, or cost is inferred.