SciCode

Can the model reason accurately about difficult scientific material?

Scientific knowledge and reasoning under the benchmark's published question set. Higher is better. Official main-problem resolve rate over the 80 main problems.

This is the repository table's main-problem measurement. The result changelog date is not stated in the pinned README; keep the repository revision, exact source label, and separate subproblem measurement attached.

Official SciCode repository README aggregate row; exact source model label "Claude3.5-Sonnet"; source table rank not ranked; main-problem resolve rate 4.6% over 80 main problems; subproblem accuracy 26% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Claude3.5-Sonnet (new)"; source table rank not ranked; main-problem resolve rate 4.6% over 80 main problems; subproblem accuracy 25.3% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Claude3-Opus"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 21.5% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Claude3-Sonnet"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 17% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Deepseek-Coder-v2"; source table rank not ranked; main-problem resolve rate 3.1% over 80 main problems; subproblem accuracy 21.2% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Deepseek-R1"; source table rank not ranked; main-problem resolve rate 4.6% over 80 main problems; subproblem accuracy 28.5% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Deepseek-v3"; source table rank not ranked; main-problem resolve rate 3.1% over 80 main problems; subproblem accuracy 23.7% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Gemini 1.5 Pro"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 21.9% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "GPT-4-Turbo"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 22.9% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "GPT-4o"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 25% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Llama-3.1-405B-Chat"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 19.8% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Llama-3.1-70B-Chat"; source table rank not ranked; main-problem resolve rate 0% over 80 main problems; subproblem accuracy 17% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Llama-3-70B-Chat"; source table rank not ranked; main-problem resolve rate 0% over 80 main problems; subproblem accuracy 14.6% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Mixtral-8x22B-Instruct"; source table rank not ranked; main-problem resolve rate 0% over 80 main problems; subproblem accuracy 16.3% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "OpenAI o1-mini"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 22.2% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "OpenAI o1-preview"; source table rank not ranked; main-problem resolve rate 7.7% over 80 main problems; subproblem accuracy 28.5% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "OpenAI o3-mini-high"; source table rank 2; main-problem resolve rate 9.2% over 80 main problems; subproblem accuracy 34.4% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "OpenAI o3-mini-low"; source table rank 1; main-problem resolve rate 10.8% over 80 main problems; subproblem accuracy 33.3% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "OpenAI o3-mini-medium"; source table rank 3; main-problem resolve rate 9.2% over 80 main problems; subproblem accuracy 33% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred. Official SciCode repository README aggregate row; exact source model label "Qwen2-72B-Instruct"; source table rank not ranked; main-problem resolve rate 1.5% over 80 main problems; subproblem accuracy 17% over 338 subproblems; repository revision e3158ea011d4235245a547460d3688d7ccbf9900 dated 2025-10-07; result notice date not stated in the pinned README. No task-level material, prompts, answers, code, test cases, or cost is retained or inferred.