🔍 Read the full analysis: FrontierCode: How Maintainers Evaluate Pull Requests Written By AI on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, a benchmark that scores repository changes against maintainer-style review criteria rather than tests alone. Claude Opus 5.5 led with a 52.2% overall score and cleared every blocking check in 57.7% of attempts. The results show a wide cost spread and leave open how well benchmark performance predicts real-world maintainer decisions.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark designed to judge whether a pull request meets maintainer-style review standards, not merely whether it passes tests. Claude Opus 5.5 led with a 52.2% score, but cleared every blocking check in 57.7% of attempts, leaving substantial room for human review.
FrontierCode’s results distinguish between an overall score and the share of attempts that pass all blocking checks. The top model, Claude Opus 5.5 (max with fallback), scored 52.2% overall and had a 57.7% blocking-check pass rate. GPT-6.1 Sol (max) placed second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Artificial Analysis reports that it ran the models through the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task.
The benchmark assesses patches across six review dimensions: hidden tests, command checks, tests that must fail against the original code, adaptive tests, limits on files and diff size, and an LLM review against maintainers’ criteria. Failing a blocking check makes an attempt score zero. Attempts flagged by Cognition’s trajectory scanner for looking up a solution also receive zero; excessive diffs, irrelevant edits and unhelpful tests can reduce scores. The review judge in this evaluation was GPT-6 Luna (medium).
The Main set used for the evaluation contains 100 tasks drawn from a larger 150-task benchmark. According to the source report, more than 20 maintainers developed tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost, and spent more than 40 hours on each task. The report says nine of the 100 tasks were not solved by any model on any attempt. It also lists cost per task from $0.09 for GPT-6 Luna to $12.00 for Claude Fable 5.1, with Claude Opus 5.5 at $10.15 and GPT-6.1 Sol at $0.81.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainer-Style Scoring Matters
Many coding evaluations focus on whether a generated change passes tests. FrontierCode adds a different question: would the change be acceptable to a repository maintainer? Its checks include whether tests meaningfully detect regressions, whether edits stay within scope, and whether the patch is easy to review. Those criteria address reasons a technically working contribution may still be rejected or require substantial revision.
The results caution against treating a high coding score as evidence that an AI agent can independently produce merge-ready changes. Even the leading system failed at least one blocking check in more than two-fifths of attempts, based on the reported 57.7% pass rate. That does not establish how often models succeed on ordinary pull requests, but it indicates that, on this benchmark, human review remains relevant.
Cost results also make the ranking consequential for teams considering repeated model use. Artificial Analysis identifies GPT-6 Luna, MiMo-V2.6-Pro, GPT-6.1 Sol and Claude Opus 5.5 as points on its cost-performance frontier. Moving from Sol to Opus raises the score by 3.1 percentage points while the reported cost per task rises from $0.81 to $10.15. Those figures describe this test setup, not a universal operating cost or a guarantee of better outcomes in a particular codebase.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Its version 1.1 followed in July with rules against searching online for solutions, according to the source report. The task set is private, intended to reduce the chance that models encountered the solutions during training. Artificial Analysis added the benchmark to its evaluations on October 12 and tested 23 models.
The benchmark’s scoring design makes passing tests necessary but not sufficient. Each attempt receives zero if it fails a blocking check, while attempts that pass all blocking checks score between 70% and 100%, depending on quality checks. Maintainers’ criteria inform the review component, and the benchmark also penalizes patterns that can burden reviewers, such as unnecessarily broad changes. The report attributes this distinction to Celery maintainer Tomer Nosrati, who said FrontierCode grades “like a tech lead,” rather than “like a CI.”
Performance also differs from rankings on general model evaluations cited in the source. Claude Sonnet 5.5, for example, is listed at 45.8% and fifth on FrontierCode, while DeepSeek V4.1 Flash scores 44.0% and ranks seventh. Such comparisons suggest that general capability rankings may not map directly to this benchmark’s review-focused criteria; they do not by themselves establish which model is best for every software task.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Current Results
The published figures do not show how FrontierCode scores translate into acceptance rates across live open-source projects. The 100-task Main set is a benchmark sample, and the source material does not establish that its tasks represent the frequency or difficulty of changes encountered by maintainers day to day. Nor does a blocking-check pass rate reveal whether a particular patch would be accepted after discussion or revision.
Several models were missing from the results as of October 13, according to the source report, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the table is not a complete comparison of all models discussed in the report. The source also does not provide enough detail here to assess uncertainty ranges or the statistical significance of small score differences, such as the 0.1-point gap between GPT-6.1 Sol and Claude Fable 5.1.
Costs and scores depend on the evaluation setup, including the agent harness, attempt limits and model configurations. The reported costs should not be read as fixed prices for other workloads. The source material gives no results from independent replications or a breakdown of performance across all repositories and task types.
As an affiliate, we earn on qualifying purchases.
Pending Models and Further Tests
Further results could change the comparison as more models are evaluated. The source report specifically identifies Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as lacking FrontierCode results at the time of publication. Their scores, if released, would broaden the field but would not answer on their own how well benchmark results predict real maintainer decisions.
Readers should watch for updates to the leaderboard, details about results by task or repository, and independent evaluations using the same benchmark. Until then, the current findings support a narrower conclusion: the leading tested model passed all blocking checks in a majority, but not all, of its attempts, and higher scores can come with materially higher reported costs.
AI coding assistant for developers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates repository changes using tests, scope rules and a review against maintainers’ criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in the Artificial Analysis results, with a 52.2% overall score and a 57.7% rate of clearing every blocking check.
Does a passing score mean the AI pull request would be merged?
No. The benchmark measures performance on its tasks and checks; it does not establish that a patch would be accepted in a live project. Maintainers may request changes or reject a contribution for reasons not captured by the evaluation.
How many tasks and models were included?
Artificial Analysis tested 23 models on the 100-task Main set. The full benchmark has 150 tasks, according to the source report.
What remains unknown?
The results do not establish how scores translate to real-world pull-request acceptance, and several models had no result as of October 13, 2026. The source also does not provide statistical uncertainty for close scores.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
