Coding benchmarks mismeasure ability, and even the best models fail, critics say

Oct 12, 2026 · Issue 11 · 3 sources, 3 articles

Cognition's FrontierCode 1.1 benchmark tests whether maintainers of real open-source repositories would accept AI-generated changes, and leading models still often fail. AI21 says coding benchmarks mismeasure ability, and one study finds that AI vulnerability patchers can stop a crash without actually fixing the underlying bug.

Who is affected and how

tech lead

Do not choose a tool based on its benchmark score; run it on your own tasks and judge whether the output is mergeable.

Sources

• Announcing FrontierCode on Artificial Analysis: can AI models write code maintainers would merge? | Artificial Analysisartificialanalysis.ai · Oct 12, 2026
• AI21: coding benchmarks mismeasure | AI Newsaihirek.info · Oct 14, 2026 · magyar
• AI vulnerability patching has a grading problem - Help Net Securityhelpnetsecurity.com · Oct 12, 2026

Machine-generated summary based on the titles and excerpts of the source articles. The facts come from the linked article.