Google’s engineering benchmark finds top AI agents scoring just 50.6%
Google / Wikimedia Commons (Public domain)
A new Google Cloud AI Research paper tests AI agents on 85 real-world engineering tasks, and the best model barely clears half
The best AI agent available today can pass about half of a realistic engineering exam. That would not get you licensed.
Google Cloud AI Research has released a benchmark called PLE: Physical-Engineering’s Last Exam. It tests whether frontier AI agents can do the work engineers actually bill for. The top performer, Anthropic’s Claude Opus 5.5, posted a mean score of 50.6%. The harder number is this one: the benchmark found its top agent met every hard engineering constraint in only 41.3% of design tasks.
Those constraints are the rules that keep bridges standing and circuits from catching fire. They matter more than the headline score.
What the PLE benchmark actually measures
The benchmark contains 85 long-horizon tasks spread across four disciplines. Mechanical engineering accounts for 30 of them, electrical for 27, civil for 18 and structural for 10.
Each task asks the agent to produce a billable deliverable. Think calculation packages and design reviews, the documents a client pays an engineering firm to deliver.
The agent works from a project corpus that averages 61 files per task. So the model has to sift through a pile of drawings, specs and data before it can start on the actual problem.
Task construction followed two tracks. Experts authored 44 of the tasks end to end. The remaining 41 came from an adversarial creator-solver pipeline, with experts reviewing the output.
The scoreboard, and what it cost
Claude Opus 5.5 led the field at 50.6%, averaged across three runs. OpenAI’s GPT-6 Astra came in close behind at 48.3%.
Google’s own Gemini configurations landed in the low-to-mid 40s.
AI, tech, and the markets they move—in one daily briefing.
Daily. Free. Join 34,000+ readers across crypto, finance, and policy.
The winning performance was not cheap in compute terms. Claude Opus 5.5 consumed roughly 27 million tokens per task and needed 63 minutes to finish each one. That works out to approximately $17 in inference costs per task.
Seventeen dollars for an hour of engineering output sounds like a bargain. The catch is that a deliverable violating a hard constraint is not a bargain at any price.
Why hard constraints are the real story
The gap between the 50.6% mean score and the 41.3% constraint-pass rate tells you where these systems stumble.
A mean score can reward partial credit. An agent might get the methodology mostly right, format the report nicely and still produce a design that breaks a non-negotiable requirement.
The research framing is direct on this point: meeting these stringent constraints is where the difficulty lies, and even the leading models have substantial room to improve. An agent that clears every hard requirement in fewer than half of design tasks is not ready to sign off on anything.
Background: a push toward realistic evaluation
The paper went up on arXiv around October 21-23, 2026, under the identifier arXiv:2610.30537.
Google paired the paper with a public GitHub repository. It includes 29 development tasks along with supporting tools.
The other 56 tasks stay hidden. They are being held back for a potential competition leaderboard.
Keeping most of the test private is a sensible defense against a familiar problem. Once benchmark questions are public, models can end up trained on them, and scores start measuring memory rather than skill.
What this means for engineering and AI developers
For engineering firms, the results draw a clear line. Current agents look capable of meaningful assistance on document-heavy work, such as digesting a 61-file project corpus and drafting a calculation package. They do not look capable of working unsupervised on anything where a missed constraint carries real consequences.
The public development tasks give researchers something concrete to iterate against, which the research summary suggests could encourage collaborative improvement. The hidden competition set means any claimed gains will eventually face an independent test.
The cost profile deserves attention too. If models need tens of millions of tokens and about an hour per task just to reach roughly coin-flip performance, efficiency will matter as much as accuracy.
The metric to track is the constraint-pass rate, not the mean score. At 41.3%, the conversation about AI in licensed engineering is still mostly theoretical.
Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our Editorial Policy.

