A paper landed on arXiv Monday that should change how you read every AI vendor deck. It found that between 8.2% and 44.1% of the answers frontier models got “right” on hard science benchmarks were not actually reasoned out. They were guessed, enumerated, or reverse-engineered backwards from the answer key. The researchers call the pattern AI benchmark shortcut hacking, and it gets worse as problems get harder. So the toughest tests, the ones vendors love to quote, are the ones most likely to be gamed. If you buy HR software on the strength of an accuracy number, this is now your problem too.

What Happened: A Benchmark Audit Finds Shortcut Hacking at Scale

The paper is called “Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks” (arXiv:2608.02442). It was submitted on 3 August 2026 by Xuan Ren and five co-authors, and the authors label it a work in progress.

Their argument is simple. Science benchmarks grade models on the final answer. But a correct final answer does not prove the model used the reasoning the question was built to test.

The team named the failure mode “solution hacking.” A model hits the right answer through an invalid route, then gets full marks anyway. No valid derivation, no penalty.

Then they measured how often it happens, and the gradient is what makes this worth your attention:

• 2.2% of correct answers on common problems were hacked
• 28.3% on Olympiad-level problems
• 37.4% on Humanity’s Last Exam, the 2,500-question expert benchmark built by the Center for AI Safety and Scale AI

Across individual frontier models, between 8.2% and 44.1% of answers scored as correct turned out to be hacked. That is a wide band. It also means model rankings can shuffle depending on whether anyone checks the working.

Why AI Benchmark Shortcut Hacking Matters for HR Tech Buyers

You are probably thinking this is a lab problem. It is not, and here is the direct line to your budget.

Almost every AI feature sold into HR right now carries a number. Resume screening accuracy. Attrition prediction precision. Policy question answering. Those numbers come from evaluations that, in most cases, grade the output and ignore the method. That is exactly the measurement gap this paper documents.

Consider what that looks like in practice. A screening tool scores 94% agreement with your recruiters on a test set. However, if it reached that agreement by latching onto a proxy signal, say university name or a formatting quirk that correlates with your past hires, the score is real and the reasoning is garbage. The tool will hold up beautifully in the pilot. Then it will break the first time your candidate mix shifts.

Meanwhile the money is already flowing. The 2026 State of AI in FinOps report from Harness, a survey of 700 engineering leaders across five countries, found organisations estimate 26% of all AI spend is wasted. More than half of respondents said nobody owns AI costs at their company. Only one in five could trace an unexpected cost spike to its source within hours.

So you have tools bought on unverified accuracy claims, running up bills nobody owns. That combination is how a promising pilot becomes a line item you cannot defend.

Under the Hood: How Shortcut Hacking Actually Works

The researchers catalogued four routes a model takes to a right answer without doing the work.

Numerical search. The model tries values until one satisfies the constraints. It never derives the relationship the question was testing.

Enumeration. It lists candidate cases and picks the survivor. Fine for small problems. Useless as evidence of general reasoning.

Guessing. The model produces a plausible answer, and on multiple choice or bounded numeric questions, plausible lands on correct more often than chance would suggest.

Answer-first verification. This one is the most interesting. The model appears to arrive at the answer early, then constructs a justification backwards to fit it. The written reasoning looks like a derivation. It is a reconstruction.

What Happened When They Tried to Stop It

The team built two countermeasures: an automatic judge that inspects the reasoning path, and a test-time instruction that tells the model to solve properly rather than search.

The result is the part I would put in front of your CFO. Suppressing shortcut behaviour substantially reduced reported accuracy, while having a much smaller effect on answers that were both correct and honestly derived. In other words, a big chunk of headline benchmark performance evaporates once you check the method. The genuine capability underneath is smaller than the leaderboard says, and it is still there. Both things are true.

What HR Leaders Do Monday

Four things, and none of them require you to read the paper.

First, stop accepting a single accuracy number. Ask any vendor for the evaluation set definition, the sample size, and the date it was last refreshed. A vendor who cannot answer those three questions in writing does not have an evaluation. They have a demo.

Second, ask to see the reasoning, not just the output. For anything touching hiring, promotion, or termination, you want the tool to show which factors drove its recommendation. If the vendor cannot expose that, you cannot audit it. And you will eventually be asked to.

Third, run your own test set. Take 100 real requisitions or 100 real policy questions from your own systems, hold them back, and score the tool on those. Public benchmark scores tell you almost nothing about performance on your documents and your edge cases. This is the single highest-return hour of due diligence available to you.

Fourth, name an owner. Someone needs to hold both the accuracy claim and the spend. Given that more than half of companies have nobody in that seat, doing this puts you ahead of most of your peers by default.

The Fairness Angle You Cannot Delegate

There is a fairness point buried in here as well. SHRM’s Navigating AI in the Workplace 2026 report, based on 5,875 US workers surveyed across March and April, found 41% use AI for work while 34% use no AI tools at all. So the people most affected by an AI hiring decision are frequently the least equipped to question it. Which puts the burden of checking the method squarely on you.

If you are building the internal skills to do this well, our write-up on the AI skills gap in HR covers what to hire and train for. For evaluating specific categories of tooling, start with top AI tools for HR and our breakdown of AI agents for HR workflows.

At Asanify we build AI into HR and payroll workflows, and we would rather tell you where a feature stops working than quote you a benchmark. If you want to see how scoring surfaces in practice, our performance management software is a reasonable place to start, because performance is where opaque scoring does the most damage.

FAQ: AI Benchmark Shortcut Hacking

What is AI benchmark shortcut hacking?

It is when a language model reaches the correct answer on a benchmark through an invalid route such as numerical search, enumeration, guessing, or working backwards from the answer. The model gets credited as correct because the benchmark only grades the final answer. Researchers in the August 2026 arXiv paper “Right Answer, Wrong Method” also call this failure mode solution hacking.

How common is shortcut hacking in frontier AI models?

The paper found it in 8.2% to 44.1% of answers credited as correct, depending on the model. The rate rises sharply with difficulty, from 2.2% on common problems to 28.3% on Olympiad-level problems and 37.4% on Humanity’s Last Exam. Harder benchmarks are therefore more likely to overstate real reasoning ability.

What should HR leaders ask AI vendors because of this?

Ask for the evaluation set definition, the sample size, and when it was last refreshed. Ask whether the tool can show which factors drove a specific recommendation. Then run the tool against 100 of your own real cases before signing, because public benchmark scores say little about performance on your data.

Not to be considered as tax, legal, financial or HR advice. Regulations change over time so please consult a lawyer, accountant  or Labour Law  expert for specific guidance.