Benchmarking and evaluating advantage claims
Since 2019 quantum advantage has been announced many times, and classical computers have taken back several of those announcements within weeks. As a reader of press releases, a reviewer of grant applications, or someday an investor or policymaker, you need a way to tell a solid claim from a fragile one. This chapter gives you the metrics hardware makers quote, what each hides, and a ten-question checklist you can apply to any claim.
On the plate: four famous claims, scored against the checklist.
Metrics and what they hide
Quantum volume (IBM, 2019) is for the largest at which random square circuits ( qubits, depth ) pass a heavy-output test; it rewards fidelity and connectivity but saturates for large machines. Layer fidelity and EPLG (error per layered gate) measure full layers of two-qubit gates across a chain of qubits, catching crosstalk that isolated-gate numbers miss. CLOPS counts circuit layers per second, including classical overhead. For error-corrected machines, the number that matters is logical error per cycle and how it scales with distance (Chapter 139). Average gate fidelity alone hides coherent and correlated errors (Chapter 135). Established
Ten questions for any advantage claim
(1) Is the task defined precisely? (2) Is the output verifiable, classically or by another quantum device? (3) Was the classical baseline the best available, run by experts in that method? (4) Is the classical cost estimate for exact or for approximate simulation at the device’s actual fidelity? (5) Have independent groups tried to simulate it? (6) Is the task useful, or a benchmark only? (7) Are error bars and raw data public? (8) Does the advantage grow with size, or is it a single point? (9) Is post-selection or error mitigation hiding a cost? (10) Who paid for the press release?
Choose a claim: . IBM’s June 2023 Nature paper by Youngseok Kim and colleagues simulated spin dynamics on 127 qubits with error mitigation. Within two weeks Joseph Tindall and colleagues reproduced it with tensor-network belief propagation, and Tomislav Begušić, Johnnie Gray and Garnet Chan with sparse Pauli dynamics, both on ordinary computers. That did not make the experiment worthless: it showed what error mitigation can do. It did show that the classical side had not been tried hard enough. The scores on the plate are this book’s judgement, shown so you can disagree. Debated
Value against cost
The US Defense Advanced Research Projects Agency asks a sharper question in its Quantum Benchmarking Initiative: will a given approach deliver computations whose value exceeds their cost, at utility scale, by 2033? In November 2025 it advanced eleven companies to Stage B, a year of detailed scrutiny of their designs by independent teams. That framing, value over cost, rather than speed on a contrived task, is the right one for anyone deciding where money and careers should go. Established for the programme; Speculative for its outcome.
What you now have
Quantum volume, layer fidelity and EPLG, CLOPS, logical error per cycle, a ten-question checklist for advantage claims, the IBM 2023 case and its classical replies, and DARPA’s value-over-cost test.
· · ·
Three questions to carry with you.
And one to work out. A machine passes the quantum-volume test for square circuits of width 20 but not 21. What is its quantum volume?
Sources: A. W. Cross et al., Phys. Rev. A 100, 032328 (2019). D. C. McKay et al., arXiv:2311.05933 (layer fidelity). Y. Kim et al., Nature 618, 500 (2023). J. Tindall et al., PRX Quantum 5, 010308 (2024). T. Begušić, J. Gray and G. K.-L. Chan, Sci. Adv. 10, eadk4321 (2024). DARPA Quantum Benchmarking Initiative, Stage B selection (November 2025). Last checked 18 November 2026.