88
Part Two · Quantum fields and particles

Collider data analysis and statistics (5 sigma, look-elsewhere effect)

In December 2015 both ATLAS and CMS showed a small excess of photon pairs with combined mass near 750 GeV. Within months, theorists had written more than five hundred papers explaining it. In August 2016, with more data, it was gone: a fluctuation. Particle physicists demand five standard deviations before claiming a discovery, a chance of about one in 3.5 million that background alone would do it, precisely because smaller excesses, three sigma and below, have come and gone many times. This chapter is about the statistics that decide when a bump is real.

On the plate: a falling mass spectrum with nothing in it but noise. Look for bumps.

From σ to p

A p-value is the probability, if only background were present, of seeing a result at least as extreme as the one observed. Physicists quote it as a number of standard deviations of a Gaussian, one-sided: 3σ is p=1.35×10−3p = 1.35\times10^{-3}, “evidence”; 5σ is p=2.87×10−7p = 2.87\times10^{-7}, “observation”. Neither is the probability that the signal is false; that would need prior beliefs (Chapter 10). Five sigma is a convention, chosen to protect against systematic errors and the multiple-testing problem below as much as against chance. Established

Likelihoods

Given a model with signal strength μ\mu and nuisance parameters θ\theta (background normalization, energy scale, and so on), the likelihood L(μ,θ)L(\mu, \theta) is the probability of the observed data. To test μ=0\mu = 0, compare the best fit with μ\mu free against the best fit with μ=0\mu = 0, each time adjusting θ\theta to its best value: the profile likelihood ratio. By Wilks’ theorem its logarithm follows a known distribution, giving the significance directly. For a simple count with expected background bb and signal ss, the median significance is

Z=2[(s+b)ln⁡(1+s/b)−s]Z = \sqrt{2\left[(s+b)\ln(1 + s/b) - s\right]}

the “Asimov” formula of Glen Cowan, Kyle Cranmer, Eilam Gross and Ofer Vitells; it reduces to s/bs/\sqrt b when s≪bs \ll b. For setting upper limits, experiments use a modified method called CLs, which avoids excluding signals they have no sensitivity to.

The look-elsewhere effect

If you search a spectrum at many masses, some bin will fluctuate high by chance. a few times and watch the largest local excess. In 60 independent bins, the biggest fluctuation is typically above 2σ, and one time in thirteen above 3σ. A 3σ local excess found by scanning is far less significant globally. Eilam Gross and Ofer Vitells gave a practical correction in 2010, counting how often the significance curve crosses a threshold. The 750 GeV excess was 3.9σ local in ATLAS but about 2σ global.

Now add a real signal of 0 events at bin 30, and regenerate. A real signal stays at the same place every time; noise moves.

Blinding

The most dangerous bias is the analyst’s own: stopping when the answer looks right, choosing cuts that enhance a bump. So analyses are blinded: the selection, background model and uncertainties are fixed using simulation and control regions, and only then is the signal region opened. The Higgs search of 2012 was blinded this way. Machine-learning classifiers, now used in most analyses to separate signal from background, must also be trained and validated without looking at the data they will be applied to. Established

What you now have

p-values and their σ equivalents, what 5σ does and does not mean, likelihoods and profile likelihoods, the Asimov significance formula, CLs limits, the look-elsewhere effect, and blinding.

· · ·

Three questions to carry with you.

Is p = 2.87×10⁻⁷ the probability that the signal is fake?tap
No. It is the probability of data this extreme if only background were present.
Why is a local 3σ bump often unremarkable?tap
Searching many places makes some fluctuation that large likely; the global significance is much lower.
Why blind an analysis?tap
To prevent choices from being tuned, consciously or not, toward the desired result.

And one to work out. With s=10s = 10 signal and b=4b = 4 background events, what is the Asimov significance ZZ?

Z =

Sources: G. Cowan, K. Cranmer, E. Gross and O. Vitells, Eur. Phys. J. C 71, 1554 (2011). E. Gross and O. Vitells, Eur. Phys. J. C 70, 525 (2010). A. L. Read, J. Phys. G 28, 2693 (2002) (CLs). Particle Data Group, statistics review. ATLAS, JHEP 09, 001 (2016) and CMS, Phys. Rev. Lett. 117, 051802 (2016) (750 GeV diphoton). Checked by code: one-sided p for 3σ and 5σ; Asimov Z = 3.88 for s = 10, b = 4; in 60 bins the median maximum local significance is 2.28σ and P(max > 3σ) = 0.079.

Next · Chapter 89
The Higgs discovery and what it measured

The 4 July 2012 announcement, rebuilt from the data.

Turn the page