Lies and predictions in particle Physics

To most of us, 93-to-1 odds would make for a clear-cut bet. To physicists? Not so much.
On December 15, the New York Times reported that Santa may have brought physics a new subatomic particle, a hitherto unknown entity materialisng in the giant collider at CERN, near Geneva. It wasn’t a sure thing, but according to the Times, the odds are in the scientists’ favour, with only a 1-in-93 chance that the data pointing to the particle represent a statistical fluke.
If real, this is the kind of discovery that would alter the course of a scientific field, catapult careers and revolutionise the world’s understanding of the universe. Focusing on the slim chance of a statistical fluke seems almost as irrationally pessimistic as Dick Cheney’s assertion that if there’s even a 1 per cent chancethat Pakistani scientists were helping Al Qaeda develop nuclear weapons, the response should be the same as if it were a certainty.
It turns out, though, that the physicists’ caution is quite rational. And that there’s a lesson in it for others who would jump to supposedly obvious conclusions about statistics.
Physicists say that it’s not quite correct to state that those 1-in-93 odds refer to the chance of a statistical fluke. The correct interpretation, says UCLA physicist Robert Cousins, is that the odds are just 1-in-93 that scientists would see strong evidence of a particle's presence even if it actually wasn't there. To understand why that’s too high a risk, he refers to a problem of statistics called the prosecutor’s fallacy.
Take this example, from a website called Statistics for the Terrified. It starts with a purse-snatching incident and an eyewitness describing the thief as a tall man between 20 and 30 years old with red hair and a limp. The police scour the city, which happens to be London, and seize the first tall, male redhead they find limping down the street.
The prosecutor argues that the odds of fitting such a peculiar description are tiny - roughly one in half a million. So he reasons that there’s only a 1-in-500,000 chance that the man they apprehended is innocent.
That would convince many a jury, but it’s wrong. There’s a 1-in-500,000 chance that someone picked at random would fit that description, but that’s not the same as the odds of innocence. In a city of 10 million people, 1-in-500,000 translates to 20 people. If cops picked one of them at random, there’s only a 1- in-20 chance he’s their purse snatcher.
There’s a similar doctor’s fallacy. Many doctors assume that if a test for a disease is 99-per cent accurate, there must be a 99-per cent chance that a patient testing positive has the disease. Nope. Imagine it's a disease that affects 1 person in 1,000. Assuming the test has a 1-per cent false-positive rate, then there will be many more people testing positive than there are actual cases. Only 9 percent of those testing positive will have the disease.
To understand how this fallacy applies to particle physics, it helps to consider the way the experiments work. Physicists don’t discover new particles the way scientists might find a new species of dinosaur. They're too small to see and they last just a fraction of a second before disintegrating into a spray of other particles.
Those other particles leave tracks in a detector, but the tracks don’t come labelled. Deciding whether an unusual confluence of tracks came from a new particle is a little like determining whether leaks from a chemical plant might be increasing cancer risk among the local population. Some regions will have more cancer cases than others by chance. But the more cases that turn up in the region in question, the less likely an observed cluster is the result of chance alone.
It might prompt some worry if researchers calculated that there was only a 1-in-93 chance that, absent any carcinogenic chemical spill, a small town would see two dozen cases of some type of leukemia over a short time. But physicists aren't dealing with sick people. For them, waiting is the prudent reaction to uncertainty.
They’ve been burned before. In 1984, the New York Times ran a story about a possible game-changing particle called the Zeta. Don’t worry if you haven’t heard of it. It never existed.
For doctors and lawyers, it’s useful to know the rarity of a disease or the fact that just one person in London committed a particular crime. Physicists have no equivalent information in estimating the odds that their new particle exists. Cousins said that all they can do is get a lot more data and wait until they see something that’s much more unusual – a pattern in the data with less than a one-in-a-million chance of appearing if there’s no new particle.
In a world ruled by uncertainty, how do scientists know when to declare a discovery? Cousins said that it’s a subjective decision, depending on how easy it is to collect more data, whether a scientist is in danger of being scooped and how much someone’s reputation will be hurt by being wrong.