### Answers to Probability Puzzle

As usual, smart readers knocked the cover off the ball almost immediately. Some day I’ll stump you.

First question taken from *Randomness* by Deborah J Bennett:

“If a test to detect a disease whose prevalence is one in a thousand has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person’s symptoms or signs?”

First to answer correctly was Matt Crawford:

“Assume that the test is performed on everyone regardless of symptoms of the disease. Then out of every thousand people who receive the test, one has the disease and 999 do not. Further, assume that the test has no false negatives: anyone who actually has the disease gets a positive result. Then 1 out of every thousand tests are true positives. The remaining 999 should be negative results, but the 5% false positive rate means that 49.95 (so round to 50) of these people will receive false positive results. Then out of our 1000 tests, 51 return positive results. But only one of these is a true positive, so the chance that a positive test identified someone who actually has the disease is 1/51 or about 2%.”

You might quibble that 5% false positives means 50 false positive out of a population of 1000 (plus one correct positive) but this is close enough. It’s fair to make the assumption that there are no false negatives since this isn’t stated in the question (and otherwise you’d be unable to answer) but Aswath is right to point out this should have been specified.

Second question: “what percentage of the physicians, residents, and fourth year medical students at a prominent medical school who were asked this question got it right?”

jb guessed that 80% of those tested would give the tempting wrong answer of 95%. Actually, only 19% gave the right answer but only 50% said 95%. jb, you would’ve nailed it if you hadn’t given more detail in your answer than called for. Rob’s an optimist and hoped that 80% would get it right because their care is so important and getting into medical school requires critical thinking. He’s dead right that it’s scary that so many get it wrong.

Interesting answers to third question: “why is it critically important that doctors be able to get this one right? Give one example.” Most not about doctors, though. This type of bad thinking does cover lots of ground.

Matt Crawford cites the Red Cross using an HIV test on donated blood which is known to have a high incidence of false positives and speculates that many donors are probably panicked by the result. “However, the Red Cross continues to use the same test, probably because it combines low cost with very low false negative rate. In this case it may be justified to trade a high false positive rate for a low false negative rate, because a false positive merely requires a second test but a false negative would spread HIV through transfusions.”

Curtis Carmack says: “the medical profession as a whole has given insufficient thought to how to address the false positive issue with patients, leading to much more angst than is necessary when patients receive a positive test result -- invariably late on Friday -- and have to wait at least a couple of days to ask questions about it. ;-)”

Dennis Shanley posts: “This directly effects the overall cost of health care in a huge way. Assume that it costs $10,000 to cure a patient who presents positive. Not an unlikely assumption. Assume further that the 50 false positive patients do not exhibit negative effects as a result of their treatment that require further medical treatment and they do not litigate as a result of the unnecessary treatment. This is a highly improbable assumption made for the sake of simplicity.

“The true cost to cure 1 patient is $10,000.

“The cost to cure that one patient and treat the 50 false positives is $510,000.”

Aswath writes: “Suppose now we are told that the false positive predominantly affects a biological group - gender or a racial group. Will that decision stand reason? Let us assume that the situation is internment during WWII in US. A nation has to live with the effects of a callous operation decision to accept a large false positive.”

Otmar: “There is another interesting application for this kind of statistics: The beloved war on terror. The chance of a random person to be a terrorist is hopefully less than 1/1000. Imagine you manage to build some automated system which somehow claims to spot suspicious behavior, known faces, or miscreants by some other clever scheme.

These systems all have a non-negligible error-rate. If you're really lucky, you might push that one down to less than 1%.

“Now do the math again, assuming a 1/100000 terrorist-rate and 1% false positives. No wonder I read that one trial for such a system got terminated.”

The point is that you must weight the costs of being right and the costs of being wrong both for the positive and the negative case. Back to medicine, suppose your doctor is one of the benighted 81%. He or she tests you using the test in the first question and you come up positive. Let’s suppose that the disease is always fatal if not treated and there’s a treatment available but it has a 25% chance of killing you itself. If the doctor believes that there’s a 95% chance you have the disease, the dangerous treatment is clearly justified; but, since the true likelihood is less than 2%, the treatment is more dangerous than your untreated prognosis. Always a good idea to get a second opinion AND check your doctor’s math.

The crux of the issue is not the poor understanding of probability by doctors. It is the poor understanding (or ambiguous definition) of the term "false positive." If the probability of a false positive is defined (the question implicitly defines it this way) as the probability that the patient tests positive given that the patient is negative for the disease (ie. P(T+ | D-)) then common answer is wrong. However, if false positive is understood as the probability that the person does not have the disease given that the person has tested positive (ie. P(D- | T+), then the common answer is correct. A better version of the puzzle would have explicitly defined "false positive." I would wager that a significant percentage of physicians would still find the wrong answer, but this way the question would be intellectually honest.

Posted by: 'Ken | November 30, 2011 at 06:49 AM

In my opinion while the calculations presented are correct, some of the inferences drawn are not. You state:

“If the doctor believes that there’s a 95% chance you have the disease, the dangerous treatment is clearly justified; but, since the true likelihood is less than 2%, the treatment is more dangerous than your untreated prognosis.” I’ll admit that my probability theory is a little rusty. However, I think under normal, real-world circumstances the above assessment of the “true likelihood” would not be justified. In your concluding paragraph you have introduced “your doctor” into the equation and thereby probably invalidated a critical assumption in Bennett’s/Taleb’s hypothetical situation.

Part of Taleb’s statement of the problem is “People are tested AT RANDOM (my emphasis), regardless of whether they are suspected of having the disease.” That is normally not the case in real life, and appears not to be the case here. As a result, we are probably no longer looking at a random sample. Since we’re talking about a disease that is “always fatal” – let’s call it FD -- I’m guessing we’re looking at a population of people a) who aren’t feeling up to par and as a result have gone to a doctor, and b) whose doctor has examined them and based on a significant amount of data (physical exam, current symptoms, medical history, family history, etc) has concluded that FD is one of a very limited number of likely explanations for what’s going on.

That is WAY far from “at random.”

So, let’s try a different, and probably more realistic, approach to approximating the “true likelihood” that you have FD. I’ll stick with our original population of 1000 random people. On average, one of them has the dreaded FD disease.

1) Let’s assume 90% are feeling great and 10% (probably a high estimate) are feeling poorly and go to the doctor. So, 100 people go to the doctor. As a simplifying assumption, let’s further assume that the probability of a person having FD but feeling great is zero.

2) Let’s assume of the people who go to the doctor 10% (a very high estimate) are diagnosed as potentially having FD and in need of testing. So, 10 people get tested. Again, as a simplifying assumption let’s further assume that the probability of the doctor missing the FD symptoms and therefore not testing a person that actually has FD is zero.

3) Finally, let’s assume that you flunk the test.

Here’s my proposed calculation:

Expected number of afflicted persons in the group of TEN = 1

Expected number of positive test results in the group of ten = 1.5

(=1 real positive, +10 x 5%= 0.5 false positives)

Approximate probability that you have FD = 1/1.5 = 67%

Note: The probability would be higher still if we had assumed fewer people going to the doctor or fewer being diagnosed as potentially having FD.

A note re my simplifying assumptions: They were made only to minimize the calculations. You can make more realistic assumptions if you like. However, unless you assume a) that people RANDOMLY choose when to go to a doctor, and b) doctors RANDOMLY choose patients to be tested for FD, you will always find that the probability that you have FD given that you felt sick enough to go to the doctor, and were chosen to be tested, and tested positive is significantly higher than 2%.

Bottom line. The original analysis is correct only in those situations where people are tested truly at random. In the real world that is rarely the case. I’m not saying you should not get a second opinion in the hypothetical situation presented. However, if the situation is anything like the one I’m postulating, I think it’s better than 50-50 that you’re in deep doodoo.

Posted by: Al Collins | December 15, 2007 at 01:32 PM

Make my comments explicit: I am indicting the medical community (or for that matter any group that devices tests, like legal community) for ensuring one type of error to be zero while not explicitly identifying the necessary increase in the other type of error. On top of that, they advocate aggressive treatment. One may be rational at other times, but it may be difficult to do such trade-off that Tom points out when it comes to life and death situation that too at close quarters. (Isn't this one of the reasons for generally prohibiting a physician to attend to the dear ones?)

For mathematical reasons Fisher has to talk about keeping one error at a low specified level and devising a test that minimizes the other type of error. But when that theory is taught, the students are not taught the implications of the two types of errors and how to balance them.

If I am a bit agitated and incoherent it is because I know a specific instance of this.

Posted by: Aswath | October 08, 2007 at 11:18 AM

it's just a test. so from my point of view, i consider the corrective rate of the test is understandable, because it's just a test by dr.s and professionals, even a friend on positivesingles.com who actually received a false results still understanded it.

i'm not saying it should be or something like this, but i consider the test should be maded more accurate so that less people wouldn't get troulbes just resulted from the test itself!

Posted by: shark | October 08, 2007 at 01:37 AM