Feeling a little lost about home testing for Covid-19? If you’re not, either you are an expert or you aren’t paying attention. I say this because there is no possible way to interpret the results of this kind of medical test without knowing three numbers, all of which are at best hard to find. Once you do find them, the implications are underwhelming, to say the least, as I’ll explain below.
This isn’t some pedantic technicality–tests truly don’t mean jack if you don’t know those numbers. The best you can do without them is to go by some simple rule of thumb like positive=quarantine. That’s generally fine for positive results because if you get a false positive, all it means is a few unnecessary days alone watching Netflix. Unfortunately, not knowing what a negative result means gets people killed.
It’s bizarre that two years into the pandemic, this is so little understood either by the public or by the people who present the news. The public health people certainly understand, because it’s basic medical statistics, but mostly continue to speak to the public as if this were not a thing. At the risk of seeming cynical, testing is a very positive sounding thing to talk about at a time when the public health authorities have conspicuously failed to cover themselves with glory. It took almost two years to make rapid testing readily available. After all that, it’s only natural to want it seen in the best light.
There are a few numbers involved, but it doesn’t much math to get the gist of it. Percentages and multiplication, basically. If you are at risk, or you’ve got anyone in your orbit who is especially at risk, such as your aging Granny, it’s definitely worth reading this.
If the TLA “PPV” means anything to you, feel free to skip to the section The Numbers for At-Home Tests near the bottom–you’ll probably know everything between here and there.
Test Results Don’t Mean What You Probably Think
All statistics students and most doctors will have run into these ideas at some point, but I suspect that the average family doctor would need a quick refresher if they ever had to apply it, because the situations where you need it mostly come up after the case has already been booted up to a specialist. The average GP probably remembers this about as well as you remember trigonometry.
There Is No Such Thing As Test Accuracy
You see everywhere that such and such a test is some percent accurate, but in fact, there is literally no such thing as “test accuracy.” If someone tells you a test is X% accurate, either they don’t know what they are talking about or they do, and they are afraid that the full story would just confuse you.
The mathematical fact is that any meaningful statement about the accuracy of a medical test needs at least two numbers, and usually three. So universally disregarded is this truth that it takes some effort to even find out what the three numbers are for Covid tests. They’re out there, but you have to know what to look for and what it means.
I encourage the reader to sit still for the explanation–there’s not much math in it–It’s more about words. The first two numbers have such exact and non-intuitive meanings that you’d think they were dreamed up by a lawyer.
Sensitivity is the first number. A sensitivity score of X means that when you test 1000 people who you know for a fact have the disease, on average, X percent will test positive. Say X = 99% (which would be very good.) If you test 1000 people who have a disease, 990 will test positive and 1%, i.e 10 of them will falsely test negative.
That sounds great, but there’s a catch. Any number of people who don’t have it can also test positive and it has no effect on the sensitivity number. These are called false positives.
The second number is specificity. A specificity of Y means that if we test 1000 people who definitely do not have the disease, Y percent will in fact test negative. So if it’s highly specific, almost everyone who doesn’t have it will test negative, but it doesn’t say anything about how many who do have it will also test negative.
You can see that high specificity tends to squelch false positives because if they don’t have it, they test negative.. And the opposite is true too–high sensitivity tends to squelch false negatives, because if they do have it, they test positive.
A test that’s strong in both is great–high sensitivity catches false negatives and high specificity catches false positives, leaving only mostly the true negatives and true positives.
Unfortunately, the two numbers can be very different for the same test and neither is usually 99%. For instance, imagine a test that is very sensitive, i.e., it catches almost every case, but not very specific, i.e., it can also have lots of false positives. If you get a negative result for a test like that, great! Why? Because the high sensitivity rules out most false negatives leaving only the true negatives. The reverse is the case for a positive result. The test isn’t very specific, so it’s bad at ruling out false positives. Therefore, a positive result doesn’t tell you much, except that that the disease hasn’t been ruled out.
Some tests are the other way around. Not very sensitive but highly specific. So if you get a positive result for a test like that, it has a high chance of being accurate, because the high specificity squelches most of the false positives, leaving the true ones. So if you did get a positive, it’s most likely true. On the other hand, the negative results are not to be relied upon because low sensitivity does a bad job of canceling out false negatives. If such a test says you have it, you probably do, but if it says you don’t, you can’t rely on it.
To sum up,
- High sensitivity and high specificity gives good results both ways.
- High sensitivity and low specificity means gives reliable negatives but positive result doesn’t tell you much.
- Low sensitivity and high specificity gives reliable positives, but unreliable negatives.
The Pesky Third Number
Actually, it’s not that simple, because I glossed over the third number which is the density of actual cases in the population of concern. This is the giant gotcha in interpreting test results.
Say Dr Bob has a super good test for a hideously awful, incurable disease. The test is 99% sensitive, i.e. and 99% specific. As good as it gets, basically. Fortunately, the disease it tests for is extremely rare; only one in a million Americans have it. (That makes it a little more common than leprosy, aka Hansen’s disease.)
Now say the over-zealous Doctor Bob decides to just randomly give the test to Alice, and she happens to test positive. Alice is screwed, right? She got a positive test with a 1% false positive rate so I guess it’s time for her to subscribe to the Hemlock Society YouTube Chanel.
Certainly not! The key here is that Dr. Bob just randomly picked Alice without any reason to think she has it. The test result has 99.99% probability of being a false positive.
Let’s see how it works. If you gave the test to all 325 million Americans, you would get 3.25 million false positives, i.e. 1%. But it actually affects only one person in a million, so only be about 325 people in the US actually have it. Of them, you would expect about 321 would test positive because the sensitivity is 99%. In the absence of any reason to think she Alice has it, the probability that the result is valid is the ratio of the number of true positives to the number of all positives, both true and false.
This is a really tiny number. 325/(3,250,000 + 321) = 0.0001, which is one in ten thousand. So even with a positive result from an excellent test, Alice is less likely to have the disease than she is to die in a car accident this year, a risk so small she rarely even thinks about it.
This calculation is the essence of what statisticians call “Positive Predictive Value” or PPV.
The Numbers for At-Home Tests
So what does that say about home tests? There are five at-home tests for Covid on the market. One is strikingly worse than the other four, with a sensitivity of 34.1 and a specificity of 88.1. The rest have sensitivities of between 44% and 54% and they all have specificity of either 100% or very close. (Which is phenomenal.)
Let’s toss the outlier and round the others four to 50% and 100%.
As we saw above, the meaning can depend heavily upon that third number, which is the percentage of cases in the population. In the example, the expected number of false positives completely swamped the number of true positives, giving it a PPV of almost 0.
However, in this case, the specificity is almost perfect, which means there are zero false positives. The PPV is the number of true positives divided by the total number of all positives, true and false. With zero false positives, you get a PPV of 1.0 (which is freakishly good.) This means that the probability that a positive result means you have Covid is 100%.
That’s nice, but who cares? Even with a less reliable result, you would still going to have to quarantine as it it were, so what does that perfection really get you?
What we really care about are the false negatives. A negative result, true or false, is what gets you on the plane or gets you a seat at the table with all of your elderly, obese, or immune compromised relatives. So what does it say about negative tests?
Just glancing at the numbers tells the story. All the test promises is that if you’re positive, then you definitely have it. The test caught only half the people who have it–the rest all got false negatives. Low sensitivity means low protection from false negatives.
We know that half the cases went uncaught, so if you got a negative result, all it means is that the probability that you have Covid is about half of what you would have estimated the probability was if you didn’t take a test at all. Not a very impressive result, to say the least.
Are The Test Kits Going To Protect Granny?
If you get a positive result, great, it’s nice to know. Wait, I mean sorry–you definitely have Covid. It’s only great if you’re into statistics and statistically speaking, you probably are not. But random people taking the test will overwhelmingly get negative results, primarily because most people don’t have it, but also because about half of those who do have it test negative anyway.
I don’t have any idea what percentage of random people in my area who have no special reason to suspect that they have Covid, have it. I doubt anyone knows. So how is a negative result useful to me, if all is says is that the probability that I have Covid is 50% less than damn-if-I-know?
I’m not even sure if making them generally available is a good idea. The problem is, while a negative result makes it 50% less likely that you have Covid, nobody really knows how much more likely the false sense of security makes people to do things that spread Covid.
The bottom line is, a negative at-home test result is only somewhat more reliable than crossing your fingers. Do it if you want, but it’s no substitute for having everyone who comes in contact with people who are at risk being vaccinated and boosted to the max.