A study in Santa Clara, California, suggested 50 times as many as the confirmed number of cases. Reports from New York headlined 2.7 million people in the state having had coronavirus. Siouxsie Wiles on what we need to know to trust the numbers.

A few days ago it was reported that “2.7 million New Yorkers have already had coronavirus”. To put that 2.7 million in context, the Johns Hopkins University Covid-19 tracker records, as I write, 291,996 confirmed cases in New York, with 57,103 people in hospital, and 17,515 people having died.

If the 2.7 million figure is right, that would mean most people with Covid-19 are not being tested or don’t know they’ve had the virus. I saw lots of smart people musing on social media about what this could mean for how many people in Aotearoa New Zealand might have had the virus already. Before I get into the data behind the 2.7 million, let me start with what those reports might mean for us here in New Zealand. Ready?

Nothing. Nada. Zilch. Zip.

Because we went into lockdown really early in the outbreak, we are now what is known as a low incidence country. That means we have had a relatively small number of cases of Covid-19 so far. Another word that is used to describe how common a disease is somewhere is prevalence. Again, because we acted when we did, Covid-19 is not very common here in New Zealand. In all likelihood then, we are a low prevalence country. That’s in contrast to the US and many other countries that still have widespread transmission of the virus and a large number of new cases being reported each day.

Have 2.7 million New Yorkers really had coronavirus?

Back to the 2.7 million figure. The preliminary data this figure is based on comes hot on the heels of several studies (and I use that term loosely) that have surfaced in the last couple of weeks. More about them later. All these studies have used antibody tests that look for our body’s response to the virus to try to get a handle on how many people might have had Covid-19. I’ve written about the antibody tests before, but the main take home message is that the majority of kits on the market at the moment seem to be rubbish.

The 2.7 million appears to have been calculated after an announcement by the governor of New York state, Andrew Cuomo. What he said was that 3,000 samples had been collected from 40 locations in 19 counties, and that these showed a “13.9% infection rate”. And 13.9% of the population of New York state is 2.7 million.

So, the first question to ask is, given how rubbish most of the antibody tests are, how reliable was the one they used? And the second is, do those 3,000 “samples” reflect the wider population of New York state? Let’s start with that second question. The state’s population is a whopping 19.45 million. I asked statistician Thomas Lumley whether 3,000 people could be representative. He’s also written about this “study” (do go read his piece). His answer: yes, if the study was designed well.

But we don’t have the details. Instead what we have is the media reporting that people were tested at supermarkets. I’ve also seen tweets telling people where testing was happening just in case they wanted a test. That’s going to introduce a whole heap of bias. As Thomas wrote: “Even a little bit of over-volunteering by people who have been sick and want reassurance can drive up your estimate to be larger than the truth.”

As for how reliable the test was? Again, that’s hard to say without details of which one they used. I’ve seen mention on social media that they used a brand that has about a 4% false-positive rate. That means that four out of every 100 tests will give a positive result even though they are negative. Four per cent of 3,000 is 120, and 13.9% is 417. So as many of a third of those positive tests could have just been down to a bad test. And just like that we’ve brought the figure down to 2 million New Yorkers.

But this is where using that 13.9% average isn’t useful. In a subsequent tweet, Governor Cuomo showed the positivity rates for different parts of New York state: Long Island at 16.7%, New York City at 21.2%, Westchester/Rockland at 11.7%, and the rest of the state at 3.6%. Now, knowing the false-positive rate is really important. It would suggest that in New York City there are lots of people with Covid-19 that aren’t getting tested. But not so much in the bits of the state that are at 3.6%. They are just in the “noise” of the test. Of course, knowing the false-negative rate is also important – that’s the number of tests that are actually positive but come up as negative.

The ‘Santa Clara’ study

I mentioned before that the New York study came hot on the heels of a couple of other studies. One of those is what has become known as the Santa Clara study. This one’s a doozy. In early April, a group of researchers led by Jay Bhattacharya from Stanford University’s School of Medicine used antibody tests on 3,330 people from Santa Clara County in California. They’ve made a non-peer reviewed preprint of their paper available online. They conclude that by early April, there were “50-85-fold more” cases of Covid-19 than the number of confirmed cases.

Scientists have taken to social media to peer review the study. They have not held back. One of the main criticisms has been of the way those 3,330 people were recruited. In the paper, the authors describe how participants “were recruited using Facebook ads targeting a representative sample of the county by demographic and geographic characteristics”. As with the New York study, there is the potential for people who think they’ve been infected to respond, which then may end up not being representative. The authors don’t provide in the paper anything like a breakdown of the ages of the participants or their socioeconomic status for us to judge if they are in any way representative.

But it gets worse. According to an article by reporter Stephanie Lee, Professor Bhattacharya’s wife sent out an email to the parents of a high school in Los Altos inviting people to sign up for her husband’s study. In the email she claimed the results of the “FDA approved” test would tell participants if they were immune to the virus and so they would “no longer need to quarantine” and could “return to work without fear”. Umm, I didn’t see any mention of that in their paper!

As for the “FDA approved” test? Not approved. And as for the false-positive rate? In their paper they report that the specificity of the test was 99.5% (95 CI 98.1-99.9%), which means that they are 95% confident that a negative sample will be correctly identified as negative 98.1-99.9% of the time. In other words, anywhere between three and 63 of their negative samples could have been falsely identified as positive. Do you want to know how many of their 3,330 samples were identified as positive? 50. In other words, in the “noise”.

So, were there “50-85-fold more” cases of Covid-19 than the number of confirmed cases in Santa Clara County by early April? Impossible to say. Interestingly, one of the co-authors of the study is someone who has been very prominent in the media with his views that the dangers of Covid-19 are being completely overhyped.

Why are antibody tests so important?

At the moment, well-designed serology studies using reliable and accurate tests are absolutely vital for us to understand more about Covid-19, from how vulnerable different communities remain to the virus, to how deadly the disease really is, and whether lots of people really do just have a mild infection. The finding that many people have already been infected will help officials make decisions about how safe it is to ease lockdowns and other restrictions.

In places where Covid-19 is widespread, there is also talk of people having “immunity passports” – something that shows they’ve had the virus and are safe to go back to work or travel. Of course, we don’t yet know if people who’ve had the virus develop strong lasting immunity, so it’s far too soon to responsibly pursue that.

Such a passport may well end up being a reality in some countries. It might be the way we can open up to tourists again. But it will depend on there being a reliable and accurate test. And as we’ve seen, there are plenty of companies happy to flood the market with crappy tests.