Two widely followed surveys “significantly overestimated” how many American adults got their first dose of a Covid-19 vaccine last spring, says an article in the scientific journal Nature that was published online on Wednesday.
The surveys have large numbers of participants but nevertheless aren’t representative of the U.S. adult population, the article says. This flaw is a symptom of a bigger problem, which is the belief that asking lots of people to respond to a survey can make up for deficiencies in its design. It’s what one of the Nature authors, the statistician Xiao-Li Meng, calls the big data paradox. “The more the data, the surer we fool ourselves,” he wrote in 2018 in The Annals of Applied Statistics.
The Nature article analyzes two surveys of Covid-19 vaccine uptake: One is the Delphi-Facebook survey, which gathers about 250,000 responses a week and which its backers describe as unmatched in “detail and scale” for a public health emergency. The other is the Census Bureau’s Household Pulse Survey, which gathers about 75,000 responses every two weeks. The article says that earlier this year the Delphi-Facebook survey overestimated the uptake of Covid-19 vaccines by 17 percentage points (70 percent versus the actual 53 percent), while the Census Bureau’s Household Pulse Survey overestimated the uptake by about 14 percentage points.
Those are big misses, far bigger than the surveys’ own estimates of their margins of error. The reason we know the surveys were wrong is that we can compare their estimates for Jan. 9 to May 19 of this year with verified records of vaccinations for that period that were published on May 26 by the Centers for Disease Control and Prevention.
A much smaller survey conducted by Axios and Ipsos was off by only 4 percentage points, the Nature article says. That survey is based on an online panel with only about 1,000 responses per week, but it uses best practices for obtaining a representative cross-section.
By one measure the Delphi-Facebook survey produced results that are no better than a simple random sample of just 10 people, the authors of the Nature article write. “Our central message,” they conclude, “is that data quality matters more than data quantity, and that compensating the former with the latter is a mathematically provable losing proposition.”
A team of six scholars from Harvard University and the University of Oxford did the research. They are a mix of statisticians, political scientists and computer scientists. The lead authors are Valerie Bradley of Oxford and Shiro Kuriwaki, formerly of Harvard, now an incoming assistant professor at Yale. The senior team members are Seth Flaxman of Oxford and Meng of Harvard. The other two authors are Michael Isakov of Harvard and Dino Sejdinovic of Oxford.
Meng has been calling attention to the big data paradox for years. I wrote about his work earlier this year when I was working for Bloomberg Businessweek. The intuition is that if you solicit opinions about Taylor Swift while you’re at one of her concerts, you won’t be getting a good read on overall opinion. As I wrote:
Both the Delphi-Facebook survey and the Census Bureau’s Household Pulse Survey have been widely quoted in the news and cited in academic research. The Delphi Research Group, a research team at Carnegie Mellon University that collaborates with Facebook, has a web page with links to 15 publications, including ones in journals such as Science and Lancet Digital Health. (There are ways for scholars to make valid use of imperfect data, such as re-weighting it to better approximate the makeup of the population, but they need to tread carefully, the authors say.)
I reached out to the Delphi-Facebook and Census Bureau teams for their responses to the Nature article. A Census Bureau spokeswoman directed me to the bureau’s description of the Household Pulse Survey, which says that it “is designed to deploy quickly and efficiently” but is experimental. The page adds, “Census Bureau experimental data may not meet all of our quality standards.”
The Delphi-Facebook team argues in a written response that its survey “in fact performs well for its intended goals.” Alex Reinhart, an assistant teaching professor of statistics and data science at Carnegie Mellon who is the university’s principal investigator on the survey, said in an interview that it was put together at the request of the C.D.C. to meet an urgent need for data early in the pandemic. He said it was pretty good at detecting changes in infection rates and vaccination rates in local areas, but not so good at getting the levels right. (Although lately the problem has been different: Delphi-Facebook surveys have shown little change in self-reported vaccination rates even though actual vaccination rates have risen.)
“Official data reporting has gotten a lot better since early 2020,” Reinhart said. “And some of the goals have changed. Forecasting seems less important. It’s not obvious exactly what we should focus on next as the pandemic winds down. Maybe decrease the size of the survey, focus it more narrowly or discontinue it.”
I also exchanged emails with the public health scholars Phoenix Do of the University of Wisconsin-Milwaukee and Reanne Frank of Ohio State University, who have used the Delphi-Facebook data in their work. They wrote that they “adjusted for metropolitan status, political affiliation and education — factors that were known to be nonrepresentative in the Delphi-Facebook data.” They added that “there is currently a dearth of data sources” on socioeconomic factors related to Covid-19: “The Delphi-Facebook data has that needed information; moreover, it has been released in near real-time, enabling researchers such as ourselves to investigate these types of questions in a timely manner.”
The readers write
A friend recently ordered new kitchen appliances even though he is not ready to remodel “because the longer I wait the higher the price.” He will place the unboxed appliances in his garage for a year or more. We are not yet where people in the Weimar Republic were, where they insisted on being paid in the morning so they could spend the money at lunchtime, but the same psychology is at work.
John Edwards
Sarasota, Fla.
Quote of the day
“The Times 03/Jan/2009 Chancellor on brink of second bailout for banks”
— A phrase embedded in the first block of the Bitcoin blockchain, 2009
Have feedback? Send a note to [email protected].
