The integrity of science is suffering as scientists rush to save the world from the pandemic driven by the SARS-CoV-2 virus.
Or so say scientists from Switzerland and Slovenia who examined the evidence contained in 559 studies – 204 of COVID-19, the disease caused by SARS-CoV-2, and 355 non-COVID controls – published in what are considered the world’s top medical journals earlier this year.
“We conclude that the quality of COVID-19 publications in the three highest-ranked scientific medical journals is below the quality average of these journals,” they reported in a peer-reviewed study at Plos One.
The researchers examined studies published in the New England Journal of Medicine, the Journal of the American Medical Association (JAMA), and The Lancet, a British Publication, from March 12 to April 12.
In their paper, they noted that “while our article was under review, two major analyses on the use of hydroxychloroquine and cardiovascular mortality associated with COVID-19 were retracted in the Lancet and the New England Journal of Medicine because source data could not be verified.
“Such situations raise concerns as to the quality of the data, the conclusions presented by the authors, and the peer review by the editors, due to the pressure to publish highly coveted information on COVID-19. The urgency of the outbreak suddenly appears to legitimize key limitations of studies, such as small sample sizes, lack of randomization or blinding, and unvalidated surrogate endpoints.”
Rush to judgment
Small sample sizes have been a problem plaguing many early COVID-19 studies, and the issue came up again this week in the wake of an announcement by pharmaceutical giant Pfizer that testing of its experimental vaccine had shown it 90 percent effective.
“The 90 percent efficacy they report we presume to mean that of the 94 cases of COVID-19 that were recorded in the trial participants, 84 of those cases were in people who were administered the placebo (so didn’t get the vaccine),” Dr. Larisa Labzin from the Institute for Molecular Bioscience at The University of Queensland subsequently wrote at Scimex, a website for science news in New Zealand and Australia.
“There is still a long way to go however to determine how effective this vaccine is across the 43,000 participants who were enrolled (in the study), as 94 cases is a small proportion of that.”
To say that there is a widespread desire for a vaccine to help prevent a deadly disease for which treatments are limited might be an understatement. As a result, some of the rules of science are getting bent.
The Russians began inoculating some of their citizens with the so-called Sputnik V vaccine in September even before completing final trials.
“We have no idea whether this vaccine is safe or whether it works”, Ashish Jha, Dean of the Brown University School of Public Health told The Lancet at the time. “It is really worrying when people start to bypass the standard process we have for vaccine development.”
The Russians pushed back with the claim Western politicians and media were trying to “undermine the credibility of the Russian vaccine,” and on the Sputnik Vaccine website posted an op-ed – “rejected by all leading Western media” – explaining the vaccine’s development and effectiveness.
If nothing else, the Russian op-ed served to illustrate the tangle of politics and science in which COVID-19 has become enmeshed.
In the West, the biggest issue has arisen around face masks opposed by some civil libertarians. Arguments between maskers and anti-maskers have hampered a public discussion of what masks can and cannot do to help control the spread of SARS-CoV-2.
The argument has now reached the point where a long-anticipated, randomized controlled trial (RCT) of public masking conducted in Denmark can’t seem to find a publisher. One of the country’s largest newspaper reported the authors are having trouble getting a peer-reviewed journal to accept the study because the conclusions are “controversial.”
RCTs are generally considered the gold standard of medical research. RCTs of masks are sadly lacking as the Center for Infectious Disease Research and Policy (CIDRAP) at the University of Minnesota, a highly respected research center, pointed out early in the pandemic. It subsequently came under fire as “anti-mask” and was forced to defend its position.
CIDRAP stood by its initial report that mask effectiveness remains an unknown, but its website now notes that the center supports “the wearing of face coverings by the public when mandated and when in close contact with people whose infection status they don’t know. However, we also encourage everyone to continue to limit their time spent indoors near potentially infectious people and to not count on or expect a cloth mask or face covering to protect them or the people around them.”
The U.S. Centers for Disease Control this week issued a new “Scientific Brief” supporting masking based on “experimental and epidemiological data” that indicates the “prevention benefit of masking is derived from the combination of source control and personal protection for the mask wearer. The relationship between source control and personal protection is likely complementary and possibly synergistic, so that individual benefit increases with increasing community mask use.
“Adopting universal masking policies can help avert future lockdowns, especially if combined with other non-pharmaceutical interventions such as social distancing, hand hygiene, and adequate ventilation.”
The brief did not define “adequate ventilation.” It was not peer-reviewed; most government reports aren’t. Among the sources it cited as supporting masking was a study of an outbreak aboard the aircraft carrier Theodore Roosevelt.
The brief said that study “found that use of face coverings on-board was associated with a 70 percent reduced risk.” The brief did not mention the study’s significant, self-noted limitation: It was “conducted on a convenience sample of persons who might have had a higher likelihood of exposure, and all information was based on self-report, raising the possibility of selection and recall biases.”
A separate, peer-reviewed study of the Roosevelt outbreak was published in The New England Journal of Medicine. That study did not mention masking, but did observe that “members of the medical department, who wore personal protective equipment (PPE) when evaluating crew members, had a somewhat lower attack rate (16.7 percent) than the overall crew, despite being at highest risk as a result of exposure to patients with Covid-19 in a small space.”
Masks are normal PPE for medical personal dealing with people with infectious diseases. The overall attack rate on the ship was 26.6 percent, but individual attack rates varied. The NEJM study underlined the importance of ventilation.
“Not surprisingly,” it said, “crew members working in the engine room and other confined areas of the ship faced a higher risk of being infected than their shipmates on deck.” The infection rate was also higher among enlisted personnel than among officers.
“Typically, enlisted crew members sleep in open bays packed with dozens of tightly spaced bunks, work in densely populated areas, and congregate in gathering points such as the gyms and galleys,” the study said. “These conditions probably facilitated the transmission of SARS-CoV-2, as evidenced by the higher likelihood of Covid-19 among enlisted crew members than among officers.”
Modern buildings are designed to be airtight to reduce the energy need for heating and cooling. Reducing energy use minimizes the production of the greenhouse gas carbon dioxide and saves money, but creates a closed environment that can hold viruses.
The importance of building ventilation during a pandemic caused by a respiratory virus has been known since the Spanish flu more than 100 years ago, but it wasn’t until this week that the U.S. Occupational Safety and Health Administration (OSHA) issued an alert providing “guidance on ventilation in the workplace.”
Among other things, it recommended opening “windows or other sources of fresh air where possible.” Many modern office buildings lack windows that open. They are designed that way to make them more energy-efficient.
The U.S. Energy Star program estimates energy-efficiency efforts helped save U.S. commercial buildings $10 billion in 2016. Leaving doors and windows opens on commercial buildings and apartment complexes could undermine those savings and drive up costs.
Ventilation, social distancing and masks comprise a complicated mix of so-called nonpharmaceutical interventions hard to disentangle. A study of eight NPIs used in 41 countries from January to the end of May reached the conclusion that masks might actually make the pandemic slightly worse, but the Havard and Oxford university’s researchers leading that study said it was possible there are situations in which masking could help.
As with the scientists at CIDRAP, those researchers were not opposed to masking and didn’t encourage anyone to ignore masking mandates. They just said their research did not support it as an effective tool, possibly because it changed human behaviors that put people more at risk.
They’re non-peer reviewed study was posted on MedRxiv and received little media attention. Similar views expressed by Michael Osterholm, the CIDRAP director, received much more attention. They came under so much fire he felt forced to defend himself online.
“I’ve received increasing criticism in recent weeks because I’ve offered more nuanced messaging on whether everyone should wear cloth face coverings in public to protect against COVID-19 transmission – messaging that some view as unacceptable,” he wrote.
Osterholm made it clear he wasn’t against masks. “I wear one myself on the limited occasions I’m out in public. In areas where face coverings are mandated, I expect the public to follow the mandate and wear them,” he wrote.
But he warned of a dangerous “message creep” that could lead people to look at masks as a cure.
“We need to be clear that cloth face coverings are one tool we have to fight the pandemic, but they alone will not end it,” he wrote. “And we need to underscore the key role that physical distancing plays – even when you wear a face covering.”
Since Osterholm wrote those words, a New York University pilot study has reported almost a quarter of the transit workers in the nation’s largest city have come down with COVID-19, despite a mandatory mask requirement in place since April.
Those findings have, in turn, been challenged by the American Public Transportation Association (APTA) which argues that “leading scientists emphasize that mask wearing may be the most important way to reduce the transmission of COVID-19.”
All of which might best serve to underline Osterholm’s most astute observation: “Science, when done well, can be messy, imperfect, and slower than we wish. And it’s ever-evolving.”
And that is the essence of the argument for subjecting scientific studies to thorough and aggressive peer-review.
For whatever reason, the scientists involved in the Plos One study indicated that is not happening with COVID-19 research. In general, they said, when COVID studies were weighed against non-COVID studies, the former were found to have gone to press with far less evidence in hand than the latter.
The researchers voiced the hope the situation would improve as the pandemic progresses, but also admitted to potential problems even bigger than those found by their examination.
“We did not expand our analysis to check source data,” they wrote. “The data scandal leading to retraction of two major studies emerged while our article was under peer-review. The tools we used would not be suitable to have detected this.”
They then offered something of a backhanded compliment to preprint servers – such as Medrix – where much SARS-CoV-2 research is now being published sans peer-review.
“Public data repositories and an ‘open science’ approach may facilitate data validation,” they said.
The pros and cons of open research are now much debated. MedRxiv (pronounced med-archive) co-founders Dr. Harlan Krumholz and Dr. Joseph S. Ross argue the “research revolution will not be peer-reviewed” while scientists at Stanford University warn against “the danger of un-vetted science seeding mainstream media stories with deleterious results.”
Unvetted science seeding mainstream media would be the interface where bad science meets what has come to be called “fake news” although, as with the bad science, not so much is fake – as in made up – as it is distorted, contrived or simply unsupported by the evidence.
As Toby Hopp, a University of Colorado Boulder professor studying the subject has observed, “we tend to call it ‘fake’ news, (but) a lot of this stuff is not completely false. Rather, it is grossly biased, misleading and hyper-partisan, omitting important information.”
Fake news and bad science share a common trait. Both tend to paint black and white a world full of grays to justify their conclusions. Unfortunately science, especially medical science, is seldom black and white, which is why drugs have long been tested against placebos.
Almost everyone has heard of the “placebo effect,” which can cause an inert sugar pill to work as a pain killer. No one, as of yet, knows what mechanism or mechanisms make that happen.
“While the placebo effect has been observed in many scenarios, there’s still a lot about it that we don’t understand,” microbiologist Jill Seladi-Schulman writes at Healthline. “One of the big questions is the connection between mind and body. How are psychological factors like expectations affecting what’s going on inside us?
“We know that the placebo effect can lead to the release of various small molecules like neurotransmitters and hormones. These can then interact with other parts of the body to cause changes. However, we still need to work out more details about the specifics of these complex interactions.”
In science, as in journalism, it is easy to ignore all the confounders in order to make a point or support a conclusion. This problem gets especially tricky in medicine where lives are often at risk and people want treatments to be simple and straightforward even if nature makes that impossible as it regularly does.
It has, for instance, long been known different people react differently to different drugs. Differences in male and female responses are well documented, and in recent years there has been an increasing focus on genetic differences that alter how drugs work on individuals.
Researchers are now debating how genetics influence individual responses to the SARS-CoV-2 virus. There are suggestions that type O blood could reduce the risks of infection with the virus manifesting itself as the disease COVID-19. There are indications that a genetic variation on a chromosome that traces back to the Neanderthals with whom the ancestors of some of today’s humans interbred 50,000 years ago could make modern humans more vulnerable to the disease.
There is a lot of interest in these discoveries because of the significant number of people infected with SARS-CoV-2 who don’t get sick. Whether the number of these so-called “asymptomatic people” is relatively small or relatively large varies from study to study.
Ontario, Canada, researchers who went looking for SARS-CoV-2 in blood samples collected by Canadian Blood Services in that province in May 2020, this week reported an “estimated…5.88 infections occurred for every case identified,” an indication that a lot of Canadians caught the disease and were either asymptomatic or suffering so little they might have thought they only had a common cold.
Other studies have reported as few as one in five infections went unidentified. Science is messy.
The Canadian study was published on MedRxiv and was not peer-reviewed. Peer-review would surely have lent more significance to the study. Whether it would have altered the conclusions is unknown.
For every scientist arguing the good of peer review, there is likely one arguing the opposite with the majority probably somewhere between. Dozens of scientists interviewed in the past several months have uniformly expressed both their belief in peer-review and then confessed that they know it can be “gamed.”
“Sadly, very often journals publish manuscripts that do not use correct (logical) approaches because not all scientists are great and reviewers are often lazy, too busy, don’t care, unlogical, or don’t know enough,” one confessed in an officially off-the-record communication. “(And)
if you’re rejected, you just go to a different journal.”
A messy world
For better or worse, this all adds to the confusion in a world influenced by politics, economics, the speed of communication, and the psychology of the inherently fearful human animal in need of talismans to hold back those fears.
There are reasons humans created so many religions throughout history.
Science evolved against this historical background of Gods and demons not as a new faith, but as a method for finding answers to how the world works free of faith. Science grounded itself in the concept of demonstrable and reproducible results.
The ball goes up, the ball comes down. The ball goes up; the ball comes down. The ball goes up; the ball comes down. Voilà, there is a force at work on the ball, a force that came to be called gravity.
But not all science produces demonstrable and reproducible results. Some science operates largely in theory because it is hard to design the experiments the produce testable results or because the required experiments are deemed unethical. And some science produces results open to all sorts of interpretations.
When politics and economics get mixed in with the latter, things get real messy as Alaskans know well from the sometimes dueling studies of government scientists and oil industry scientists in the wake of the Exxon Valdez oil spill in Prince William Sound in 1989.
In the years after – as litigation against cigarette companies working its way through the courts revealed company-paid scientists who claimed the product was safe knew it could increase the risks of some developing lung cancer – the economics of scientific research has become an even bigger issue. Almost all publishers of scientific reports now require authors disclose their financial connections.
There is no such disclosure required for political biases, and that can cause problems when scientists go political. Kamran Abbasi, the executive editor The BMJ – formerly the British Medical Journal – warned against these entanglements in a scathing editorial on Friday headlined “Covid-19: politicisation, ‘corruption,’ and suppression of science.”
As have other scientists of late, he noted the risks to the credibility of science when scientists move beyond the bounds of cold, hard evidence to join political and/or economic scrums.
Some of the scientists have self-serving agendas. Others are driven only by the best of desires to do good, but as an old English proverb says, “The road to hell is paved with good intentions.”
And with money.
“The UK’s pandemic response relies too heavily on scientists and other government appointees with worrying competing interests, including shareholdings in companies that manufacture covid-19 diagnostic tests, treatments, and vaccines,” Abbasi wrote. “Government appointees are able to ignore or cherry-pick science – another form of misuse – and indulge in anti-competitive practices that favor their own products and those of friends and associates.”
The editorial was sparked in part by BMJ’s discovery that the British government blocked a study of a rapid diagnostic test that was the cornerstone of the country’s Operation Moonshot anti-COVID program. The study found the test fell “well short of performance claims made by its manufacturers.”
Abbasi noted the complex problem here in observing that “politicians often claim to follow the science, but that is a misleading oversimplification. Science is rarely absolute. It rarely applies to every setting or every population. It doesn’t make sense to slavishly follow science or evidence.
“A better approach is for politicians, the publicly appointed decision-makers, to be informed and guided by science when they decide policy for their public. But even that approach retains public and professional trust only if science is available for scrutiny and free of political interference, and if the system is transparent and not compromised by conflicts of interest.”
A transparent system free of conflicts of interest is a pretty big ask, especially in an increasingly tribalized world where political leanings seem to have crept into every corner of society including science and journalism. Probably the best way to protect science is for scientists to try their best to ignore politics and focus on the science.
But that is easier said than done if you believe you’ve discovered the information that could save the world. It is hard to trust to having that vetted by the fickle politics of any democracy. The alternative might, however, be worse.
“Politicisation of science was enthusiastically deployed by some of history’s worst autocrats and dictators, and it is now regrettably commonplace in democracies,” Abbasi wrote, and scientists were sadly in the mix.
Some have argued the U.S.-led eugenics movement begun in the 1900s with the good intentions of reducing human suffering by “breeding out” disease, disabilities, and undesirable characteristics set the stage for Adolph Hitlers “final solution” to kill every last Jew in Europe.
“As the concept of eugenics took hold (in the U.S.), prominent citizens, scientists and socialists championed the cause and established the Eugenics Record Office, the History website records. “The Eugenics Record Office…maintained there was clear evidence that supposed negative family traits were caused by bad genes, not racism, economics or the social views of the time.”
Forced sterilizations, led by the state of California, soon followed in an effort to rid society of bad genes. Hitler referenced this in Mein Kampf, his autobiographical, political manifesto, and subsequently tried to eliminate a cultural group that has included some of the modern world’s greatest scientists and thinkers.