Do’s and Don’ts Regarding How To Assess Scientific Studies in the Age of the Internet and Social Media

A positive aspect of the advent of the internet is that scientific studies can be made public as soon as they are ready to be published. However, these studies are highly technical publications that are intended for scientists to study and analyze. Thus, one negative effect of greater accessibility to the scientific literature is that individuals without the education and technical knowledge necessary to evaluate the studies can now gain access to them. As a result, these individuals may disseminate in their blogs, podcasts, social media, and other outlets erroneous claims about these studies either because they misunderstood them, or because they may have an agenda directed at favoring certain interpretations of the studies, even if these interpretations are not supported by the data.
 
I have lost track of the number of times I have seen someone on Twitter making claims about some issue by citing the latest published scientific article. Invariably the purpose of the individuals making these claims is not to discover or debate the truth, but rather to support their political or social agendas. I have tried to explain that truth in science is not established by one or even a few studies, even if they are published in peer-reviewed journals. Scientists have to debate the merits and flaws of each other’s studies, and this is a process that will take time. During this process scientists may make claims that they may later recant when more evidence becomes available, or a study that was heralded as a good study may fall in disfavor if it is realized that certain variables that turn out to be important were not controlled. But when scientists do these normal things that are part of the scientific process, they are accused of flip-flopping or selling out to special interests.
 
The above process is amplified by various types of media which reach millions of people and contribute to create confusion and suspicion when people see narratives change. I saw this happen with hydroxychloroquine. A study would come out indicating hydroxychloroquine (HCQ) had an effect against COVID-19, and all the HCQ proponents would brag about how the issue was settled and HCQ worked. Then another study would come out showing that HCQ did not have an effect, and all the HCQ critics would claim HCQ did not work. In the middle of the storm, certain responsible scientists or organizations would comment about the studies pointing out flaws or strengths, and they would be denounced by the pro or con side. Eventually enough studies accumulated, and they showed not only that HCQ does not work against COVID-19, but also why it does not work. However, by then HCQ had lost its appeal as a political issue.
 
I have seen this happening again with the drug ivermectin. A study came out of Brazil using a population of 88,012 subjects where ivermectin brought about a reduction of 92% in COVID-19 mortality rate. The pro-ivermectin crowd declared victory, bragged about how they had been right all along, and pointed out that the withholding of ivermectin had led to many preventable deaths from COVID-19. The truth, however, was very different. This was an observational study where the allocation of patients to treatments was not randomized, which can lead to serious biases in the data. And while a sample size of 88,012 subjects sounds impressive, the actual comparisons were performed on much smaller subsets. For example, the 92% result came from comparing 283 ivermectin users to 283 non-users. Additional problems involved the exclusion of a large number of subjects and the non-control of ivermectin use. Finally, there is no way that an effect of such a large magnitude (92%) would not have been detected when performing better designed and controlled trials, but that has not been the case.
 
As I have pointed out before, the politization of science creates a caustic environment where the work of scientists is mischaracterized or attacked by unscrupulous individuals, and this makes the process of science much harder than it already is.
 
To avoid all the problems mentioned above, I have put together a list of do’s and don’ts regarding how to assess scientific studies in the age of the internet and social media.
 
1) Do listen to what scientists have to say about the studies. They are experts in their field, and an expert is called that for a reason. They have studied many years and trained to do what they are doing. Do not assume you know more than the experts. Do not merely quote a study in your blog or social media to defend a position. Rather, do report on the debate among scientists regarding the strengths and weaknesses of the study and identify unanswered questions.
 
2) Do give scientists the time to evaluate and debate the studies and to reassess the studies as more information becomes available. Do not attack scientists for changing their minds.
 
3) To make up your mind, do wait for several studies to accumulate and for the majority of scientists to reach a consensus regarding the studies. However, this consensus will not be arrived at based on the total number of studies, but rather on their quality. One study of good quality can be more meaningful than dozens of low-quality studies, and the community of scientists (not a single scientist) is the ultimate arbiter regarding the quality of the studies.
 
4) Do not judge scientists or the results from their studies by their affiliations to companies or other organizations. The studies have to be judged on their merits. Do not make offhand claims that conflicts of interest have corrupted the science if you don’t have any evidence for it. Hearsay, innuendo, and ignorance are not proof of anything.
 
5) Success in science is measured by the ability of scientists to convince their peers. Scientists who have been unable to convince their peers and who bypass the normal scientific process to take their case to “the people” are a huge red flag. Do not blindly trust the renegade scientists who claim they are ignored by their peers. These scientists are often ignored because their studies are deficient and their ideas are unconvincing.
 
6) Do not defend and promote a scientific claim just because a celebrity or politician whom you trust or like has endorsed it. Endorsement by a non-scientist of a scientific claim without any hard evidence is irrelevant to the scientific debate. Science is not politics.
 
If everyone follows these guidelines, we can hopefully restore a measure of rationality to the scientific discourse among the public.
 
Image by Tumisu from pixabay is free for commercial use and was modified.

---