On September 21st, 2021 I received an email from Tommy Caldwell, author and owner of Hybrid Fitness in my hometown of London Ontario, with the subject line “COVID-19, Kids, and Vaccines: A difficult decision for parents to navigate”. The email was sent from Hybrid Fitness’ email address (which I had subscribed to sometime in 2018, in the first year of my PhD). In that email, Mr. Caldwell acknowledges that vaccinating is a “no-brainer decision” and professes he is an inexpert in COVID19, science, and research, yet continues on to interpret statistics from a clinical trial intended to evaluate the vaccine against COVID19 in persons aged 12-15 years.
Mr. Caldwell cites that the 121 severe adverse events (SAAs) occurred in the intervention arm of the trial, constituting a 5:1 ratio when compared to placebo arm. He goes on to say
“A direct quote from the trial also suggested that researchers were concerned that they were missing severe adverse events as well due to the study design. In their words, ““There was serious concern of indirectness because the body of evidence does not provide certainty that rare serious adverse events were captured due to the short follow-up and sample size. There was also very serious concern for imprecision, due to the width of the confidence interval.”
I have included the entire email as a png below so that you can read his words from him in context.
Mr. Caldwell has, in my opinion, grossly misinterpreted these statistics and in doing so weaponizes them to support a narrative that vaccinating one’s children is a “difficult decision” when proper interpretation of the statistics should lead one to believe that the vaccine is at the very least consistent with no increase in risk of harm. This blog post is intended to rectify the mistakes Mr. Caldwell has made to his assumedly large readership.
In what follows, I present corrected claims found in Mr. Caldwell’s email as section titles with more details in the section body. If you question my ability to intelligently comment on statistics at this level (which you should), I encourage you to poke around this website for proof that I understand statistics sufficiently well. If you need additional proof, please see some of the many answers I provide on statistical forums, or review some of the papers I have published in medical and epidemiology journals.
5:1 Ratio of People Reporting Reactogenicity, Not Severe Adverse Events
Table 3d in the link above summaries the number of participants who reported “Reactogenicity”. I assume these are the numbers Mr. Caldwell refers to in his email as I can’t find any others which match the aformentioned ratio. The ratio of reactogencitity between intervention and placebo arms is roughly 5 (121 / 22 is approximately 5.5). However, the definition of reactogenicity is (from the text below table 3d)
Reactogenicity outcome includes local and systemic events, grade ≥3. Grade 3: prevents daily routine activity or requires use of a pain reliever.
This means that 121 people may have, for example, needed an Advil, or got a sore arm, or may have needed to take a nap. The study authors do not list the outcomes explicitly, but we can surmise they are very minor from the definition used. On the other hand, a severe adverse event (which Mr. Caldwell originally claimed the 5:1 ratio applied) means (see table 3c in the link above)
Death, life-threatening event, hospitalization, incapacity to perform normal life functions, medically important event, or congenital anomaly/birth defect
Note further that counts of SAAs is much lower than the counts he mentions in the email. Mr. Caldwell has, I assume mistakenly, reported a less severe reaction as a more severe reaction. This can (and probably has) garnered fear from readers. I’m not a parent myself, but I wouldn’t want my dog undergoing a SAA let alone my mother or future child.
Data Are Consistent with No Increase of Risk of SAAs.
Table 3c in the link above examines SAAs. Sample sizes in each arm are comparable (1131 and 1129 in intervention and placebo respectively) and so an absolute comparison of number of SAAs is allowable.
Five people in the intervention arm reported an SAA and 2 people in the placebo arm reported an SAA, a difference of 3 people. This difference is very small, and is consistent with the assumption the vaccine does not increase the risk of SAAs. A difference of 3 people or more is completely in line with sampling variability. For those comfortable in programming in R, we can simulate this and determine the probability of seeing one of the two groups have 3 or more people than the other.
# The risk under the null hypothesis would be # 0.003097345. # Simulate 1 million trails where there is no difference in # risk of SAA # Compute the probability one of the groups # has 3 more SAAs than the other set.seed(0) #Set the random seed for reproducibility risk = 0.003097345 # Generate 1 million similar scenarios where there is no real difference # in risk of an SAA intervention = rbinom(1e6, 1131, risk) placebo = rbinom(1e6, 1129, risk) # Here is the probability we find one of the two groups having a difference of 3 or more mean(abs(intervention - placebo)>=3)  0.334566
So there is approximately a 33% chance we would see one of the two groups have 3 or more SAAs than the other when in truth no difference in risk exists between groups.
The study authors report something called the relative risk (RR in the table). The relative risk is the risk of SAA in intervention arm divided by the risk of SAA in placebo arm. If the relative risk is greater than 1, then the risk of SAA is bigger in the intervention group than the placebo. The authors report a relative risk of 2.5 (meaning the risk of SAA in the intervention arm was estimated to be 2.5x greater than the placebo arm). However, that does not tell the whole story.
A 2.5x increase in risk is one thing, but the authors also report something called a confidence interval. A confidence interval gives values of the true relative risks which might have plausibly generated the data we have seen. Note that the associated confidence interval is 0.49 to 12.84, meaning the data are consistent with a true relative risk as small as 0.5 (again, meaning the risk of SAA would be smaller in the intervention arm than in placebo arm).
A popular rebuttal would be “Demetri, the confidence interval is also greater than 1, meaning the data are consistent with relative risks as big as 13!”. Agreed, that is true, but if the data are consistent with reductions in risk and increases in risk, then we should conclude that we can’t make any conclusions based on the conflicting evidence. The confidence interval is too wide (or as statisticians would say, the estimate is too imprecise, a word used by the study authors and quoted by Mr. Caldwell). This does not mean we can conclude the vaccine does not increase the risk of SAA, it only means that the data are consistent with no change in the risk of SAA. To know with more certainty if the risk of SAA changes between groups, we would need more data.
Which leads me to my final point…
Very Serious Concern Regarding The Size of the Confidence Interval is About Inability to Comment on Change in Risk of SAA, Not About How the Study Was Conducted
As I mentioned before, that the confidence interval spans numbers less than 1 and greater than 1 means we can’t comment on a change in risk of SAA. In Mr. Caldwell’s email, it appears Mr. Caldwell seems to interpret the authors’ concern about “indirectness” as meaning they were missing SAAs or that the concern was about the study design.
For the reasons I have commented on above, this is false. SAA has been very well defined in this paper, and what I anticipate Mr. Caldwell is concerned about is long term effects of the vaccine. An understandable concern, but not one I can see being studied in sufficient depth to allow for an expedient release of a vaccine to combat COVID19.
This letter in response is much larger than Mr. Caldwell’s initial email, a testament to Brandolini’s Law better known as the “The Bullshit Asymmetry Principle”
The amount of energy needed to refute bullshit is an order of magnitude larger than to produce it.
And make no mistake that I believe Mr. Caldwell’s claims about these studies to be nothing short of bullshit. I have no personal vendetta against Mr. Caldwell, nor do I take issue with his personal choices or concerns about vaccinations, but I do take issue with making such gross errors in interpretation of statistics tp laymen audiences which trust your work on matters of health, broadly construed. I’ve contacted Mr. Caldwell and politely and kindly asked him to send another email to the same mail list correcting some of his errors. He has not explicitly said “No” but has yet to directly address the request at this time.