The NY Times has just reported (here), with everything but triple exclamation points, that the newest Covid vaccine by Sanofi and GSK is 100% effective against severe disease and hospitalizations. Such a claim should perk up any skeptics ears. 100%? Really? So I delved into the data.
I went to Sanofi's press release to find out the basics (see it here).
Before breaking it down, let me provide a quick explanation of "point estimates" and "confidence intervals." (You can skip the next three paragraphs if you want and go right to the 95% confidence for the new Sanofi GSK vaccine further below).
A point estimate is the best guess based on a sample. It's typically the average. So if you roll a die one time and get a 6, your point estimate is the average of your one roll: 6. Of course, it's kind of absurd to even "estimate" based on just one roll, but it's not invalid -- it's just incomplete. If I told you I rolled it only once, then you would implicitly understand that my estimate doesn't really say much about whether the die is "fair" or not. I need to roll it a bunch of times, at least, to argue that the die is "fair" (which would generate an average roll of 3.5, by the way), or that it isn't. While we know that rolling the die a bunch of times will give us a better estimate of the average, just giving the number of times we rolled it doesn't quite do it, because we need some gauge of how far the average roll is likely to vary from the fair die average of 3.5.
Statistics provides a straight-forward way of figuring out the expected variation. About 95% of averages will be within 2 standard deviations of the "true" average (for a die, the true average is the average generated if you were to roll the die an infinite number of times). Therefore, when an average is generated from a sample, you can add 2 standard deviations and subtract 2 standard deviations to get the range (as long as the sample size is fairly large) and this will cover the true average 95% of the time. For that reason, this 2 standard deviation range on either side of the "point estimate" is called a 95% confidence interval, and it's typical to provide one along with any sample estimate, so the reader knows how imprecise your estimate might be.
I won't go into how to compute a standard deviation for a die or for any average we are trying to measure, but an important fact that the standard deviation for the average of n rolls is the standard deviation of 1 roll divided by the square root of the sample size. This means that if your estimate is typically "off" by 10% after 100 rolls, you would need 4 times that (400) to get the error down to 5%. Also, important for my computations below, the 2 standard deviation rule does not apply perfectly to small samples, so I used a more accurate measure (using the binomial distribution) in my calculations, but the need to quadruple the sample size in order to double the precision still applies.
Now here is a breakdown of the Sanofi results.
For symptomatic disease the "efficacy" point estimate was 58% with a confidence interval of 27 to 77% (Sanofi provided this interval without providing the numbers behind it, but given the wide interval it's likely based on about 26 cases among vaxed and 45 cases among unvaxed). This poor performance (not statistically significantly above 50%) might be a reason why Sanofi delayed attempting to get FDA approval in the past, as the FDA has said that 50% would be a minimum efficacy for approval for a Covid vaccine.
For moderate-severe disease Sanofi reported the vaccine was 75% effective, based on 3 cases among vaccinated and 11 among controls. Sanofi does not include a confidence interval. I calculated it to be 39%-94% effective for moderate-severe disease, with 95% confidence. In other words, the sample size is too small to say much-- 39 to 94 is a really wide range.
For severe disease and hospitalizations, there's even less data: Sanofi found 0 among vaccinated and 10 among unvaxed after dose 1; and 0 vs 4 after dose 2. The post-dose 1 results are really irrelevant, because the vaccine is a 2-dose vaccine, but I computed the confidence interval for both:
post dose 1: 69% - 100% vs severe disease/hospitalization
post dose 2: 40% - 100% vs severe disease/hospitalization
Based on these results, what's a good guess? There probably is a greater effectiveness for more severe disease, but it's absurd to think it's 100% when the intervals are this wide.
By the way, the initial mRNA approval was not based on severe disease efficacy (where the numbers were very small and confidence intervals were also wide), but there's a sharp contrast with Sanofi's results for symptomatic disease and the mRNA results. For Pfizer, 8 in the vaccine group and 162 in the unvaxed group got Covid, for an effectiveness of 95% and a confidence interval of 90% - 98% effective (see Pfizer's FDA briefing Table 12).
So what's the upshot on the new Vax and the NY Times reporting?
The Times mis-reports the results, because it bases its reporting on Sanofi's deceptive, or at least overly optimistic, press release. At barely 50% effective, it will be a hard case for FDA approval, IMO.