Each week, Public Health England publishes a report with the latest information on Covid-19 case rates, hospitalisations and deaths according to individuals' vaccine status. On page 17 you'll find the below chart - of Covid rates by vaccination status among different age groups.
Up to age 39, it looks like pretty good news for vaccines - there are much higher case rates among those who haven't had the vaccine than those who have. So it's working, right? But then you look to the right, and see that for people older than 40, case rates are higher among vaccinated individuals than unvaccinated ones. Huh?
What's going on here? Well, the challenge is we aren't sure of our base sizes - a nightmare for any quant researcher. Whilst the number of vaccinated people within each group is easy to know, because it's recorded, we don't have fully up-to-date information on the size of each population in total. The report uses population numbers from GP registrations, rather than ONS estimates - which make the number of unvaccinated people appear to be smaller, and therefore the rate of cases among this group higher. When ONS estimates are used, we get a very different picture, as calculated here by mathematician James Ward:
So we can see that with a different, arguably more accurate population estimate, now the case rates are significantly lower for those who have been fully vaccinated. Many, many more factors are also at play, including behavioural ones; those who are vaccinated are likely to be the types of people who will get tested, whereas it's likely that more of the unvaccinated group might be choosing not to test either. And vaccinated people may now be exhibiting riskier behaviours because of a sense of security, meaning they might be more likely to get coronavirus.
There is an extremely brief, vague note to this effect to the right of the initial table where the data is shown on page 13 ("Interpretation of the case rates in vaccinated and unvaccinated population is particularly susceptible to changes in denominators and should be interpreted with extra caution."). But we're visual beings, and the chart tells a much more powerful story: I can see the bars are higher for vaccinated people. This is one of the only visual representations in a 37-page report which is largely quite dry reams of text and tables, so it stands out. And it makes vaccines look bad.
Not only is data presentation like this misleading, I would argue it is downright dangerous. It is extremely easy for people wanting to share vaccine disinformation to seize upon something like this - published on the official government website - as proof that vaccines don't work. Being able to link to an official source gives those spreading false information a huge advantage; anyone questioning their sources can be directed to this report, and without pretty good stats knowledge and some critical thinking, it's going to be tough to argue against.
This has real world consequences - I cycled home past a small group of anti-vax protestors on Whitehall, with one sign reading "60% of deaths are in fully vaccinated people". Googling this, Full Fact have written a clear and concise explanation of how this is a misinterpretation of data - data that came from another Public Health England report. The stat is not wrong; but it doesn't show the whole picture - fully vaccinated people at that time were also substantially more likely to be older and at much higher risk of developing serious Covid-19 in the first place.
Disinformation spreaders will always find ways to manipulate data, but right now we're at risk of poorly presented data in official reports doing that job for them.
