Search This Blog

Thursday, October 21, 2021

Data Analytics vs Confirmation Bias

 I've often said that cooperate culture is the largest impediment to effective use of data. The problem is simple. Organizations say that they are "data driven" but in practice people embrace data that supports their prior conclusions and reject data that doesn't match.

We call this confirmation bias.

This morning, I was reading the Peggy Noonan's column in the 10/7/2021 edition of the Wall Street Journal and came across this:

"I’m not a huge respecter of polls (only snapshots, not a measure of greatness or consequence) but when polls put numbers on what you’re sensing you pay attention."

Wow. I'd like to call it a "textbook" example of confirmation bias but I think it's beyond textbook. Few people are this self-aware regarding confirmation bias and, of those who realize it, even fewer will openly admit it.

I'm a fan of Noonan's writing. I applaud her honesty but I'm disappointed by her lack of trust in data.

As a long-term colleague and co-author often says - you have to be willing to let the data surprise you.


Wednesday, October 20, 2021

Survivorship Bias and Covid-19

One would think that a global pandemic that caused many of us to spend much more time at home would have resulted in great blogging productivity but that didn't happen. I haven't written anything in over a year. Instead during that time I:

  • Took over the Chair's role in an academic department that lost 25% of its faculty less than a month before the school year started.
  • Turned every class that I teach into either an online course or a hybrid (and got much better at quickly creating and editing videos),
  • Published a co-authored paper on interdisciplinary teaching, and
  • Moved over 400 miles to take a position with a new employer (go Wildcats!).
Oh well. Life gets busy. But some recent discussion I've had regarding covid-19 inspired me to create another post. Yes, another covid-19 post.

Ever since vaccines became available, there's been some debate on the role of natural immunity but there's been little actual data. For example, here's an excerpt from an October 19 Fast Company article.

-------------------------
One example: August 15, 2021 data showed cases at their peak for the period of time this tool’s data covers. On that day there were:
  • Unvaccinated: 736.72 infections per 100,000 people
  • Janssen-vaccinated: 171.92 infections per 100,000 people
  • Pfizer-vaccinated: 135.64 infections per 100,000 people
  • Moderna-vaccinated: 86.28 infections per 100,000 people
-------------------

I'm in the Janssen-vaccinated group, but I also had covid a couple of months before I was able to get the vaccine. What does "171.92 infections per 10,000 people" tell me about my risk of breakthrough infection? It's unlikely that the risk is the same for (Janssen-vaccinated/Recovered) and (Janssen-vaccinated/Never Infected). By ignoring the infected/recovered variable, these numbers aren't very useful.

Finally, in late August, a study came out of Israel claiming that natural immunity is even stronger than vaccine immunity. There is current debate on that study, but for the moment, let's assume that its findings are correct.

So what? Does this mean that you should try to get covid instead of a vaccine?

Statistically speaking - No.

Using this study to promote infection instead of vaccination commits a serious logical fallacy: Survivorship Bias. When you attempt to generalize from a dataset you need to think carefully about the population represented by your data and the population to which you wish to generalize. Survivorship bias occurs when the entities (people, airplanes, etc.) in your data set are systematically different from those that were eliminated from the data.

In the case of natural immunity, the data includes only those who quite literally survived the disease and excludes those killed by it. It should be no surprise that their current immunity is stronger than the immunity of those who died. It's entirely possible that survivors had stronger immune systems in the first place.

Therefore, the Israeli study cannot be used to recommend natural immunity over vaccination for the general population. 

So what can we say about the general population? 

If you have never had covid-19 and you are currently unvaccinated, then you have a choice:
  1. Get vaccinated and face the side-effect risks.
  2. Take your chances on getting covid and face the disease risks. 
For both of those decisions, the data is out there. Thankfully, the overall hospitalization/death rates of covid-19 are small. Still, the vaccine side-effects risks for most people are even smaller. In some demographic groups, the side-effects risks are much, much smaller than disease risks. 

Based on the data that I've seen, I recommend #1 but I respect your right to look at the same data and make a different decision based on your own medical situation.

 If you have recovered from covid-19 and you are currently unvaccinated, then you have a choice:
  1. Get vaccinated and face the side-effect risks.
  2. Take your chances on natural immunity.
This is a more difficult decision. The vaccine side-effect risks (#1) are still small. In the absence of data, I suspect that they're even smaller for the infected-recovered than for the never-infected but I think we have to assume that some risk is still there. On the other hand, we aren't sure what your reinfection risks are under #1 or #2 but they aren't zero under either.

Knowing what I know now, I would still choose vaccination. I consider the side-effect risks small and it's likely that the combination of my recovery and the vaccine is giving me even stronger protection now.

However, I see no reason to require vaccination for those who are infected-recovered. Without much stronger evidence to the contrary, they should be treated as if they were vaccinated**. 

Next, we'll have the booster issue. I can't draw any conclusion on boosters because I haven't seen any data the takes into account the difference between Vaccinated/Recovered and Vaccinated/Never-infected. This problem will not go away as long as studies and public policy continue to ignore natural immunity.

Summary: You should not seek out covid-19 in order to get natural immunity, but if you already survived covid-19 (thankfully) then your natural immunity needs to be considered.

==================
**In hindsight, I should have been denied a vaccine in March 2021. Vaccines were in short supply and many people wanted them. Those of us who already had covid should have been pushed to the back of the line. It wouldn't have hurt us to wait until June or July.