How to Defend Yourself From Statistical Lies: Part 3

Breaking down actual news stories to show the statistical lies, using the example of school shootings.

COMING UPDATES: If you’d like to support my continuing to do this, you can send an Amazon giftcard to hollymathnerd at gmail or CashApp me at $HollyMathNerd. Thank you!

Part 2 of this series explains how to lie with percentages, using examples of statistics about school shootings.

My statistical self-defense series has many other topics: go here for the list.

At the end of 2018, several agencies that work to fight gun violence released reports asserting that 2018 was the worst year on record for “school shootings” or “school shooting incidents”. In early 2019, the CDC issued a report in its weekly “Morbidity and Mortality” report about an aspect of American death. December 2018 and January 2019 both feature a plethora of media sources misinterpreting statistics.

Vox

In December 2018, Vox ran a story with this headline: “2018 was by far the worst year on record for gun violence in schools.”

They use the US Naval database (linked in the list of sources at the bottom of part 2), which counts: “each and every instance a gun is brandished, is fired, or a bullet hits school property for any reason, regardless of the number of victims, time of day, or day of week.” That database shows 97 incidents and 55 deaths and make comparisons to years with smaller numbers.

Here’s why these numbers are not an honest representation:

Major reason #1: the counting of incidents is overly broad.

• A drug deal on the school playground at 3 in the morning during summer vacation (indeed, ALL summer vacation gun activity on school property) increases the count as much as a massacre. So does a stray bullet at 2am on Christmas morning or a homeless person trying to sell a gun he found in a dumpster at midnight on a Saturday.
• Incidents count that have nothing to do with the fact that they happened at a school. When a school secretary in the midst of a divorce has her angry soon-to-be-ex brandish a gun in the parking lot, it’s a terrible thing. But it is domestic violence, not school violence.
• Suicides and suicide attempts that involve only the victim are counted. Again, they are terrible and such students deserve sympathy and help. But they are not school violence. A depressed teenager who commits suicide in the school bathroom so that his mother won’t be the one to find him is a modern tragedy. He is not making American schools more dangerous if he happens to do this with a gun vs. an overdose or a razor blade.

What would a more reasonable metric be for counting incidents? I propose any of these three are more appropriate:

• Counting only incidents where at least one bullet was discharged, the total for 2018 is 36 incidents — just slightly more than 1/3 of the claimed number.
• Counting only incidents where at least one person was injured, the total for 2018 is 34 incidents.
• Counting only incidents where at least one person was killed, the total for 2018 is 15 incidents: about 15% of the claimed number that the scary headline is based upon.

Major reason #2: the counting of deaths is also overly broad.

• The report that Vox based its fear-mongering headline on counted 55 deaths. That number was 44 according to a data set that does not include police actions, murder-suicides by estranged spouses or jilted lovers, OR suicides (all of which it makes sense to exclude, as they are not related to the fact of happening at school; any more than the estranged spouse of a Walmart cashier committing murder-suicide in the parking lot proves that Walmart is a dangerous place to work).
• 12 of those 44 were not students. Some were teachers or staff killed in mass shootings. Others were parents killed by their own child when they came to campus, bystanders, or community members who lived near the school. (I did not try to separate out these numbers since I couldn’t locate reliable enough information.)

What happens if you use the more reasonable numbers of 19 incidents (34–15=19) with an injury but no deaths and 15 incidents with at least one death? It’s absolutely terrible that such things happen at all, of course. But you don’t have a clickbait headline or easy political fodder anymore, do you?

The Guardian

In December 2018, the Guardian (a British news source) also ran a story about 2018 school shooting statistics in America. Their headline: “2018 is worst year on record for gun violence in schools, data shows.”

They cite 2018 as a 60% increase over the previous high, 2006, in which they claim 59 incidents.

The 59 count is based on the same data set with the same overly broad definition as the Vox article. In more reasonable metrics:

• 2006 had 10 incidents with at least one bullet being discharged.
• 2006 had 9 incidents with at least one injury.
• 2006 had 6 incidents with at least one death.
• 2006’s totals were 15 deaths and 18 injuries. The Guardian does not mention these numbers — perhaps because their readers would compare the claimed number of incidents to these numbers and then it would be hard to explain how bad Americans must be at shooting, huh?

CDC Report

The CDC does a great job of laying out the facts, which the media then interprets. I am including a bit of their report with some commentary here because I think it’s enlightening.

• More media sources than I could count reported breathlessly that gun violence increased dramatically between June 2009 and June 2018. Here’s what the CDC actually said:

From the CDC “Morbidity and Mortality” report in January 2019.

Note that they explicitly state that multiple-victim homicides fluctuated substantially annually and that there is not a “significant trend” for multiple-victim homicides for the overall period. Such incidents declined during July 1994-June 2009 and then increased. They specifically note that July 2016-June 2018 had 8 incidents and make no attempt to say that this anything more than part of the significant fluctuation noted throughout the time for which they are referring to data.

Two other important facts that no media source mentioned (that I could find). In the July 1994-June 2018 period, 60.5% of the perpetrators in these multiple-victim homicides were under 18. That means almost 40% were adults! It makes the media narrative of American kids murdering each other at “epidemic” rates look a little different, yes? To be sure I wasn’t inadvertently lying with statistics myself, I got the raw numbers. 17 of the total 47 perpetrators of multiple-victim homicides in that time were adults, and 1 was of unknown age.

39.5% of the shooters in multiple-victim homicides on campus were adults.

Age data for the perpetrators in the July 1994-June 2018 period is in the last column on the right.

A note on how and why such narratives come to be supported by statistical lies: to the best of my knowledge, journalists don’t study statistical analysis in “J-School,” and I tried to make no implication about whether their use of statistics to fit a pre-existing narrative was intentional or implicit. Even if they do have the knowledge to do statistical analysis correctly, well, we have good evidence that most journalists are liberal in their personal politics, and it’s difficult to examine statistics dispassionately. I hate guns and have witnessed gun violence. I picked this issue, about which I have deeply conflicting and strong emotions, to challenge myself and my commitment to being an honest statistician. It has in fact been a challenge, and I don’t have a boss breathing down my neck expecting a story to fit any particular narrative.

Let’s end with a story that got the statistical analysis right. Congratulations, NPR.