1. Who stated the statistic? What organization do they belong to?
   1. Is the statistic coming from an activist trying to arouse concern about a social problem?
   2. Is the statistic reported by the media in an effort to prove a story is newsworthy?
   3. Does the figure come from someone who may not have much stake in what the numbers show?
2. What is being claimed? What are they trying to prove?
3. What is the history of this issue?
   1. Could recent events have temporarily skewed the public’s opinion on the issue?
   2. Has sufficient time past to allow multiple groups to study the issue?
4. Do they use anecdotal evidence?
   1. Does their anecdotal evidence contradict well-established evidence?
   2. Is their anecdotal evidence backed up by case studies published in peer reviewed journals?
5. Do they state a “dark figure”?

   Every social problem has a “dark figure” or a portion of the problem that goes unreported. For example: some crimes go unreported because people are too afraid, too busy to call or think the police may not be able to do anything about it (Damned Lies and Statistics, Joel Best, p33). Activists prefer to state round numbers so estimates tend to err on the side of exaggeration. Once a number is reported by the news, the number takes on a life of its own and goes through a “number laundering” (p35). This statistic is sometimes repeated so often that the people lose track of the estimate's original source. People assume that it must be correct since it appears everywhere. People who create or repeat a statistic often feel they have a stake in defending the number (p36). Challenging the motives of anyone who disputes the figure can defend any estimate. The “dark figure” often plays a role in defending guesses.

6. Find the primary source of the statistic.

   “Studies say” is a common phrase used by the media and advocates. To evaluate statistics critically, we need to go to the primary source. A primary source is the original source of information that is often detailed, peer reviewed and not interpreted from someone else. (Is that a fact? Mark Batterby, p5)

7. Who created the statistic?

   The identities of the creator of statistics often lead to clues about their motives: promoters of causes tend to report large numbers while opposition of causes tend to report small numbers. We need to be aware that the people who produce the statistics are often using the numbers as tools of persuasion. Advocates who conduct their own surveys can decide how to interpret the results. They justify such measurement decisions as efforts to reveal the true scope of social problems. They devise measurements that will minimize false negatives (incorrectly identifying cases as not being part of the problem). Rarely do they try to minimize false positives (mistakenly identifying cases as part of the problem).

8. Were questions worded in such a way that results may be biased?

   How a survey question is worded affects the results. Advocates who can afford to sponsor their own surveys can shape the results to try to demonstrate support for their position (referred to as advocacy research). For example: Gun control advocates may phrase a question “Do you favor cracking down against illegal gun sales?” (Damned Lies and Statistics, Joel Best, p 47) While gun control opposition would ask “Would you favor or oppose a law giving the police the power to decide who may or may not own a firearm?” Both questions could result in both sides of the issue appearing to having the majority of people supporting their position.

9. Are measurement decisions are hidden?

   For example: different agencies define the poverty line by a budget they consider to be reasonable for an individual or family. When the number of people who fall below the poverty line is reported, rarely the agency's definition of the determination of the poverty line is mentioned (p51-52)

10. Why was the statistic created?
11. Does the argument have sufficient evidence?
    1. Did the questions require thought and information for credible answers?
    2. Is the argument consistent with the direction of the previous research or evidence?
12. How do the conclusions in the primary source differ from the stated statistic in the media?

Back to Introduction