- Who created the statistic?
- What organization do they belong to?
- Is the statistic coming from an activist trying to arouse concern about a social problem?
- Is the statistic reported by the media in an effort to prove a story is newsworthy?
- Does the figure come from someone who may not have much stake in what the numbers show?
The identities of the creator of statistics often lead to clues about their motives: promoters of causes tend to report large numbers while opposition of causes tend to report small numbers. We need to be aware that the people who produce the statistics are often using the numbers as tools of persuasion.
Advocates who conduct their own surveys can decide how to interpret the results. They justify such measurement decisions as efforts to reveal the true scope of social problems. They devise measurements that will minimize false negatives (incorrectly identifying cases as not being part of the problem). Rarely do they try to minimize false positives (mistakenly identifying cases as part of the problem). Activists trying to create a new social problem view false negatives as more troubling as false positives. They often feel frustrated because they want people to be concerned about some social condition that has been ignored so they sometimes use broad definition to justify big numbers. For example defining rape and flashing as sexual violence. They may both be unwanted but defining them together may imply that they are equally serious. (p 41)
- Is the creator of the statistic a credible source?
- Were the results published in a peer-reviewed journal?
- Has the author simply gone from a correlation or even coincidence to a causal claim without good evidence?
- Is it likely that the apparent convergence is a result of a group think or bandwagon jumping, rather than an emerging consensus driven by the careful research?
- Have the researchers indicated financial or other interests?
- Has the funding prohibited the researcher from publication results without authorization of the funding party?
- Was the research plainly funded to establish an interest of the funding party?
- What is being claimed? What are they trying to prove?
- What is the margin of error? Claims about the population based on a sample always involve a margin of error.
- Does the report mistake the information about the sample, for the claim about the population?
- Does the report really reflect the questions asked?
- How good is the evidence? What is the evidence? Assuming the evidence is true, how much support does the evidence provide for the conclusion? Is the evidence credible?
It is far too expensive and difficult to count every instance of a problem. The sample chosen should represent the larger population of all cases. Since samples tend to be much smaller than their populations, it is not uncommon to have only a few dozen people in a sample. Large samples are not necessarily good samples as the representativeness of a sample is actually far more important than sample size. A good sample accurately reflects the population. Random samplings best represent the population but few samples are random. It can be time consuming and expensive to draw a random sample or the population cannot be defined.
Generalizations are often made on the basis of minimal evidence they do have. For example, people claim that living near wind turbines causes health problems. However people living near wind turbines are often poor and it is a factor that is rarely included in studies by opponents of wind turbines near residential areas. The media often fail to question activists’ generalizations because they may be unable to find anyone with better evidence and thus they would lose a news story (p58)
- Was the sampling method unbiased?
- Was the sample self-selected (not random)?
- Could a sampling bias occurred from non-responses, lack of phone, and minority languages?
- Was the sample large enough?
Approximately 1000 people are needed to generate the kind of margin of error and confidence level that has become accepted as a reasonable basis for claims about national issues (+/- 3 percent age points). For local surveys 500 is typically used (+/- 5 percentage points).
- Is the margin of error allowed for and credible?
The margin of error given is the mathematical ideal; it is reasonable to assume that the actual margin is greater. Watch out for claims about sub-polls that have a much larger margin of error. Local polls will usually be smaller and as a result will have margins of error larger than national polls. For comparisons between two polls to be credible the difference in the samples should exceed both pools’ margin of error.
- Were there non-sampling biases? Did the questions, question order, survey introduction, or interviewer invite biased answers?
- Think about the question/answer options: did they invite a certain kind of response?
- Do people understand the issue?
- Would respondents likely be honest and accurate in response to these types of questions?
- Was the sponsor of the poll biased? Did the bias affect the poll?
How a survey question is worded affects the results. Advocates who can afford to sponsor their own surveys can shape the results to try to demonstrate support for their position (referred to as advocacy research). For example: Gun control advocates may phrase a question “Do you favor cracking down against illegal gun sales?” (Damned Lies and Statistics, Joel Best, p 47) While gun control opposition would ask “Would you favor or oppose a law giving the police the power to decide who may or may not own a firearm?” Both questions could result in both sides of the issue appearing to having the majority of people supporting their position.
- Were appropriate comparisons made?
- Over time
Evidence that a problem is getting worse depends on measuring the change of a condition over at least of points in time. There is usually an implication that things will continue to deteriorate. (p98) We need to remember that differences do not actually reflect changes since the definition or method of measurement may have changed drastically changing the statistical comparison. For example: reports of child abuse – definition of child abuse is very different today than 40 years ago.
- Among Places
Agencies define social conditions in different ways. For example: gang-related killings are sometimes count regarded of circumstances
- Among Groups
Labels can shift over time or are not clearly defined. For example in 1996, the US Census Bureau defined Hispanic as an ethnic group rather than a racial group so some Hispanics were classified as white, black or Native American.
Also the number of cases from groups of different sizes is often unfairly compared.
- Among Social Problems
Society tends to worry about more serious threats rather than more prevalent threats (i.e. murders vs deaths due to auto collisions). It is difficult to decide which issue is more serious and would depend on what society values: monetary cost, big numbers, rapid increases in numbers, geographic and group comparisons (worse in one place than another) (p123). It is flawed to emphasize a problem's importance by focusing on a narrowly defined population. For example: “Suicide is the second leading cause of death among adolescents” remembering that few adolescents die from natural causes. One the other hand, too broad of a definition can be deceiving. For example defining rape and flashing as sexual violence. They may both be unwanted but defining them together may imply that they are equally serious. (p 41)
- What is the history of this issue?
- Could recent events have temporarily skewed the public’s opinion on the issue?
- Has sufficient time passed to allow multiple groups to study the issue?
- Does the argument have sufficient evidence?
- Did the questions require thought and information for credible answers?
- Is there a strong correlation?
- Is the argument consistent with the direction of the previous research or evidence?
- If in conflict with the previous research, does the argument deal effectively with opposing evidence or arguments; is it strong enough to counter this previous research?
- Do they use anecdotal evidence?
- Does their anecdotal evidence contradict well-established evidence (p9)?
- Is their anecdotal evidence backed up by case studies published in peer reviewed journals?
- Do they state a “dark figure”?
Every social problem has a “dark figure” or a portion of the problem that goes unreported. For example: some crimes go unreported because people are too afraid, too busy to call or think the police may not be able to do anything about it (Damned Lies and Statistics, Joel Best, p33). Activists prefer to state round numbers so estimates tend to err on the side of exaggeration.
Once a number is reported by the news, the number takes on a life of its own and goes through a “number laundering” (p35). This statistic is sometimes repeated so often that the people lose track of the estimate's original source. People assume that it must be correct since it appears everywhere. People who create or repeat a statistic often feel they have a stake in defending the number (p36). Challenging the motives of anyone who disputes the figure can defend any estimate. The “dark figure” often plays a role in defending guesses.
- Are measurement decisions hidden?
For example: different agencies define the poverty line by a budget they consider to be reasonable for an individual or family. When the number of people who fall below the poverty line is reported, rarely the agency's definition of the determination of the poverty line is mentioned (p51-52)
- Why was the statistic created?
- Does the argument have sufficient evidence?
- Did the questions require thought and information for credible answers?
- Is the argument consistent with the direction of the previous research or evidence?
Back to Introduction
|
