How can you tell if someone is trying to deceive you with statistics?

1. Look for Missing Context

Selective Data: Sometimes, a statistic might sound convincing but lacks context. For example, someone might say, “9 out of 10 people prefer this product.” But if the sample size was only ten people or if it was a specific group that already favored the product, the statistic becomes misleading.

Lack of Comparisons: Deceptive statistics often present a number without a frame of reference. For example, “Our program increased test scores by 20%!” sounds good, but without knowing the baseline, you can’t tell if it’s a significant improvement. Always ask, “Compared to what?”

2. Beware of Cherry-Picking

Time Frame Manipulation: People may choose data from specific time periods that suit their narrative. For example, if they want to show an economic decline, they might choose data from a brief recession, ignoring the bigger picture.

Omitted Data Points: By excluding certain data points, a person can create a trend that doesn’t really exist. Watch for patterns that seem suspiciously smooth or dramatic, and ask whether other data points could alter the conclusion.

3. Understand the Sample Size and Representativeness

Small Sample Sizes: A small sample size can easily lead to unreliable or skewed results. For example, a survey of only 50 people about a national issue isn’t statistically representative and may not reflect broader opinions.

Unrepresentative Samples: Even with a larger sample, if it isn’t representative, it can lead to misleading conclusions. For instance, polling only young people to gauge general opinions on social media use would give skewed results because it doesn’t account for age diversity.

4. Check for Misleading Averages

Mean, Median, and Mode: Different types of averages can create very different impressions. For example, a company might report high average salaries by using the mean, which is inflated by a few top earners. The median salary might reveal a much more modest number.

Outliers: Sometimes, one or two extreme values can skew the average. If a statistic seems unusually high or low, ask whether outliers were included and whether a different average might give a clearer picture.

5. Examine Percentage Changes Carefully

Relative vs. Absolute Change: A “50% increase” sounds significant, but if the base number is small, it might be negligible. For instance, if a town’s crime rate increased by 50% from 2 incidents to 3, the impact isn’t as dramatic as it sounds.

Percentages of Percentages: Sometimes, people use percentages of percentages to make changes seem larger than they are. For example, a “20% improvement on a 10% increase” can sound impressive but often has a smaller impact than it appears.

6. Watch for Correlation vs. Causation

Assuming Causation: Just because two variables correlate doesn’t mean one causes the other. For example, if there’s a correlation between ice cream sales and drowning incidents, it doesn’t mean ice cream causes drowning; both increase with warm weather.

Hidden Variables: There may be a third factor affecting both variables, creating a spurious correlation. Scrutinize any claims that suggest “A causes B” and ask whether other factors could be influencing the data.

7. Look Out for Graphical Manipulation

Manipulated Axes: Changing the scale of a graph, especially the y-axis, can make small changes look large. For example, if a bar chart’s y-axis starts at 90 instead of 0, a tiny difference will appear exaggerated.

3D Charts and Visual Distortions: Sometimes, 3D or complex graphics can distort data, making one value look larger than another by changing perspective or design. Simplified, two-dimensional charts are often more reliable for accurate comparisons.

8. Consider the Source and Possible Bias

Motivated Reasoning: Statistics from interested parties, like a company promoting its product or a political organization, may be selectively presented to support their agenda. Look for sources with minimal bias or, better yet, independent studies that verify the claim.

Funding Sources: Ask who funded the research. If a study on the health benefits of a certain product is funded by the company that sells it, the findings may be biased. Independent or peer-reviewed sources are generally more reliable.

9. Beware of Sweeping Generalizations

Overgeneralization from Small Findings: Claims like “Studies show” can be deceptive if the cited study is small or limited in scope. A single study rarely provides conclusive evidence and is more reliable when corroborated by additional research.

Implying Universality: Statements like “Everyone agrees” or “This is always the case” are rarely backed by solid evidence. Be wary of conclusions that suggest a universal truth without substantial backing.

10. Ask Questions and Dig Deeper

Interrogate the Statistic: If a statistic seems surprising or out of place, ask questions like, “What’s the sample size?” “Is this change meaningful?” or “What’s the context?”

Seek Original Sources: If you’re suspicious, try to find the original study or data source. Often, you’ll find additional details in the source material that help clarify any deceptive presentation.