Evaluating Evidence

How to assess the quality and strength of evidence, understand statistics, and avoid being misled by data.

Why Evidence Evaluation Matters

Evidence is the foundation of justified belief. But not all evidence is equal. A single anecdote, a controlled experiment, and a meta-analysis of 50 studies all count as "evidence," but they tell you very different things with very different levels of reliability.

The ability to distinguish strong evidence from weak evidence is the single most practical critical thinking skill.

Types of Evidence

The Evidence Hierarchy

From weakest to strongest:

STRONGEST
  ↑  Systematic reviews / Meta-analyses
  |  Randomized controlled trials (RCTs)
  |  Cohort studies
  |  Case-control studies
  |  Cross-sectional studies
  |  Case reports / Case series
  |  Expert opinion
  |  Anecdotal evidence
  ↓  "I heard somewhere that..."
WEAKEST

Why the Hierarchy Exists

The hierarchy is a starting point, not a rigid rule. A well-done cohort study can outweigh a poorly designed RCT.

Sample Size and Representativeness

Sample Size

Key principle: Larger samples are more likely to reflect the true population. Small samples can easily produce misleading results through chance alone.

Representativeness

A large but biased sample is worse than a smaller representative one.

Ask: "Who was studied? Who was left out? Does the sample look like the population I'm generalizing to?"

Correlation vs. Causation

The most important statistical distinction in everyday reasoning.

Three Requirements for Causation

Why Correlation Isn't Causation

Causal Traps

Understanding P-Values and Statistical Significance

What a P-Value Actually Means

A p-value is the probability of seeing results at least as extreme as the observed results, IF the null hypothesis is true.

What P-Values Do NOT Mean

Statistical Significance vs. Practical Significance

Always ask: "How big is the effect?" not just "Is it statistically significant?"

Evaluating Study Quality

Red Flags in Research

Questions to Ask About Any Study

1. Who funded it? Industry-funded studies are more likely to find favorable results.
2. How big was the sample? Small samples are unreliable.
3. Was there a control group? Without one, you can't attribute effects to the treatment.
4. Was it randomized? Without randomization, differences between groups could explain results.
5. Was it blinded? If participants or researchers know who gets what, it introduces bias.
6. Has it been replicated? A single study is a starting point, not a conclusion.
7. What's the effect size? Statistically significant doesn't mean practically important.
8. Was the study pre-registered? Pre-registration prevents HARKing (Hypothesizing After Results are Known).

The Replication Crisis

Many published findings, especially in psychology, medicine, and social sciences, don't hold up when other researchers try to replicate them.

Key Facts

Implications for You

• Don't treat any single study as definitive
• Look for replications and meta-analyses
• Be especially skeptical of surprising or counterintuitive findings from small studies
• "A new study says..." is one of the weakest forms of evidence

Base Rates and Bayesian Thinking

What Are Base Rates?

The base rate is the underlying frequency of something in the population. Ignoring it leads to wildly wrong conclusions.

Classic example:

A disease affects 1 in 1,000 people. A test is 99% accurate (99% true positive, 1% false positive). You test positive. What's the probability you actually have the disease?

Probability you have it: 0.99 / 10.98 ≈ 9%. Not 99%.

Most people guess 99%. The base rate (1 in 1,000) is crucial.

Bayesian Updating

Instead of treating beliefs as binary (true/false), treat them as probabilities that update with new evidence.

Prior belief → New evidence → Updated belief
  (what you believed)  (what you learned)  (what you now believe)

Practical process:

1. Start with a prior probability (your initial estimate)
2. Evaluate the new evidence (how likely is this evidence if your belief is true? If false?)
3. Update your probability accordingly
4. Repeat as new evidence arrives

Example:

Key Principles

• Strong prior + weak evidence = minor update
• Weak prior + strong evidence = major update
• Multiple independent evidence sources compound
• Update incrementally, not all-or-nothing

Common Evidence Pitfalls

The Narrative Fallacy

We're wired to prefer stories over statistics. A single compelling story can override mountains of data.

The story may be true. But it's one data point. Always check whether the story represents the typical case or an extreme outlier.

Denominator Blindness

We focus on the numerator (events that happened) and ignore the denominator (total opportunities for events).

Example: "5 people died from X this year!" Scary? Depends on the denominator:

• 5 out of 10 = catastrophic (50%)
• 5 out of 10,000,000 = trivially rare (0.00005%)

Always ask: "Out of how many?"

The Availability Cascade

Vivid, dramatic evidence gets remembered and shared more, creating the illusion that it's more common.

Why you think: Shark attacks are common (dramatic news stories) Reality: You're more likely to be killed by a vending machine

Practical Evidence Evaluation Framework

When confronted with any claim backed by "evidence," run through this:

Key Takeaways

1. Not all evidence is equal. Learn the hierarchy and use it
2. Correlation ≠ causation. Always look for confounds, reverse causation, and coincidence
3. P-values are widely misunderstood. Statistical significance doesn't mean practical significance
4. Single studies prove nothing. Look for replications and systematic reviews
5. Base rates matter enormously. Ignoring them leads to wildly wrong conclusions
6. Stories aren't data. Vivid anecdotes can override statistics but shouldn't
7. Always ask "out of how many?" The denominator changes everything
8. Update incrementally. Bayesian thinking beats binary true/false