Introduction

In Topic 6.4, we learned to set up hypothesis tests by stating null and alternative hypotheses and checking conditions. Now comes the heart of hypothesis testing: the p-value.

The p-value is perhaps the most important—and most misunderstood—concept in all of statistical inference. It's the number that determines whether we reject the null hypothesis or not. It's reported in virtually every research paper, clinical trial, and scientific study. Yet it's frequently misinterpreted, even by professionals.

Understanding p-values correctly is essential not just for AP Statistics, but for being a critical consumer of research and data. This topic will clarify what p-values actually mean, how to find them, and most importantly, how to interpret them properly.

The Null Distribution

Before we can understand p-values, we need to understand the null distribution.

What is the Null Distribution?

The null distribution is the probability distribution of the test statistic assuming the null hypothesis is true.

Think of it this way: If $H_0$ were actually true, what values would we expect to see for our test statistic? The null distribution answers this question.

Creating the Null Distribution

There are two main ways to obtain a null distribution:

For the one-sample z-test for a proportion, we use the theoretical approach. The null distribution is approximately a standard normal distribution (z-distribution) when conditions are met.

Why It's Called the "Null" Distribution

We call it the "null" distribution because it describes what happens if the null hypothesis is true. Every calculation we do assumes $H_0$ is correct—that's the starting point for hypothesis testing.

What is a p-Value?

The p-value is the probability of obtaining a test statistic as extreme or more extreme than what we actually observed, given that the null hypothesis is true.

Breaking Down the Definition

Let's unpack each part:

"Probability of obtaining...": The p-value is a probability (between $0$ and $1$)

"...a test statistic as extreme or more extreme...": "Extreme" means far from what we'd expect if $H_0$ were true

"...than what we actually observed...": We compare to our actual sample result

"...given that the null hypothesis is true": This is crucial—the entire calculation assumes $H_0$ is correct

The Conditional Nature of p-Values

The p-value is a conditional probability. We're asking: "If the null hypothesis were true, what's the probability of seeing data like ours (or more extreme)?"

What a p-value IS: $P(\text{data as extreme or more extreme }|H_0\text{ is true})$

What a p-value is NOT: $P(H_0\text{ is true }|\text{ data})$

This distinction is critical. The p-value does NOT tell us the probability that the null hypothesis is true!

An Analogy

Imagine a friend claims they can predict coin flips. They predict $8$ out of $10$ flips correctly.

The p-value answers: "If they were just guessing randomly (like the null hypothesis), what's the probability they'd get $8$ or more correct just by chance?"

The p-value does NOT answer: "What's the probability they were guessing?"

Finding p-Values for Different Alternative Hypotheses

How we calculate the p-value depends on the direction of the alternative hypothesis. Let's say our test statistic is $z=x$ (some calculated value).

One-Sided Alternative: Greater Than Hₐ: p > p₀

When the alternative hypothesis is "greater than," we want to know how often we'd see a test statistic at or above what we observed.

p-value = $P(z\ge x)$

This is the area in the right tail of the standard normal distribution.

Example:

Interpretation: If the true proportion were $0.30$, there's about a $1.58\%$ chance of getting a test statistic of $2.15$ or higher.

One-Sided Alternative: Less Than Hₐ: p < p₀

When the alternative hypothesis is "less than," we want to know how often we'd see a test statistic at or below what we observed.

p-value = $P(z\le x)$

This is the area in the left tail of the standard normal distribution.

Example:

Interpretation: If the true proportion were $0.50$, there's about a $3.36\%$ chance of getting a test statistic of $-1.83$ or lower.

Two-Sided Alternative: Not Equal To Hₐ: p ≠ p₀

When the alternative hypothesis is "not equal to," we care about extreme values in either direction. We want test statistics that are far from $0$ in either the positive or negative direction.

p-value = $P(z\le -|x|)+P(z\ge |x|)$

This is the area in both tails of the standard normal distribution.

Equivalently: p-value = $2\cdot P(z\ge |x|)$ (by symmetry of the standard normal distribution)

Example:

Interpretation: If the true proportion were $0.60$, there's about a $2.08\%$ chance of getting a test statistic at least as far from $0$ as $-2.31$ (in either direction).

Visual Understanding

For a two-sided test, we look at both tails because we're testing for a difference in either direction. If our test statistic is $z=2.5$, we also consider $z=-2.5$ equally extreme, so we include both tail areas.

Finding p-Values Using Technology or Tables

In practice, we use technology (calculators, software) or standard normal tables to find p-values.

Using a Standard Normal Table

Standard normal tables typically give $P(z<x)$ (area to the left).

For $H_a:p<p_0$:

For $H_a:p>p_0$:

For $H_a:p\ne p_0$:

Using Technology

Most calculators and statistical software have built-in functions:

Important: Always verify that your technology is using the correct alternative hypothesis direction!

Interpreting p-Values in Context

A proper interpretation of a p-value must include several components:

The Complete Interpretation Template

"The p-value of [value] is the probability of obtaining a sample proportion of [observed value] or [more/less extreme in direction of $H_a$], assuming that the true population proportion [state $H_0$ in context]."

Example 1: One-Sided (Greater)

Context: Testing if more than $40$% of voters support a proposal

Good interpretation: "The p-value of $0.023$ is the probability of obtaining a sample proportion of $0.47$ or greater, assuming that the true proportion of all voters who support the proposal is $0.40$."

Example 2: One-Sided (Less)

Context: Testing if less than 20% of items are defective

Good interpretation: "The p-value of $0.087$ is the probability of obtaining a sample proportion of $0.14$ or less, assuming that the true proportion of all items that are defective is $0.20$."

Example 3: Two-Sided

Context: Testing if the proportion has changed from 0.65

Good interpretation: "The p-value of $0.042$ is the probability of obtaining a sample proportion as far or farther from $0.65$ than $0.58$ (in either direction), assuming that the true proportion is $0.65$."

What to Include

Every p-value interpretation should include:

What NOT to Say

❌ "The probability that the null hypothesis is true is $0.023$"

❌ "There's a $2.3$% chance that $p=0.40$"

❌ "The probability the alternative is true is $0.977$"

❌ "The null hypothesis is false with probability $0.977$"

Small p-Values: Evidence for the Alternative

Small p-values indicate that the observed test statistic would be unusual if the null hypothesis were true.

The Logic

If we observe something that would be very rare under $H_0$, we have two options:

Small p-values suggest option 2 is more reasonable.

What "Small" Means

The smaller the p-value, the more convincing the statistical evidence for the alternative hypothesis.

Common benchmarks (significance levels):

Interpreting Small p-Values

When the p-value is small:

Example: p-value = $0.008$ "A sample result this extreme would occur only about $0.8\%$ of the time if the null hypothesis were true. This provides strong evidence for the alternative hypothesis."

Large p-Values: Lack of Evidence

p-values that are not small indicate that the observed test statistic would not be unusual if the null hypothesis were true.

The Logic

If we observe something that's reasonably likely under $H_0$, we don't have reason to doubt $H_0$.

Important: This does NOT mean $H_0$ is true! It means our data are consistent with $H_0$.

What Large p-Values Tell Us

When the p-value is large:

What Large p-Values Do NOT Tell Us

❌ Large p-values do not provide evidence that $H_0$ is true

❌ Large p-values do not prove $H_0$

❌ Large p-values do not mean $H_a$ is false

The key principle: Lack of evidence against $H_0$ is not the same as evidence for $H_0$.

An Analogy

If a jury finds a defendant "not guilty," that doesn't mean the defendant is innocent. It means there wasn't sufficient evidence for conviction. Similarly, failing to reject $H_0$ means we lack sufficient evidence against it—not that it's true.

Example: p-value = $0.34$ "A sample result this extreme would occur about $34\%$ of the time if the null hypothesis were true. This does not provide convincing evidence for the alternative hypothesis, though it doesn't prove the null hypothesis is true either."

Common Misconceptions About p-Values

Misconception 1: The p-value is the probability H₀ is true

WRONG: "The p-value of 0.05 means there's a 5% chance the null hypothesis is true."

CORRECT: The p-value is the probability of observing data like ours if $H_0$ is true. It's not the probability that $H_0$ itself is true.

Misconception 2: Large p-values prove H₀

WRONG: "The p-value is $0.62$, so the null hypothesis is true."

CORRECT: A large p-value means the data are consistent with $H_0$, but many other parameter values might also be consistent with the data.

Misconception 3: The p-value is the probability of making an error

WRONG: "A p-value of $0.03$ means there's a $3\%$ chance we're making a mistake."

CORRECT: The p-value is about the probability of observing our data given $H_0$, not about the probability of making an error in our decision. (Type I error probability is the significance level $\alpha$, which we'll discuss in Topic 6.6.)

Misconception 4: p = 0.051 and p = 0.049 are drastically different

WRONG: "$p=0.049$ is significant but $p=0.051$ is not, so they lead to completely different conclusions."

CORRECT: p-values near the significance level are borderline cases. The difference between $0.049$ and $0.051$ is negligible—both suggest weak to moderate evidence. Don't treat the significance level as a rigid boundary.

The Relationship Between p-Values and Strength of Evidence

p-values exist on a continuum. Here's a rough guide:

$\begin{array}{cc}\text{p-value Range}& \text{Strength of Evidence Against }{H}_0\\ p<0.01& \text{Very strong evidence}\\ 0.01\le p<0.05& \text{Strong evidence}\\ p\ge 0.10& \text{Little to no evidence}\end{array}$

Remember: These are guidelines, not rigid rules. Context matters, and the formal decision-making process (Topic 6.6) uses a predetermined significance level.

Simulated Null Distributions

Essential Knowledge 3.A.1 mentions that if the null distribution has been simulated, the p-value is found as a proportion rather than a probability from a theoretical distribution.

How Simulation Works

Finding p-Values from Simulations

For $H_a:p>p_0$: p-value = proportion of simulated statistics $\ge$ observed statistic

For $H_a:p<p_0$: p-value = proportion of simulated statistics $\le$ observed statistic

For $H_a:p\ne p_0$: p-value = proportion of simulated statistics $\ge |$observed$|$ + proportion $\le -|$observed$|$

Example: We observe $z=2.3$ with $H_a:p>0.50$

Putting It All Together

Understanding p-values is about understanding this logical chain:

The p-value is the bridge between our sample data and our conclusion about the population.

Practice Problems