SAT Statistical Claims & Study Design
Statistical Claims & Study Design
Learn how the SAT expects you to judge statistical claims, distinguish observational studies from experiments, understand random sampling and random assignment, and decide when a conclusion can be generalized or interpreted as causal.
Population, Sample, and Statistical Claims
A population is the entire group a study is interested in. A sample is the smaller group actually studied.
What Does It Mean to Generalize?
To generalize a result means to use the study’s result to describe a larger population beyond the people directly studied.
If researchers randomly select 200 ninth-grade students from all ninth-grade students in a school district, their results can be extended to the ninth-grade student population of that district—not automatically to all teenagers in the country.
Random Sampling
In a random sample, members of the population are selected using a process designed to give members a fair opportunity to be included.
Random sample
Researchers randomly select people from the population of interest. This supports generalizing results to that population.
Convenience sample
Researchers select people who are easy to reach. The sample may not represent the population of interest.
Voluntary response
People choose whether to participate. People with strong opinions may be more likely to respond.
Large sample ≠ automatically good sample
A very large biased sample can still fail to represent the population.
Random Sampling and the Population
The wording of the population matters.
Suppose a researcher randomly selects 300 students from all students at Northview High School and finds that 62% prefer a particular school lunch. The result can be used to estimate the preference of students at Northview High School.
It does not automatically describe students at every high school in the city, state, or country.
Observational Studies
In an observational study, researchers observe characteristics, behaviors, or outcomes without assigning participants to treatments.
Example
Researchers record the amount of sleep students usually get and compare their average grades.
What is observed?
Students’ existing sleep habits and grades. Researchers do not decide who sleeps more.
An observational study by itself does not establish that changing one variable caused a change in the other.
Experiments and Treatments
In an experiment, researchers deliberately assign a treatment or condition to participants and then compare outcomes.
For example, researchers might assign students to either a new study-skills program or the usual study routine and then compare their test scores.
An experiment can provide evidence for a causal relationship when its design supports that conclusion—especially when participants are randomly assigned.
Random Assignment
Random assignment means that participants in a study are randomly placed into different treatment groups or conditions.
For example, after selecting 200 volunteers, a researcher might randomly assign 100 to use Method A and 100 to use Method B.
Random assignment helps make the groups comparable with respect to factors that could otherwise affect the outcome. Therefore, if the groups differ in their outcomes, the treatment is a plausible explanation for the difference.
Random Assignment and Causation
The phrase causal relationship means that changing one variable produces a change in another variable.
Association
Two variables are related. A study may show that people who do X tend to have different values of Y.
Causation
Evidence supports the conclusion that changing X causes a change in Y. Random assignment in an experiment is a key source of that evidence.
Association vs. Causation
Consider two studies about exercise and stress.
Study A — Observational
Researchers survey people about their usual exercise habits and stress levels. People who exercise more report lower stress.
Supported: an association between exercise and stress.
Study B — Experiment
Researchers randomly assign participants to an exercise program or a control condition, then compare stress levels.
Supported: evidence that the assigned treatment caused a difference in stress.
The subject matter is almost identical, but the study designs justify different conclusions.
Sampling Bias and Convenience Samples
A sample can be large but still systematically different from the population it is supposed to represent.
Convenience sample
A school surveys the first 100 students who enter the cafeteria.
Voluntary response
A website posts a poll and analyzes answers from people who choose to respond.
Neither method guarantees that the sample represents the entire population. People who are easiest to reach or most motivated to respond may differ systematically from others.
Large Samples Don’t Fix Bad Sampling
Suppose 20,000 people voluntarily answer an online survey about a new law. That is a large number of responses, but the people who chose to respond may not represent the entire population.
Sample size concerns how much information the study has.
Sampling method concerns whether the sample represents the population.
A larger sample generally improves precision when the sampling conditions are appropriate, but simply collecting more biased responses does not automatically make a study representative.
Random Sampling vs. Random Assignment
Two Questions — Two Randomizations
These two ideas sound similar, but they answer completely different questions about a study.
Who gets into the study?
Random sampling determines whether results can be generalized to a population.
Who gets which treatment?
Random assignment helps support a causal conclusion in an experiment.
A useful SAT habit is to ask these questions separately every time you read a study description.
The Four Study-Design Cases
A study can have random sampling, random assignment, both, or neither. The conclusions depend on which features are present.
| Random sample? | Random assignment? | What can we conclude? |
|---|---|---|
| Yes | Yes | The study can support a causal conclusion, and results can be generalized to the population from which the random sample was selected. |
| Yes | No | Results can be generalized to the sampled population, but an observational relationship does not establish causation. |
| No | Yes | Random assignment can support a causal conclusion for the study participants, but the result cannot automatically be generalized beyond the population actually represented by the sample. |
| No | No | The study does not have the design features needed to justify a broad causal claim or broad population generalization. |
How Strong Is the Claim?
The wording of an answer choice matters. Match the strength of the claim to the strength of the study design.
Association
“The variables are associated.”
Generalization
“The result can be extended to the sampled population.”
Causation
“The assigned treatment caused a difference.”
Strongest supported claim
Use the strongest statement that the design actually justifies—not a stronger one.
A Reliable SAT Statistical-Claims Strategy
- Identify the population. Who is the study trying to describe?
- Identify the sample. Who was actually studied?
- Ask how participants were selected. Was the sample random, convenient, or voluntary?
- Ask whether participants were assigned to treatments. If so, was the assignment random?
- Separate generalization from causation. They are different conclusions.
- Match the answer choice to the design. Do not claim more than the evidence supports.
The SAT Shortcut
Random sample?
Think: GENERALIZATION.
To which population can the result be extended?
Random assignment?
Think: CAUSATION.
Does the experiment provide evidence that the treatment caused the outcome?
Advanced Practice — Statistical Claims & Study Design
A researcher records the number of hours 120 students sleep each night and records each student’s test score. The researcher does not assign students to different sleep schedules.
Which type of study is this?
- A) An experiment
- B) An observational study
- C) A randomized experiment
- D) A controlled trial with random assignment
The researcher only observes existing sleep habits and test scores. No treatment is assigned, so this is an observational study.
A city has 18,000 high school students. A researcher randomly selects 300 students from all 18,000 students and surveys them about transportation to school.
To which population can the survey results reasonably be generalized?
- A) The 300 selected students only
- B) All high school students in the city
- C) All high school students in the state
- D) All teenagers in the country
The 300 students were randomly selected from the city’s 18,000 high school students. Therefore, the results can reasonably be generalized to all high school students in the city.
Which action is an example of random assignment?
- A) Randomly selecting 100 students from a school
- B) Asking students to volunteer for a study
- C) Randomly placing study participants into a treatment group or a control group
- D) Surveying the first 100 students who enter a cafeteria
Random assignment concerns which treatment or condition each participant receives. Randomly placing participants into treatment and control groups is random assignment.
Researchers randomly select 400 students from all students at a university. They ask the students how many hours they exercise each week and how much stress they report. Students who exercise more tend to report less stress.
Which conclusion is supported by the study?
- A) Exercise causes lower stress.
- B) Lower stress causes students to exercise more.
- C) Exercise and reported stress are associated among the university’s students.
- D) Increasing exercise will cause every student’s stress to decrease.
The study is observational: researchers measured existing exercise and stress levels. It can show an association, but not a causal relationship.
A school wants to estimate the percentage of all students who support a new schedule. Which sampling method is most appropriate?
- A) Survey the first 80 students who arrive at school.
- B) Survey 80 students who volunteer after an announcement.
- C) Randomly select 80 students from the school’s complete student list.
- D) Survey 80 members of the student council.
A random sample from the full student population is the best choice for estimating the views of all students.
A researcher randomly selects 500 customers from a store’s customer list and asks whether they prefer a new checkout system. The researcher does not assign any treatment.
Which statement is best supported?
- A) The checkout system causes customers to prefer it.
- B) The result can be generalized to the store’s customer population, but it does not establish causation.
- C) The result can be generalized to all shoppers everywhere.
- D) Random sampling proves that the checkout system caused the preference.
Random sampling supports generalization to the population from which the sample was selected. Because no treatment was assigned, the study does not establish causation.
Researchers recruit 240 volunteers from one university. They randomly assign 120 volunteers to use a new study app and 120 to use no study app. After four weeks, the app group has a higher mean test score.
Which statement is justified?
- A) The app caused the higher scores for the study participants.
- B) The app can be proven to improve scores for every university student.
- C) The results can be generalized to all university students because assignment was random.
- D) No causal conclusion is possible because the volunteers were not randomly assigned.
The participants were randomly assigned, so the experiment provides evidence for a causal effect among the study participants. However, they were not randomly sampled from all university students, so broad generalization is not automatically justified.
A researcher randomly selects 600 employees from a company and asks how many hours they work remotely each week. The researcher finds that employees who work remotely more hours tend to report greater job satisfaction.
Which additional feature would be needed to support a causal conclusion that remote work causes greater job satisfaction?
- A) Increase the sample from 600 to 1,200 employees.
- B) Randomly assign employees to different remote-work conditions.
- C) Survey only employees with high job satisfaction.
- D) Ask the employees to volunteer again.
The original study is observational. Randomly assigning employees to different treatment conditions would turn the study into an experiment capable of providing evidence for causation.
Two surveys are conducted about a city’s proposed recycling program. Survey A receives 2,000 responses from people who choose to answer an online poll. Survey B randomly selects 500 residents from the city’s complete resident list.
Which statement is most appropriate?
- A) Survey A must be more representative because it has more responses.
- B) Survey B is more suitable for generalizing to city residents because its participants were randomly selected.
- C) Survey A proves that the recycling program causes a change in residents’ opinions.
- D) Neither survey can provide information about residents’ opinions.
Survey A has more responses, but voluntary response can introduce bias. Survey B uses a random sample from the population of interest, making it more appropriate for generalization.
A district randomly selects 800 students from all students in the district. The students are then randomly assigned to either a new mathematics program or the standard mathematics program. After one semester, the students in the new program have a higher mean score.
Which conclusion is best supported?
- A) The new program caused the higher mean score, and the result can be generalized to the district’s student population.
- B) The new program is associated with higher scores, but causation cannot be considered because the sample was random.
- C) The result can be generalized to all students in the country, and the program caused the difference.
- D) The program caused the difference only because the students were randomly sampled.
This study has both features. Random sampling supports generalization to the district’s student population, while random assignment supports a causal conclusion about the treatment.
A university wants to know whether a new tutoring program increases exam scores. Researchers randomly select 300 students from all students at the university. They then let students choose whether to participate in the new program or continue with their usual studying. Students who choose the program have higher average scores.
Which statement is correct?
- A) The program caused the higher scores because the students were randomly selected.
- B) The results can be generalized to the university’s students, but the study does not establish causation.
- C) The study establishes causation because the sample size is 300.
- D) The results cannot be generalized because students were allowed to choose their treatment.
Random selection supports generalization to the university’s student population. But students chose their treatment, so there was no random assignment. The observed difference may be related to characteristics of students who chose the program.
A researcher wants to determine whether a new type of fertilizer causes tomato plants to produce more fruit. Which design provides the strongest evidence for a causal relationship?
- A) Observe 200 plants that already receive different fertilizers.
- B) Randomly select 200 plants and let their owners choose the fertilizer.
- C) Randomly assign 200 comparable plants to receive either the new fertilizer or the standard fertilizer.
- D) Survey 200 gardeners who use the new fertilizer.
Random assignment creates treatment groups whose differences are less likely to be explained by preexisting differences. That makes C the strongest design for causal evidence.
A researcher wants to estimate the percentage of all residents of a city who support a proposed park. They randomly select 1,000 residents and survey them. Separately, researchers want to determine whether a new park-planning workshop causes residents to become more supportive.
Which pair of methods best matches the two goals?
- A) Random sampling for the estimate; random assignment for the workshop study
- B) Random assignment for the estimate; voluntary response for the workshop study
- C) Convenience sampling for the estimate; random sampling for the workshop study
- D) Random assignment for both goals
The first goal is generalization, so random sampling is the relevant design feature. The second goal is causation, so random assignment to workshop and control conditions is the relevant feature.
A study randomly selects 450 adults from a particular county. The adults are then randomly assigned to receive either a new health-information message or the current message. Afterward, the group receiving the new message has a higher average score on a health-knowledge test.
Which statement is NOT justified?
- A) The new message provides evidence of a causal effect on the test score.
- B) The result can reasonably be generalized to the population from which the adults were randomly selected.
- C) The new message will cause higher health-knowledge scores for every adult in the county.
- D) The difference in average scores provides evidence that the treatment affected the outcome.
Random sampling and random assignment provide strong evidence for a population-level causal conclusion, but they do not guarantee that every individual will respond in the same way.
A national organization wants to evaluate a new reading program for ninth-grade students in a state. Researchers randomly select 1,200 ninth-grade students from the state’s public schools. The selected students are randomly assigned to either the new program or the standard program. After one year, the new-program group has a higher mean reading score.
Which conclusion is best supported?
- A) The program caused higher reading scores among the study participants, and the result can be generalized to the population from which the students were randomly selected.
- B) The program is associated with higher scores, but causation cannot be inferred because the sample was random.
- C) The program caused higher scores for every ninth-grade student in the country.
- D) The results can be generalized to all students worldwide because assignment was random.
This study deliberately contains both randomizations. Random sampling supports generalization to the population represented by the sampling frame. Random assignment supports a causal conclusion about the effect of the program. The claim must not be extended beyond the population from which the sample was selected.
Statistical Claims — Final Checklist
Identify the population and the sample.
If yes, think about generalization.
If yes, determine whether assignment was random.
If yes, an experiment can provide causal evidence.
Then an association does not by itself establish causation.
A large convenience or voluntary sample may still be unrepresentative.
Do not generalize beyond the population from which the random sample was selected.
Match the answer choice to the actual study design.
One thought on “SAT Statistical Claims & Study Design”